Creating Custom SOAP Extensions - Compression Extension

Introduction
Web Services have already been standardized and its usage if definitely on the rise. Developers are quickly finding new and better ways to apply Web Services in their existing or new application designs. Reviewing many of the Web Services designs I have noticed that developers are increasingly performing heavy weight tasks (passing large SOAP messages) using Web Services. They are passing thousands of records through the Web Methods, there is nothing wrong with this approach, but the only problem (or nightmare) that comes forward is the size of data being passed around. Web Services use XML formatted SOAP messages to communicate but XML is a very verbose language and it can certainly add-up a lot to the size of data being transferred. Global bandwidth conditions do not currently warrant such massive data transfer easily.
One solution to this would be to compress (zip) the SOAP Message before sending it across the wire and then deflate (unzip) the data back on the client. This will help to reduce the amount of data transferred over the wire drastically! Luckily, the .NET Framework provides a very extensible Web Services API which makes it very easy to implement this functionality.

SOAP Extensions
The .NET Web Service API uses SOAP Extensions to provide hooks for developers into the messaging Web Service architecture, both on the client and server side. SOAP Extensions are used to inspect ( and modify ) the SOAP Message at each stage in the Web Services communication cycle.
To visualize the life cycle of a SOAP Request and Response, consider this scenario the client (of the Web Service) prepares the parameters (if any) and then calls the web method through the proxy class. Here the proxy class, inspects the method parameters, marshals them into XML format (serialization) and creates a SOAP Message. This SOAP message is then sent to the server  (Web Service). The Web Service receives the SOAP message it converts (deserializes) the XML formatted parameters into the language data-types and then invokes the relevant method. Next, the method on the server-side executes, does the necessary processing and it may return a value. The return value is again marshaled (serialized) into a SOAP message and sent back to the client. The client proxy receives the SOAP message converts (deserializes) the XML value and returns it back to the calling class. In short both client and server carry the task of serialization and deserialization.
SOAP Extensions in ASP.NET allow you to inspect and alter the SOAP Message at all the stages of the XML Web Service Lifetime both at client and server side. This provides you with a very powerful way to extend the existing Web Service framework to suit your requirements.
You can learn more about the Anatomy of an XML Web Services Lifetime in the MSDN documentation.

SOAP Message Compression
Currently, there are no standards involving compression of SOAP Messages, hence we are on our own implement a suitable extension. I have chosen to use zip algorithm, you can feel free to choose your own algorithm, but since I am using industry standard algorithms I can be sure they will be supported across platforms and programming languages. I am using the #ziplip library from ICSharpCode which is a GPL based implementation for the standard ZIP, GZIP, TAR and BZip2 algorithms in C# (with source code) by Mike Krueger. A special thanks to Mike Krueger for developing such a wonderful library in C#!

Another important question that comes into mind is Should we compress the whole SOAP Message before we send it across the wire or parts of it? The up-coming new standards rely a lot on the SOAP Headers, hence we chosen not to compress the SOAP Headers since they might be needed in plain XML format for your Web Service to function correctly. So we just compress the contents of the SOAP Body and then send the message across the wire. Also remember we are implementing this extension on both the client and server side, so all communication between the client and server will be compressed.

Please note that you will have to download and install the #ziplib library on both the client and server for this extension to work correctly. Instructions for installation of the library are clearly outlined in the support documentation.

Let's Dive into some code.
In order to implement a SOAP Extension we need to define 2 classes, the first class extends the SoapExtention class and it contains all the processing logic for the extension. The class extends the SoapExtensionAttribute class which is the attribute used to apply on the Web Methods that need to support the SOAP Extension.

CompressionExtension
In the CompressionExtension class I check for the two stages, AfterSerialize and BeforeDeserialize. In the AfterSerialize stage, I inspect the contents of the SOAP Body and zip them, then the zipped stream is saved back into the SOAP Body and the SOAP Message is passed over the wire. On the other hand in BeforeDeserialize stage, I inspect the SOAP Body and retrieve the zipped contents, unzip them, and restore the original content. Later, the SOAP Message is sent to the Web Method for processing.
Also remember while zipping the XML nodes we get a binary array, but binary arrays are not correctly represented in XML Web Services, hence we have to use Base64 encoding to convert the binary array into appropriate format.
 
Update: As per Tobias Gansen's suggestion, the source code has been updated. Now I use InnerXml instead of InnetText to retrieve the contents of the SOAP Body, so complex obhjects like DataSet's are correctly handled.
25 October 2002 - As per Don Bunt's suggestion the code has been updated. Now all the WebMethod parameters get compressed correctly.
 

using System;
using System.IO;
using System.Text ;
using System.Web.Services;
using System.Web.Services.Protocols ;
using ICSharpCode.SharpZipLib.Checksums;
using ICSharpCode.SharpZipLib.Zip;
using ICSharpCode.SharpZipLib.GZip;
using System.Xml ;

namespace MasterCSharp.WebServices
{
  /// <summary>
  /// Summary description for ConpressionExtension.
  /// </summary>
  public class CompressionExtension : System.Web.Services.Protocols.SoapExtension
  {
    Stream oldStream;
    Stream newStream;


    public override object GetInitializer(LogicalMethodInfo methodInfo,
				SoapExtensionAttribute attribute)
    {
      return attribute;
    }

    // Get the Type
    public override object GetInitializer(Type t)
    {
      return typeof(CompressionExtension);
    }

    // Get the CompressionExtensionAttribute
    public override void Initialize(object initializer)
    {
      CompressionExtensionAttribute attribute =
		(CompressionExtensionAttribute) initializer;

      return;
    }

    // Process the SOAP Message
    public override void ProcessMessage(SoapMessage message)
    {
      // Check for the various SOAP Message Stages 
      switch (message.Stage)
      {

        case SoapMessageStage.BeforeSerialize:
          break;

        case SoapMessageStage.AfterSerialize:
          // ZIP the contents of the SOAP Body after it has
          // been serialized
          Zip();
          break;

        case SoapMessageStage.BeforeDeserialize:
          // Unzip the contents of the SOAP Body before it is
          // deserialized
          Unzip();
          break;

        case SoapMessageStage.AfterDeserialize:
          break;

        default:
          throw new Exception("invalid stage");
      }
    }

    // Gives us the ability to get hold of the RAW SOAP message
    public override Stream ChainStream( Stream stream )
    {
      oldStream = stream;
      newStream = new MemoryStream();
      return newStream;
    }

    // Utility method to copy streams
    void Copy(Stream from, Stream to)
    {
      TextReader reader = new StreamReader(from);
      TextWriter writer = new StreamWriter(to);
      writer.WriteLine(reader.ReadToEnd());
      writer.Flush();
    }


    // Zip the SOAP Body
    private void Zip()
    {
      newStream.Position = 0;
      // Zip the SOAP Body
      newStream = ZipSoap(newStream);
      // Copy the streams
      Copy(newStream, oldStream);
    }

    // The actual ZIP method
    private byte[] Zip(string stringToZip)
    {
      byte[] inputByteArray = Encoding.UTF8.GetBytes(stringToZip);
      MemoryStream ms = new MemoryStream();

      // Check the #ziplib docs for more information
      ZipOutputStream zipOut = new ZipOutputStream( ms ) ;
      ZipEntry ZipEntry = new ZipEntry("ZippedFile");
      zipOut.PutNextEntry(ZipEntry);
      zipOut.SetLevel(9);
      zipOut.Write(inputByteArray, 0 , inputByteArray.Length ) ;
      zipOut.Finish();
      zipOut.Close();

      // Return the zipped contents
      return ms.ToArray();
    }

    // Select and Zip the appropriate parts of the SOAP message
    public MemoryStream ZipSoap(Stream streamToZip)
    {
      streamToZip.Position = 0;
      // Load a XML Reader
      XmlTextReader reader = new XmlTextReader(streamToZip);
      XmlDocument dom = new XmlDocument();
      dom.Load(reader);
      // Load a NamespaceManager to enable XPath selection
      XmlNamespaceManager nsmgr = new XmlNamespaceManager(dom.NameTable);
      nsmgr.AddNamespace("soap", "http://schemas.xmlsoap.org/soap/envelope/");
      XmlNode node = dom.SelectSingleNode("//soap:Body", nsmgr);
      // Select the contents within the method defined in the SOAP body
      node = node.FirstChild.FirstChild;
      // Check if there are any nodes selected
      while( node != null )
      {
        if( node.InnerXml.Length > 0 )
        {
          // Zip the data
          byte[] outData = Zip(node.InnerXml);
          // Convert it to Base64 for transfer over the internet
          node.InnerXml = Convert.ToBase64String(outData) ;
        }
        // Move to the next parameter
        node = node.NextSibling ; 
      }
      MemoryStream ms = new MemoryStream();
      // Save the updated data
      dom.Save(ms);
      ms.Position = 0;

      return ms;
    }

    // Unzip the SOAP Body
    private void Unzip()
    {
      MemoryStream unzipedStream = new MemoryStream();

      TextReader reader = new StreamReader(oldStream);
      TextWriter writer = new StreamWriter(unzipedStream);
      writer.WriteLine(reader.ReadToEnd());
      writer.Flush();
      // Unzip the SOAP Body
      unzipedStream = UnzipSoap(unzipedStream);
      // Copy the streams
      Copy(unzipedStream, newStream);

      newStream.Position = 0;
    }

    // Actual Unzip logic
    private byte[] Unzip(string stringToUnzip)
    {
      // Decode the Base64 encoding
      byte[] inputByteArray = Convert.FromBase64String( stringToUnzip ) ;
      MemoryStream ms = new MemoryStream(inputByteArray) ;
      MemoryStream ret = new MemoryStream();

      // Refer to #ziplib documentation for more info on this
      ZipInputStream zipIn = new ZipInputStream(ms);
      ZipEntry theEntry = zipIn.GetNextEntry();
      Byte[] buffer = new Byte[2048] ;
      int size = 2048;
      while (true)
      {
        size = zipIn.Read(buffer, 0, buffer.Length);
        if (size > 0)
        {
          ret.Write(buffer, 0, size);
        }
        else
        {
          break;
        }
      }
      return ret.ToArray();
    }

    // Unzip the SOAP Body
    public MemoryStream UnzipSoap(Stream streamToUnzip)
    {
      streamToUnzip.Position = 0;
      // Load a XmlReader
      XmlTextReader reader = new XmlTextReader(streamToUnzip);
      XmlDocument dom = new XmlDocument();
      dom.Load(reader);

      XmlNamespaceManager nsmgr = new XmlNamespaceManager(dom.NameTable);
      nsmgr.AddNamespace("soap", "http://schemas.xmlsoap.org/soap/envelope/");
      // Select the SOAP Body node 
      XmlNode node = dom.SelectSingleNode("//soap:Body", nsmgr);
      node = node.FirstChild.FirstChild;

      // Check if node exists
      while( node != null )
      {
        if( node.InnerXml.Length > 0 )
        {
          // Send the node's contents to be unziped
          byte[] outData = Unzip(node.InnerXml);
          string sTmp = Encoding.UTF8.GetString(outData);
          node.InnerXml = sTmp;
        }
        // Move to the next parameter
        node = node.NextSibling ; 
      }

      MemoryStream ms = new MemoryStream();

      dom.Save(ms);
      ms.Position = 0;

      return ms;
    }

  }
}

CompressionExtensionAttribute
This class defines an Attribute which will be used to mark-up Web Methods that need to be compressed. This class inherits the SoapExtensionAttribute class and overrides its ExtensionType and Priority properties.

using System;
using System.Web.Services;
using System.Web.Services.Protocols;

namespace MasterCSharp.WebServices
{

  /// <summary>
  /// Summary description for CompressionExtensionAttribute.
  /// </summary>

  // Make the Attribute only Applicable to Methods
  [AttributeUsage(AttributeTargets.Method)]
  public class CompressionExtensionAttribute : SoapExtensionAttribute
  {

    private int priority;

    // Override the base class properties
    public override Type ExtensionType
    {
      get { return typeof(CompressionExtension); }
    }

    public override int Priority
    {
      get
      {
        return priority;
      }
      set
      {
        priority = value;
      }
    }

  }
}

Code Compilation
Once you have the CompressionExtension.cs and CompressionExtensionAttribute.cs files ready, its time to compile them and convert them into a library. If you use VS.NET then you need to Add Reference to the System.Web.Services, System.Xml and SharpZipLib assemblies and compile the project. If you are manually compiling the files, then following compilation string:
csc /t:library /out:CompressionExtension.dll /r:SharpZipLib.dll CompressionExtension.cs CompressionExtensionAttribute.cs

This will produce the CompressionExtension.dll library. Now you can use this library in your clients and Web Services to enable compression extension. Note: Remember the clients and servers will also need to install the SharpZibLib.dll library also for this extension to work!

Sample Web Service
Our Compression Extension is ready for usage. I will create a simple Web Service which will demonstrate the usage of the Compression Extension, note the goal here is show the usage of the Compression Extension and not to teach how to create and deploy Web Services. If you want to learn more about Web Service basics please read this article.

I am using the Pubs sample database which installs with MSDE 2000 (SQL Server), just because I want to return a lot of data for testing purpose. But the Compression Extension is in no way tied to the database, it can be applied on any Web Method. As a matter of fact the Compression Extension does not even dictate there method parameter and  return parameter data type. Hence you can apply this extension on any Web Method no matter what parameters / return types it uses.
In the listing below, I have defined a basic Web Service that returns a string containing the XML representation of the Authors, Titles and Publishers tables from the Pubs database.
There are only two changes; Firstly, I have referenced the MasterCSharp.WebServices namespace that contains the Compression Extension. Secondly, I have applied the Compression Extension on the GetDetails Web Method. That's all you  need to do!!! It just can't get any simpler than this :).
While deploying the Web Service, you should ensure the SharpZipLib assembly has been registered in the GAC, alternatively you can also copy the library into the BIN directory of the Virtual Directory hosting this Web Service. Also don't forget to copy the CompressionExtension.dll library into BIN directory of the Virtual Root hosting this Web Service.

One important point to be noted is that we have applied a SOAP Extension, and this SOAP Extension is called only when a SOAP call is made to a Web Method. Hence if we are making HTTP GET / POST calls the Compression Extension is NOT in effect. Hence when we check the Web Service in Internet Explorer using the ASP.NET rendered test interface, you cannot see any compressed output.

<%@ WebService Language="C#" Class="PubsService" %>

using System.Web.Services;
using MasterCSharp.WebServices ;
using System.Data ;
using System.Data.SqlClient ;

public class PubsService {

  [WebMethod]
  [CompressionExtension]
  public string GetDetails() {

    // Replace with the connection string to connect to your database
    SqlConnection myCon = new SqlConnection("server=(local)\\NetSDK;database=pubs;" ;
				+Trusted_Connection=yes");
    SqlDataAdapter myCommand1 = new SqlDataAdapter("SELECT * FROM Authors", myCon);
    SqlDataAdapter myCommand2 = new SqlDataAdapter("SELECT * FROM Titles", myCon);
    SqlDataAdapter myCommand3 = new SqlDataAdapter("SELECT * FROM Publishers", myCon);

    DataSet ds = new DataSet();
    myCommand1.Fill(ds, "Authors");
    myCommand2.Fill(ds, "Titles");
    myCommand3.Fill(ds, "Publishers");

    return ds.GetXml() ;
  }

}

Test Web Service Client
In order to consume our PubsService I create a console based test client. In VS.NET you can create a new Console Project and then Add Web Reference to the PubsService and Add Reference to the CompressionExtension.dll library. Remember even method parameters sent from the client are compressed, hence you need to reference the Compression Extension library even at the client side.
In case of manual class creation, use the WSDL tool to generate a Proxy Class for the PubsService (replace the URL to the Web Service as per your setup):
wsdl http://localhost/PubsService/PubsService.asmx?WSDL

This creates a PubsService.cs Proxy Class file. Now let's create the Pubs Test Application using this proxy class. The listing below shows the code for the PubsTestApp. In this test application, I just make an instance of the Proxy class and call a Web Method. The value returned by the Web Service is saved to a file (so I can measure the bytes returned) and also printed on the console screen. Compile this application using the following compilation string:
csc /out:PubsTestApp.exe PubsService.cs PubsTestApp.cs

using System;
using System.IO ;

public class PubsTestApp
{
  public static void Main()
  {
    // Create an instance of the Proxy Class
    PubsService pws = new PubsService();
    // Open a StreamReader to a local file
    // This file will be used to log the return value
    // of the SOAP call
    StreamWriter sw = new StreamWriter( "log.txt" );
    // Call the Web Method and write the return value to file
    sw.Write( pws.GetDetails() ) ;
    sw.Close();
    // Call the Web Method again and write value to Console
    Console.WriteLine( pws.GetDetails() ) ;

    Console.WriteLine();
    Console.WriteLine("Hit Enter to Close Window!");
    Console.ReadLine();
  }
}

Once the application is compiled we call the PubsTestApp, it runs and shows some garbage characters, this output is even saved to the log.txt file. In reality what's shown is not garbage, but its the Base 64 encoding of the zipped data. Open Windows Explorer and observe the file size of the file log.txt, its approximately 4,864 bytes on my computer. This is the compressed data that's been sent to the client.

Next, you need to apply the the Compression Extension to the Proxy Class, so that you can receive normal output (unzipped data) from the Web Service. This is the step you would normally be taking right after generating the Proxy Class. Open the PubsService.cs proxy class file and add the [CompressionExtension] attribute above the public string GetDetails() method definition. Also reference the MasterCSharp.WebServices namespace which contains the Compression Extension. Once you apply the attribute, re-compile the PubsTestApp this time using the following compilation string: 
csc /out:PubsTestApp.exe /r:CompressionExtension.dll PubsService.cs PubsTestApp.cs

Note: The CompressionExtension.dll library should be in the same directory we are compiling the application and the SharpZipLib.dll library should also be appropriately installed on the client.

Now run the PubsTestApp again, this time correct XML output is shown on the console screen, indicating the extension works on both ends seamlessly. Observe the size of the log.txt file again and this time its around 15,908 bytes, this is the amount of data that would have been passed if we did not use compression. Hence the Compression Extension has achieved almost 30% size reduction in the SOAP Messages used to communicate between the client and the server. Although, this compression ratio will vary on compression algorithm and the type of data you are compressing.

Conclusion
This article provides a practical SOAP Extension that can be used in any kind of Web Services, to utilize network resources better. In this article I not only highlighted the creation and deployment of a SOAP Extension in .NET, but I also silently displayed how to pass binary data using Web Service i.e. using Base 64 Encoding. If you are wondering if this method would still let your Web Service interop with other platforms and programming languages, well I haven't worked with other toolkits, but I am sure, they would definitely allow you to manipulate the SOAP Message. And since we are using standard ZIP algorithms, I don't see a huge barrier to the usage of this Web Service across different platforms and programming languages. If some of you get a chance to test this, please do let me know!!