Capturing and Transforming ASP.NET Output with Response.Filter
- by Rick Strahl
During one of my Handlers and Modules session at DevConnections this week one of the attendees asked a question that I didn’t have an immediate answer for. Basically he wanted to capture response output completely and then apply some filtering to the output – effectively injecting some additional content into the page AFTER the page had completely rendered. Specifically the output should be captured from anywhere – not just a page and have this code injected into the page. Some time ago I posted some code that allows you to capture ASP.NET Page output by overriding the Render() method, capturing the HtmlTextWriter() and reading its content, modifying the rendered data as text then writing it back out. I’ve actually used this approach on a few occasions and it works fine for ASP.NET pages. But this obviously won’t work outside of the Page class environment and it’s not really generic – you have to create a custom page class in order to handle the output capture. [updated 11/16/2009 – updated ResponseFilterStream implementation and a few additional notes based on comments] Enter Response.Filter However, ASP.NET includes a Response.Filter which can be used – well to filter output. Basically Response.Filter is a stream through which the OutputStream is piped back to the Web Server (indirectly). As content is written into the Response object, the filter stream receives the appropriate Stream commands like Write, Flush and Close as well as read operations although for a Response.Filter that’s uncommon to be hit. The Response.Filter can be programmatically replaced at runtime which allows you to effectively intercept all output generation that runs through ASP.NET. A common Example: Dynamic GZip Encoding A rather common use of Response.Filter hooking up code based, dynamic GZip compression for requests which is dead simple by applying a GZipStream (or DeflateStream) to Response.Filter. The following generic routines can be used very easily to detect GZip capability of the client and compress response output with a single line of code and a couple of library helper routines: WebUtils.GZipEncodePage();
which is handled with a few lines of reusable code and a couple of static helper methods:
/// <summary>
///Sets up the current page or handler to use GZip through a Response.Filter
///IMPORTANT: ///You have to call this method before any output is generated!
/// </summary>
public static void GZipEncodePage()
{
HttpResponse Response = HttpContext.Current.Response;
if(IsGZipSupported())
{
stringAcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
if(AcceptEncoding.Contains("deflate"))
{
Response.Filter = newSystem.IO.Compression.DeflateStream(Response.Filter,
System.IO.Compression.CompressionMode.Compress);
Response.AppendHeader("Content-Encoding", "deflate");
}
else
{
Response.Filter = newSystem.IO.Compression.GZipStream(Response.Filter,
System.IO.Compression.CompressionMode.Compress);
Response.AppendHeader("Content-Encoding", "gzip"); }
}
// Allow proxy servers to cache encoded and unencoded versions separately
Response.AppendHeader("Vary", "Content-Encoding");
}
/// <summary>
/// Determines if GZip is supported
/// </summary>
/// <returns></returns>
public static bool IsGZipSupported()
{
string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
if (!string.IsNullOrEmpty(AcceptEncoding) &&
(AcceptEncoding.Contains("gzip") || AcceptEncoding.Contains("deflate")))
return true;
return false;
}
GZipStream and DeflateStream are streams that are assigned to Response.Filter and by doing so apply the appropriate compression on the active Response.
Response.Filter content is chunked
So to implement a Response.Filter effectively requires only that you implement a custom stream and handle the Write() method to capture Response output as it’s written. At first blush this seems very simple – you capture the output in Write, transform it and write out the transformed content in one pass. And that indeed works for small amounts of content. But you see, the problem is that output is written in small buffer chunks (a little less than 16k it appears) rather than just a single Write() statement into the stream, which makes perfect sense for ASP.NET to stream data back to IIS in smaller chunks to minimize memory usage en route.
Unfortunately this also makes it a more difficult to implement any filtering routines since you don’t directly get access to all of the response content which is problematic especially if those filtering routines require you to look at the ENTIRE response in order to transform or capture the output as is needed for the solution the gentleman in my session asked for.
So in order to address this a slightly different approach is required that basically captures all the Write() buffers passed into a cached stream and then making the stream available only when it’s complete and ready to be flushed.
As I was thinking about the implementation I also started thinking about the few instances when I’ve used Response.Filter implementations. Each time I had to create a new Stream subclass and create my custom functionality but in the end each implementation did the same thing – capturing output and transforming it. I thought there should be an easier way to do this by creating a re-usable Stream class that can handle stream transformations that are common to Response.Filter implementations.
Creating a semi-generic Response Filter Stream Class
What I ended up with is a ResponseFilterStream class that provides a handful of Events that allow you to capture and/or transform Response content. The class implements a subclass of Stream and then overrides Write() and Flush() to handle capturing and transformation operations. By exposing events it’s easy to hook up capture or transformation operations via single focused methods.
ResponseFilterStream exposes the following events:
CaptureStream, CaptureString
Captures the output only and provides either a MemoryStream or String with the final page output. Capture is hooked to the Flush() operation of the stream.
TransformStream, TransformString
Allows you to transform the complete response output with events that receive a MemoryStream or String respectively and can you modify the output then return it back as a return value. The transformed output is then written back out in a single chunk to the response output stream. These events capture all output internally first then write the entire buffer into the response.
TransformWrite, TransformWriteString
Allows you to transform the Response data as it is written in its original chunk size in the Stream’s Write() method. Unlike TransformStream/TransformString which operate on the complete output, these events only see the current chunk of data written. This is more efficient as there’s no caching involved, but can cause problems due to searched content splitting over multiple chunks.
Using this implementation, creating a custom Response.Filter transformation becomes as simple as the following code.
To hook up the Response.Filter using the MemoryStream version event:
ResponseFilterStream filter = new ResponseFilterStream(Response.Filter);
filter.TransformStream += filter_TransformStream;
Response.Filter = filter;
and the event handler to do the transformation:
MemoryStream filter_TransformStream(MemoryStream ms)
{
Encoding encoding = HttpContext.Current.Response.ContentEncoding;
string output = encoding.GetString(ms.ToArray());
output = FixPaths(output);
ms = new MemoryStream(output.Length);
byte[] buffer = encoding.GetBytes(output);
ms.Write(buffer,0,buffer.Length);
return ms;
}
private string FixPaths(string output)
{
string path = HttpContext.Current.Request.ApplicationPath; // override root path wonkiness
if (path == "/")
path = "";
output = output.Replace("\"~/", "\"" + path + "/").Replace("'~/", "'" + path + "/");
return output;
}
The idea of the event handler is that you can do whatever you want to the stream and return back a stream – either the same one that’s been modified or a brand new one – which is then sent back to as the final response.
The above code can be simplified even more by using the string version events which handle the stream to string conversions for you:
ResponseFilterStream filter = new ResponseFilterStream(Response.Filter);
filter.TransformString += filter_TransformString;
Response.Filter = filter;
and the event handler to do the transformation calling the same FixPaths method shown above:
string filter_TransformString(string output)
{
return FixPaths(output);
}
The events for capturing output and capturing and transforming chunks work in a very similar way. By using events to handle the transformations ResponseFilterStream becomes a reusable component and we don’t have to create a new stream class or subclass an existing Stream based classed.
By the way, the example used here is kind of a cool trick which transforms “~/” expressions inside of the final generated HTML output – even in plain HTML controls not HTML controls – and transforms them into the appropriate application relative path in the same way that ResolveUrl would do.
So you can write plain old HTML like this:
<a href=”~/default.aspx”>Home</a>
and have it turned into:
<a href=”/myVirtual/default.aspx”>Home</a>
without having to use an ASP.NET control like Hyperlink or Image or having to constantly use:
<img src=”<%= ResolveUrl(“~/images/home.gif”) %>” />
in MVC applications (which frankly is one of the most annoying things about MVC especially given the path hell that extension-less and endpoint-less URLs impose).
I can’t take credit for this idea. While discussing the Response.Filter issues on Twitter a hint from Dylan Beattie who pointed me at one of his examples which does something similar. I thought the idea was cool enough to use an example for future demos of Response.Filter functionality in ASP.NET next I time I do the Modules and Handlers talk (which was great fun BTW).
How practical this is is debatable however since there’s definitely some overhead to using a Response.Filter in general and especially on one that caches the output and the re-writes it later. Make sure to test for performance anytime you use Response.Filter hookup and make sure it' doesn’t end up killing perf on you. You’ve been warned :-}.
How does ResponseFilterStream work?
The big win of this implementation IMHO is that it’s a reusable component – so for implementation there’s no new class, no subclassing – you simply attach to an event to implement an event handler method with a straight forward signature to retrieve the stream or string you’re interested in.
The implementation is based on a subclass of Stream as is required in order to handle the Response.Filter requirements. What’s different than other implementations I’ve seen in various places is that it supports capturing output as a whole to allow retrieving the full response output for capture or modification. The exception are the TransformWrite and TransformWrite events which operate only active chunk of data written by the Response.
For captured output, the Write() method captures output into an internal MemoryStream that is cached until writing is complete. So Write() is called when ASP.NET writes to the Response stream, but the filter doesn’t pass on the Write immediately to the filter’s internal stream. The data is cached and only when the Flush() method is called to finalize the Stream’s output do we actually send the cached stream off for transformation (if the events are hooked up) and THEN finally write out the returned content in one big chunk.
Here’s the implementation of ResponseFilterStream:
/// <summary>
/// A semi-generic Stream implementation for Response.Filter with
/// an event interface for handling Content transformations via
/// Stream or String.
/// <remarks>
/// Use with care for large output as this implementation copies
/// the output into a memory stream and so increases memory usage.
/// </remarks>
/// </summary>
public class ResponseFilterStream : Stream
{
/// <summary>
/// The original stream
/// </summary>
Stream _stream;
/// <summary>
/// Current position in the original stream
/// </summary>
long _position;
/// <summary>
/// Stream that original content is read into
/// and then passed to TransformStream function
/// </summary>
MemoryStream _cacheStream = new MemoryStream(5000);
/// <summary>
/// Internal pointer that that keeps track of the size
/// of the cacheStream
/// </summary>
int _cachePointer = 0;
/// <summary>
///
/// </summary>
/// <param name="responseStream"></param>
public ResponseFilterStream(Stream responseStream)
{
_stream = responseStream;
}
/// <summary>
/// Determines whether the stream is captured
/// </summary>
private bool IsCaptured
{
get
{
if (CaptureStream != null || CaptureString != null ||
TransformStream != null || TransformString != null)
return true;
return false;
}
}
/// <summary>
/// Determines whether the Write method is outputting data immediately
/// or delaying output until Flush() is fired.
/// </summary>
private bool IsOutputDelayed
{
get
{
if (TransformStream != null || TransformString != null)
return true;
return false;
}
}
/// <summary>
/// Event that captures Response output and makes it available
/// as a MemoryStream instance. Output is captured but won't
/// affect Response output.
/// </summary>
public event Action<MemoryStream> CaptureStream;
/// <summary>
/// Event that captures Response output and makes it available
/// as a string. Output is captured but won't affect Response output.
/// </summary>
public event Action<string> CaptureString;
/// <summary>
/// Event that allows you transform the stream as each chunk of
/// the output is written in the Write() operation of the stream.
/// This means that that it's possible/likely that the input
/// buffer will not contain the full response output but only
/// one of potentially many chunks.
///
/// This event is called as part of the filter stream's Write()
/// operation.
/// </summary>
public event Func<byte[], byte[]> TransformWrite;
/// <summary>
/// Event that allows you to transform the response stream as
/// each chunk of bytep[] output is written during the stream's write
/// operation. This means it's possibly/likely that the string
/// passed to the handler only contains a portion of the full
/// output. Typical buffer chunks are around 16k a piece.
///
/// This event is called as part of the stream's Write operation.
/// </summary>
public event Func<string, string> TransformWriteString;
/// <summary>
/// This event allows capturing and transformation of the entire
/// output stream by caching all write operations and delaying final
/// response output until Flush() is called on the stream.
/// </summary>
public event Func<MemoryStream, MemoryStream> TransformStream;
/// <summary>
/// Event that can be hooked up to handle Response.Filter
/// Transformation. Passed a string that you can modify and
/// return back as a return value. The modified content
/// will become the final output.
/// </summary>
public event Func<string, string> TransformString;
protected virtual void OnCaptureStream(MemoryStream ms)
{
if (CaptureStream != null)
CaptureStream(ms);
}
private void OnCaptureStringInternal(MemoryStream ms)
{
if (CaptureString != null)
{
string content = HttpContext.Current.Response.ContentEncoding.GetString(ms.ToArray());
OnCaptureString(content);
}
}
protected virtual void OnCaptureString(string output)
{
if (CaptureString != null)
CaptureString(output);
}
protected virtual byte[] OnTransformWrite(byte[] buffer)
{
if (TransformWrite != null)
return TransformWrite(buffer);
return buffer;
}
private byte[] OnTransformWriteStringInternal(byte[] buffer)
{
Encoding encoding = HttpContext.Current.Response.ContentEncoding;
string output = OnTransformWriteString(encoding.GetString(buffer));
return encoding.GetBytes(output);
}
private string OnTransformWriteString(string value)
{
if (TransformWriteString != null)
return TransformWriteString(value);
return value;
}
protected virtual MemoryStream OnTransformCompleteStream(MemoryStream ms)
{
if (TransformStream != null)
return TransformStream(ms);
return ms;
}
/// <summary>
/// Allows transforming of strings
///
/// Note this handler is internal and not meant to be overridden
/// as the TransformString Event has to be hooked up in order
/// for this handler to even fire to avoid the overhead of string
/// conversion on every pass through.
/// </summary>
/// <param name="responseText"></param>
/// <returns></returns>
private string OnTransformCompleteString(string responseText)
{
if (TransformString != null)
TransformString(responseText);
return responseText;
}
/// <summary>
/// Wrapper method form OnTransformString that handles
/// stream to string and vice versa conversions
/// </summary>
/// <param name="ms"></param>
/// <returns></returns>
internal MemoryStream OnTransformCompleteStringInternal(MemoryStream ms)
{
if (TransformString == null)
return ms;
//string content = ms.GetAsString();
string content = HttpContext.Current.Response.ContentEncoding.GetString(ms.ToArray());
content = TransformString(content);
byte[] buffer = HttpContext.Current.Response.ContentEncoding.GetBytes(content);
ms = new MemoryStream();
ms.Write(buffer, 0, buffer.Length);
//ms.WriteString(content);
return ms;
}
/// <summary>
///
/// </summary>
public override bool CanRead
{
get { return true; }
}
public override bool CanSeek
{
get { return true; }
}
/// <summary>
///
/// </summary>
public override bool CanWrite
{
get { return true; }
}
/// <summary>
///
/// </summary>
public override long Length
{
get { return 0; }
}
/// <summary>
///
/// </summary>
public override long Position
{
get { return _position; }
set { _position = value; }
}
/// <summary>
///
/// </summary>
/// <param name="offset"></param>
/// <param name="direction"></param>
/// <returns></returns>
public override long Seek(long offset, System.IO.SeekOrigin direction)
{
return _stream.Seek(offset, direction);
}
/// <summary>
///
/// </summary>
/// <param name="length"></param>
public override void SetLength(long length)
{
_stream.SetLength(length);
}
/// <summary>
///
/// </summary>
public override void Close()
{
_stream.Close();
}
/// <summary>
/// Override flush by writing out the cached stream data
/// </summary>
public override void Flush()
{
if (IsCaptured && _cacheStream.Length > 0)
{
// Check for transform implementations
_cacheStream = OnTransformCompleteStream(_cacheStream);
_cacheStream = OnTransformCompleteStringInternal(_cacheStream);
OnCaptureStream(_cacheStream);
OnCaptureStringInternal(_cacheStream);
// write the stream back out if output was delayed
if (IsOutputDelayed)
_stream.Write(_cacheStream.ToArray(), 0, (int)_cacheStream.Length);
// Clear the cache once we've written it out
_cacheStream.SetLength(0);
}
// default flush behavior
_stream.Flush();
}
/// <summary>
///
/// </summary>
/// <param name="buffer"></param>
/// <param name="offset"></param>
/// <param name="count"></param>
/// <returns></returns>
public override int Read(byte[] buffer, int offset, int count)
{
return _stream.Read(buffer, offset, count);
}
/// <summary>
/// Overriden to capture output written by ASP.NET and captured
/// into a cached stream that is written out later when Flush()
/// is called.
/// </summary>
/// <param name="buffer"></param>
/// <param name="offset"></param>
/// <param name="count"></param>
public override void Write(byte[] buffer, int offset, int count)
{
if ( IsCaptured )
{
// copy to holding buffer only - we'll write out later
_cacheStream.Write(buffer, 0, count);
_cachePointer += count;
}
// just transform this buffer
if (TransformWrite != null)
buffer = OnTransformWrite(buffer);
if (TransformWriteString != null)
buffer = OnTransformWriteStringInternal(buffer);
if (!IsOutputDelayed)
_stream.Write(buffer, offset, buffer.Length);
}
}
The key features are the events and corresponding OnXXX methods that handle the event hookups, and the Write() and Flush() methods of the stream implementation. All the rest of the members tend to be plain jane passthrough stream implementation code without much consequence.
I do love the way Action<t> and Func<T> make it so easy to create the event signatures for the various events – sweet.
A few Things to consider
Performance
Response.Filter is not great for performance in general as it adds another layer of indirection to the ASP.NET output pipeline, and this implementation in particular adds a memory hit as it basically duplicates the response output into the cached memory stream which is necessary since you may have to look at the entire response. If you have large pages in particular this can cause potentially serious memory pressure in your server application. So be careful of wholesale adoption of this (or other) Response.Filters. Make sure to do some performance testing to ensure it’s not killing your app’s performance.
Response.Filter works everywhere
A few questions came up in comments and discussion as to capturing ALL output hitting the site and – yes you can definitely do that by assigning a Response.Filter inside of a module. If you do this however you’ll want to be very careful and decide which content you actually want to capture especially in IIS 7 which passes ALL content – including static images/CSS etc. through the ASP.NET pipeline. So it is important to filter only on what you’re looking for – like the page extension or maybe more effectively the Response.ContentType.
Response.Filter Chaining
Originally I thought that filter chaining doesn’t work at all due to a bug in the stream implementation code. But it’s quite possible to assign multiple filters to the Response.Filter property. So the following actually works to both compress the output and apply the transformed content:
WebUtils.GZipEncodePage();
ResponseFilterStream filter = new ResponseFilterStream(Response.Filter);
filter.TransformString += filter_TransformString;
Response.Filter = filter;
However the following does not work resulting in invalid content encoding errors:
ResponseFilterStream filter = new ResponseFilterStream(Response.Filter);
filter.TransformString += filter_TransformString;
Response.Filter = filter;
WebUtils.GZipEncodePage();
In other words multiple Response filters can work together but it depends entirely on the implementation whether they can be chained or in which order they can be chained. In this case running the GZip/Deflate stream filters apparently relies on the original content length of the output and chokes when the content is modified. But if attaching the compression first it works fine as unintuitive as that may seem.
Resources
Download example code
Capture Output from ASP.NET Pages
© Rick Strahl, West Wind Technologies, 2005-2010Posted in ASP.NET
