Double Buffering a Stream

The center piece of our solution is a Type that derives from System.IO.Stream. The EbcdicStream takes as it's input a custom data structure that has been built to model the definition found in a copy book, as well as a source stream. This source stream will typically be a FileStream that points to the raw EBCDIC file that you wish to process into delimited ASCII.

However, it can easily be any type of Stream. Possibilities include chaining a couple streams together in a typical Decorator Pattern to take as input a compressed, EBCDIC file, that exists on a remote FTP server. Instead of downloading the file, decompressing it, and then processing it with the EbcdicStream, you could just chain the streams together so that you only have to make a single pass through the data to get it in its resulting ASCII form.

Now, back on point.

I ran into a problem of sorts with the EbcdicStream and that was it's output was not going to be the same length as the input and it wasn't going to be a predictable multiplier of the input either. So I needed some mechanism to maintain a temporary storage for the overflow of what the user requested to read.

A single buffer might work, but then I would be required to do more reads from the underlying source stream and that could mean more IO (whether they be disk, network, whatever) hits than necessary.

I decided to open up Reflector and check out the CryptoStream to see how Microsoft implemented a similar type of Stream, at least in that the input and output byte counts were different counts. The code was fairly complicated but for all intents and purposes it appears that they are simply managing two buffers, one to read chunks, or blocks in their parlance, of bytes into from the source stream, and one to write chunks out to, then finally they fill up the user's buffer by pulling bytes off this output buffer when they call Read(byte[] buffer, int offset, int count).

I thought that was pretty cool as it presented a smoother operation and one that you could even make configurable so that the user could tune the performance (by changing the desired chunk or block size for their environment).

I decided to go with a similar implementation for the EbcdicStream but instead of implementing inside the EbcdicStream class, I built a DoubleBuffer class that does all the work of managing the transferring of bytes between buffers, managing indexes, etc. It exposes a couple of delegates in the form of Events that allows whatever is using this class to decide how to manipulate the raw bytes from the source stream that would transform them into output. The second delegate provides for a way for the caller to load the input buffer as it sees fit.

This implementation allows me to clearly separate the duty of managing this "double buffer" pattern away from the duty of managing the "transformational" stream.

Tags: software, development, stream, buffer, double buffer