Buffered Writes with httplib

I was wresting pretty vigorously with trying to fix a bug in pybrightcove for past several days. The problem was occurring when trying to send a very large file (>2GB). I was getting out of memory exceptions on the server sporadically and when I didn't get them, it was another exception that said something about blowing the upper limit on a Python string.

That led me to investigate the code I that I previously borrowed from another project I found that was doing multipart form encode POSTs. It was using urllib2 to send the data, which meant that it had to load all the data up into memory before sending it across the wire.

This seemed pretty bad to me and I thought for sure that there had to be a more reasonable approach to sending data -- an approach that I have used many times before would be to simply do a buffered read/write. However, for the life of me I couldn't figure out how to do this with the classes that I found available to me within the urllib2 framework. And the stuff I did start considering (writing my own Handler and/or subclasses the Request object), seemed unnatural and fighting the library instead of working with it.

After taking a step back, I realized that the Python based Google Data client, written and published by Google, was capable of sending >2GB files via its API and it's client was pure Python. So I downloaded the code and started tinkering with how to make use of the http_core module found in the atom framework.

I copied over the module to pybrightcove as it seemed pretty independent (it only relies on core Python modules). Right away I noticed that it used a more lower level framework within Python called httplib, allowing it to build up the request and do a buffered read/write for file data.

I had to modify it quite a bit to get it to suit my needs with pybrightcove since the multi-part body needed to be formatted slightly different for the Brightcove Media API. However, after about an hour of inspecting POST bodies with Charles (highly recommended, by the way), I got things working.

So, with this addition, I am calling this v1.0 for the pybrightcove library.