Buffered Writes with httplib
October 12, 2009
That led me to investigate the code I that I previously borrowed from another project I found that was doing multipart form encode POSTs. It was using urllib2 to send the data, which meant that it had to load all the data up into memory before sending it across the wire.
This seemed pretty bad to me and I thought for sure that there had to be a more reasonable approach to sending data -- an approach that I have used many times before would be to simply do a buffered read/write. However, for the life of me I couldn't figure out how to do this with the classes that I found available to me within the urllib2 framework. And the stuff I did start considering (writing my own Handler and/or subclasses the Request object), seemed unnatural and fighting the library instead of working with it.
After taking a step back, I realized that the Python based Google
client, written and published
by Google, was capable of sending >2GB files via its API and it's client was
pure Python. So I downloaded the code and started tinkering with how to make
use of the
http_core module found in the
I copied over the module to pybrightcove as it seemed pretty independent (it
only relies on core Python modules). Right away I noticed that it used a more
lower level framework within Python called
httplib, allowing it to build up
the request and do a buffered read/write for file data.
I had to modify it quite a bit to get it to suit my needs with
since the multi-part body needed to be formatted slightly different for the
Brightcove Media API. However, after about an hour of inspecting POST bodies
with Charles (highly recommended, by the way),
I got things working.
So, with this addition, I am calling this v1.0 for the