BundleReader cannot stream

[John A Meinel]

Vincent was trying to get rid of our ugly kludge in HTTPTranpsort.get() which wraps the returned socket in a cStringIO.

We needed that because other code expected to be able to seek() on files. Originally, we needed this because gzip.GzipFile() needed to seek() on the files it was decompressing. I believe we changed that with tuned_gzip, and even so, we now use readv() and then wrap parts of the data in a StringIO (rather than all of the data).

The only place that we see (at the moment) that needs seek() capability is in our Bundle code. Specifically, when you call read_bundle_from_url() it starts reading the file (for line in file:) looking for a bundle header. After it finds one, it passes that down to the actual serializer that reads the file.

Our current serializers assume that they can seek(0) to start parsing the file from scratch.

What would seem to be the best, is to update the meta-code to read line by line, and then pass the parsed header down to the serializer so that it doesn't need to seek.