Comment 8 for bug 1240652

I am pretty sure I understand this bug.
It is a regression caused by changes for bug 1237658.

Under that bug we made each attempt in a UrlMirrorReader basically stat urls by doing an open and then a close to see if they were there. Then, if it wasn't there it would retry with a '/' added to the prefix.

This was all so the user could do:
 sstream-query --keyring /usr/share/keyrings/ubuntu-cloudimage-keyring.gpg -vvv http://cloud-images.ubuntu.com/releases
and have us figure out that they really needed a trailing /

I think the bug is actually lower in the stack somewhere. We were essentially doing:
    fp = open(someurl)
   fp.close()
   return open(someurl)

which should be ok, but I think that the quick open and close was either not correctly getting closed or was ending up triggering squid to do the full download before it started handing us data for the second open (which would then subsequently time out).

python3 seems less likely to show failure. I could not reproduce with python3 via simplestreams, but did see it using the go-requests.py program with both python2 and python3.

I have several test programs here and will attach. go-urllib2 does not show the problem. The others do.
  go-requests.py: use python-requests
  go-sstream-mirror: use UrlMirrorReader from simplestreams
  go-sstreams: use UrlContentSource without the mirror reader
  go-urllib.py

Running them is straight forward enough:
  time env http_proxy=$PROXY python3 ../go-sstreams.py http://maas.ubuntu.com/images/ephemeral/daily/precise/20131017/precise-daily-maas-amd64.tar.gz out.img