Some files are never downloaded (while under test)

[Barry Warsaw]

I'm trying to debug the intermittent timeout errors I'm seeing in s-i when integrated with u-d-m. A TimeoutError means that s-i does not see the 'finished' signal in response to a successful group download (of course, a canceled or error signal would also terminate the s-i client reactor, but I don't see those either).

I added some debugging to my test http/https server to log the requests. Using this set up I captured the output during one of the timed out tests, and what I saw was this (excerpt):

127.0.0.1 - - [04/Feb/2014 16:33:28] "GET /stable/nexus7/index.json HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2014 16:33:28] "GET /stable/nexus7/index.json.asc HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2014 16:33:28] "GET /3/4/5.txt HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2014 16:33:28] "GET /3/4/5.txt.asc HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2014 16:33:28] "GET /4/5/6.txt HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2014 16:33:28] "GET /4/5/6.txt.asc HTTP/1.1" 200 -
127.0.0.1 - - [04/Feb/2014 16:33:28] "GET /5/6/7.txt HTTP/1.1" 200 -
ERROR

What's interesting about this is that I expected /5/6/7.txt.asc to also be downloaded, but it never was. IOW, the GET for 7.txt.asc was never seen by the http server. This leads me to think that something is happening to cause u-d-m to not process the entire group download, or to hang before it requests 7.txt.asc. If it were hanging or not completing the group download, then it would make sense that s-i would never see the 'finished' signal that would terminate the reactor.

Unfortunately, due to previously reported bugs, it's nearly impossible to directly correlate this output with u-d-m output so that I can tell what u-d-m is doing while the s-i http server is expected to see one more file download.

[Barry Warsaw (barry)]

I duped this to LP: #1329681 which we'll use to track general timeouts in udm.