Intermittent BadStatusLine('Hello world!HTTP/1.1 200 OK\r\n') when executing test-suite

[Olivier Gayot]

When executing the test suite, sometimes the following exception gets raised:

E requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('Hello world!HTTP/1.1 200 OK\r\n'))

This looks bad. According to the test definition, the string "Hello world!" is supposed to be the body of the HTTP response. And somehow it appears before the status line.

We noticed this issue after uploading 0.3.0-3ubuntu1 because the package FTBFS [1] (three times out three). So Launchpad builders seem very good at reproducing the issue (that said my PPA build was green). That being said, the issue seems unrelated to the patch included in 0.3.0-3ubuntu1.

I've been able to reproduce the exception without the patch, and also on a mantic system.

Steps to reproduce:

$ git ubuntu clone python-requests-unixsocket
$ cd python-requests-unixsocket
$ while python3 -m pytest requests_unixsocket/tests; do true; done

It can take many iterations until the issue occurs. At this point I don't know yet if the issue is specific to requests-unixsocket or if it is a src:requests bug instead (or maybe something else like python-waitress?).

[1] https://launchpadlibrarian.net/714333527/buildlog_ubuntu-noble-amd64.python-requests-unixsocket_0.3.0-3ubuntu1_BUILDING.txt.gz

[Dan Bungert (dbungert)]

After debugging a bit, I speculate that what is happening the test send several requests to one socket, the HEAD request may not read the body (why would it?), then unread bytes (why does it return a body from a HEAD request?) look like an out-of-order write.

[Olivier Gayot (ogayot)]

Great job finding out the root cause! I think your theory is right.

If the client picks up all the data (i.e., headers + body) in a single call to recv(2), I think the excess data is simply ignored by the library. On the other hand, if the call to recv only returns the headers, then the next call to recv will have the excess data prepended. It's SOCK_STREAM based so anything can happen depending on timing I guess :)

Note that this is also based on speculation.

[Dan Bungert (dbungert)]

Ok, I feel good enough about this to upload.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-requests-unixsocket - 0.3.0-3ubuntu2

---------------
python-requests-unixsocket (0.3.0-3ubuntu2) noble; urgency=medium

  * Fix test flakiness due to unread data (LP: #2053154)
    - d/p/0002-fix-HEAD-method-tests.patch

 -- Dan Bungert <email address hidden> Thu, 15 Feb 2024 00:53:56 -0700