Comment 5 for bug 2003851

Got some more pcaps. I can also reproduce by sending a 503 after a 200, without aborting during/before body transfer (*).

============================
server.py logs
============================
13:29:55 [3666956] client said: b'GET /ubuntu/dists/jammy/InRelease HTTP/1.1\r\nHost: junk.devs.nu:3851\r\nCache-Control: max-age=0\r\nAccept: text/*\r\nUser-Agent: Debian APT-HTTP/1.3 (2.4.9) non-interactive\r\n\r\n'
13:29:55 [3666956] we say: 200 and serve local InRelease
13:29:55 [3666956] client said: b'GET /ubuntu/dists/jammy-updates/InRelease HTTP/1.1\r\nHost: junk.devs.nu:3851\r\nCache-Control: max-age=0\r\nAccept: text/*\r\nUser-Agent: Debian APT-HTTP/1.3 (2.4.9) non-interactive\r\n\r\n'
13:29:55 [3666956] we say: 503

============================
503 request/response (valid length, no early FIN)
============================
GET /ubuntu/dists/jammy-updates/InRelease HTTP/1.1
Host: junk.devs.nu:3851
Cache-Control: max-age=0
Accept: text/*
User-Agent: Debian APT-HTTP/1.3 (2.4.9) non-interactive

HTTP/1.1 503 OK
date: Fri, 17 Feb 2023 15:11:49 GMT
content-type: text/plain
content-length: 19

something is broken

============================

I present to you:

- https://junk.devs.nu/2023/lp2003851/not-reproduced-port-34634.txt
- https://junk.devs.nu/2023/lp2003851/not-reproduced-port-34634.pcap
- https://junk.devs.nu/2023/lp2003851/reproduced-port-58014.txt
- https://junk.devs.nu/2023/lp2003851/reproduced-port-58014.pcap

In the not-reproduced case, the error is acknowledged and the download is retried. Then it fails quickly.

In the reproduced case, the error is not acknowledged, and the result is a hanging apt-get. Logs ending with:

Hit:3 http://repo.zabbix.com/zabbix/6.2/ubuntu jammy InRelease
0% [Working]

============================
(*) observed in the wild with these req/resp:
============================
GET /ubuntu/dists/jammy-backports/InRelease HTTP/1.1
Host: apt.osso.nl
Cache-Control: max-age=0
Accept: text/*
If-Modified-Since: Tue, 21 Feb 2023 19:32:00 GMT
User-Agent: Debian APT-HTTP/1.3 (2.4.8) non-interactive

HTTP/1.1 200 OK
date: Wed, 22 Feb 2023 02:36:08 GMT
content-type: octet/stream
content-length: 106807
last-modified: Wed, 22 Feb 2023 01:49:00 GMT
x-original-source: http://nl.archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease
(+ valid body)

GET /ubuntu-security/dists/jammy-security/InRelease HTTP/1.1
Host: apt.osso.nl
Cache-Control: max-age=0
Accept: text/*
If-Modified-Since: Tue, 21 Feb 2023 23:11:00 GMT
User-Agent: Debian APT-HTTP/1.3 (2.4.8) non-interactive

HTTP/1.1 503 Connection timeout
date: Wed, 22 Feb 2023 02:36:08 GMT
content-type: text/html
content-length: 486
(+ valid body)

============================
extra contemplations:
============================
We've only recently added the /ubuntu-security/ (extra path, on the same server). Maybe that is the reason why this is biting us right now.