apt fails to properly handle server-side connection closure

[Julian Andres Klode]

[Impact]
In some cases, apt does not correctly handle server-side connection closure after a pipeline, and aborts the file being downloaded with an "Undetermined Error" when the connection has been closed.

[Test case]
This could be seen by running apt build-dep evince on cosmic with a recent apt with the pipelining fix (such as 1.6.6) against a local mirror running apache from trusty. It remains to be seen whether this is easily reproducible for anyone.

[Regression potential]
N/A yet.

[Julian Andres Klode (juliank)]

It's unclear if xenial's apt is affected or not, we did not get to testing that yet.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 1.8.0~alpha2ubuntu1

---------------
apt (1.8.0~alpha2ubuntu1) disco; urgency=medium

  * Adjust libapt-pkg Breaks aptitude to << 0.8.9

 -- Julian Andres Klode <email address hidden> Wed, 14 Nov 2018 12:07:59 +0100

[Julian Andres Klode (juliank)]

It actually did not fix the issue.

[John Paul Adrian Glaubitz (glaubitz)]

And it made multiple buildds hang for days :-(. I just only noticed it now.

[Julian Andres Klode (juliank)]

Yes, exactly, that's why it was removed from disco-proposed and I quickly reverted it in the followup upload. Though, the problem only seems to happen with apache based servers, if you put a squid in front of them, you're fine.

[Ian Childress (ianchildress)]

This can be reproduced by hosting a mirror on S3. S3 has built in functionality that closes a connection after 100 requests. We were able to add comments to apt source that output every time a new connection was made and we confirmed every 100 requests that S3 did close the connection. Intermittently, we would get the Undetermined Error during one of the connection close events. Debug output confirmed there is a connection: close header. We also confirmed apt identified a 102 connection reset by peer event by printing the error.

So this isn't apache specific, I believe apache is just more prone to sending connection: close. Because of S3's behavior of closing connections every 100 requests, it became very noticeable to us.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robert Sachunsky (bertsky)]

Affects me on Bionic, too.

Does anyone know a workaround already? Downgrading apt, or adding some apt.conf `Acquire::http` settings?

[MDE (mde-bs)]

Yes, it seems to have something with pipelining, so I disabled it with the following configuration:

    Acquire {
       http::Pipeline-Depth "0";
   };

I have some machines updating with this setting for 2 days without errors now. The performance impact seems insignificant (I have low RTT connections).
I also had the error occur on official mirrors, local mirrors with apache, and local mirrors with lighty before this.

I hope this helps fixing the issue.

[Julian Andres Klode (juliank)]

FWIW, I've not seen this bug in over a year, I totally forgot about it.

[Julian Andres Klode (juliank)]

I don't even know how I setup the reproducer last year. I think I mirrored main in a trusty container and put an apache in front of it, but not sure

[Robert Liu (robertliu)]

I met the issue when installing packages from a private PPA of Launchpad. Hope this would be helpful to reproduce the issue.

[Julian Andres Klode (juliank)]

I can't reproduce this anymore, so closing it.

[Julian Andres Klode (juliank)]

This may be fixed now even.