Automatically retry downloads on failures

[Haw Loeung]

Hi,

We operate archive.ubuntu.com as well as mirrors of it in various places. We're getting reports of various CI/CD jobs failing. Sometimes due to routing issues, issues with overloaded servers/VMs hosting archive, etc.

Any chance we can make '-o Acquire::Retries=3' the default for 'apt-get update/install/dist-upgrade'? Preferably with a randomised skew between retries.

Also, any chance of having apt-get retry with different IPs/hosts returned from the DNS lookup? (e.g. for archive.ubuntu.com - 91.189.88.142, 91.189.88.152, ...).

Thanks.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Julian Andres Klode (juliank)]

A few points to think about:

1. I am not aware of anyone using that option, we don't know if it works and how it behaves

2. Given 1, what kind of testing can we do? This needs to be thoroughly tested, given that it's not been tested at all.

3. Switching IPs might be useful, but it's not clear to me yet how that would work in practice. We can basically blacklist certain IPs so it jumps to the next one, but we need to make sure we don't run out of IPs I guess. Also we use one connection per hostname, so each failure would switch the connection to a different IP address for all following requests.

4. Does it have any kind of delay right now? I don't think it does, and it sounds hard, and also pointless if we switch IPs.

5. I'd like to retry across different host names too.

[Julian Andres Klode (juliank)]

Summary is that I don't believe the option as is is sufficient, and needs to be reworked, so I'm changing the title accordingly.

[Julian Andres Klode (juliank)]

As a compromise, and given that I ripped of the previous faulty http-internal retry code, I think we do need to go with Retries set to 2 for hirsute, and worry about improvements later, because while I appreciate all the bugs I get for failures and the chance to fix them, they also break people's workflows.

[Julian Andres Klode (juliank)]

The basic "turn the flag on" is landing in 2.3.2, I'll open other bug reports for the backoff and server switching things I suppose.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.3.3

---------------
apt (2.3.3) experimental; urgency=medium

  [ David Kalnischkies ]
  * Allow superfluous commas in build-dependency lines
  * Mark only provides from protected versioned kernel packages
  * Count uninstallable packages in "not upgraded" (Closes: #981535)
  * Reexplore providers of marked packages if some didn't satisfy before
  * Call MarkAndSweep only manually in apt-get for autoremove
  * Store versioned kernel package detectors in d-pointer

  [ Brian Murray ]
  * Fix a typo in json-hooks-protocol.md

  [ Julian Andres Klode ]
  * Temporarily Revert "2.3-only: Warn that the 0.1 protocol is deprecated"

 -- Julian Andres Klode <email address hidden> Thu, 29 Apr 2021 10:43:19 +0200