503 errors for Jammy PPAs

[fossfreedom]

For apt-cacher-ng from jammy - all PPAs seem to fail - the main archive is ok

 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
Err:12 http://ppa.launchpad.net/ubuntubudgie-dev/budgie-extras-daily/ubuntu jammy/main amd64 budgie-window-shuffler amd64 1.3.0+202202062047~ubuntu22.04.1
  503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
Err:6 http://ppa.launchpad.net/ubuntubudgie-dev/budgie-extras-daily/ubuntu jammy/main amd64 budgie-rotation-lock-applet all 1.3.0+202202062047~ubuntu22.04.1
  503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
Fetched 114 kB in 7s (15.8 kB/s)
E: Failed to fetch http://ppa.launchpad.net/ubuntubudgie-dev/budgie-extras-daily/ubuntu/pool/main/b/budgie-extras/budgie-quicknote-applet_1.3.0%2b202202062047%7eubuntu22.04.1_amd64.deb 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
E: Failed to fetch http://ppa.launchpad.net/ubuntubudgie-dev/budgie-extras-daily/ubuntu/pool/main/b/budgie-extras/budgie-recentlyused-applet_1.3.0%2b202202062047%7eubuntu22.04.1_amd64.deb 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]

ppa:ubuntubudgie-dev/budgie-extras-daily

The same PPA for impish & focal work just fine

Thoughts?

----

Continually running repeatedly sudo apt update && sudo apt dist-upgrade -y eventually forces the downloads to occur ok and the upgrades can then be installed ok

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: apt 2.3.15
ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12
Uname: Linux 5.15.0-18-generic x86_64
ApportVersion: 2.20.11-0ubuntu76
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: Budgie:GNOME
Date: Mon Feb 7 18:47:47 2022
InstallationDate: Installed on 2022-01-29 (8 days ago)
InstallationMedia: Ubuntu-Budgie 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220129)
SourcePackage: apt
UpgradeStatus: No upgrade log present (probably fresh install)

[Seth Arnold (seth-arnold)]

Are there any log entries in your proxy that might help explain what's happening?

Thanks

[Julian Andres Klode (juliank)]

It seems you use apt-cacher-ng, please try if the issue reproduces without it, and if so, if it happens with a known good proxy like squid (e.g. using deb-squid-proxy).

[fossfreedom (fossfreedom)]

Julian - correct I was using apt-cacher-ng as per sbuild instructions here https://wiki.ubuntu.com/SimpleSbuild

Sorry - I've no idea how to configure squid-deb-proxy with sbuild. So I've deleted my sbuild chroots & disabled using the .mk-sbuildrc proxy. Installing from a ppa no longer shows 503 errors.

[Julian Andres Klode (juliank)]

I don't know if anyone looks at apt-cacher-ng bug reports, but reassigning it there for further analysis.
The log says that this is apt-cacher-ng failing to talk to launchpad.

I generally recommend avoiding apt-cacher-ng due to bugs in the interaction with apt (which then go in endless circles of blame between both sides, but not resolved) and using squid instead if a proxy is needed.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rolf Leggewie (r0lf)]

I can confirm that acng running the version in jammy triggers the problem with PPA in the clients it serves (which may or may not be running jammy). Going by the fact that the version from impish when installed on jammy does not have this problem and runs fine otherwise, it looks like a regression.

Maybe it's best to downgrade the version in jammy back to the one from impish until a solution can be found. I certainly would not like to see an LTS see the light of day with this regression.

[Rolf Leggewie (r0lf)]

The problem disappeared after installing the version in impish. More importantly, it hasn't come back after reinstalling the version from jammy.

@fossfreedom, did you upgrade the system where acng is running from focal as well? Can you provide logs of the problem?

[fossfreedom (fossfreedom)]

Mine was a fresh install of jammy with it installed from the repo.

On Sun, 27 Feb 2022, 11:10 Rolf Leggewie, <email address hidden>
wrote:

> The problem disappeared after installing the version in impish. More
> importantly, it hasn't come back after reinstalling the version from
> jammy.
>
> @fossfreedom, did you upgrade the system where acng is running from
> focal as well? Can you provide logs of the problem?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1960264
>
> Title:
> 503 errors for Jammy PPAs
>
> Status in apt-cacher-ng package in Ubuntu:
> Confirmed
>
> Bug description:
> For apt-cacher-ng from jammy - all PPAs seem to fail - the main
> archive is ok
>
> 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
> Err:12
> http://ppa.launchpad.net/ubuntubudgie-dev/budgie-extras-daily/ubuntu
> jammy/main amd64 budgie-window-shuffler amd64
> 1.3.0+202202062047~ubuntu22.04.1
> 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
> Err:6
> http://ppa.launchpad.net/ubuntubudgie-dev/budgie-extras-daily/ubuntu
> jammy/main amd64 budgie-rotation-lock-applet all
> 1.3.0+202202062047~ubuntu22.04.1
> 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
> Fetched 114 kB in 7s (15.8 kB/s)
> E: Failed to fetch
> http://ppa.launchpad.net/ubuntubudgie-dev/budgie-extras-daily/ubuntu/pool/main/b/budgie-extras/budgie-quicknote-applet_1.3.0%2b202202062047%7eubuntu22.04.1_amd64.deb
> 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
> E: Failed to fetch
> http://ppa.launchpad.net/ubuntubudgie-dev/budgie-extras-daily/ubuntu/pool/main/b/budgie-extras/budgie-recentlyused-applet_1.3.0%2b202202062047%7eubuntu22.04.1_amd64.deb
> 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
>
> ppa:ubuntubudgie-dev/budgie-extras-daily
>
> The same PPA for impish & focal work just fine
>
> Thoughts?
>
> ----
>
> Continually running repeatedly sudo apt update && sudo apt dist-
> upgrade -y eventually forces the downloads to occur ok and the
> upgrades can then be installed ok
>
> ProblemType: Bug
> DistroRelease: Ubuntu 22.04
> Package: apt 2.3.15
> ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12
> Uname: Linux 5.15.0-18-generic x86_64
> ApportVersion: 2.20.11-0ubuntu76
> Architecture: amd64
> CasperMD5CheckResult: pass
> CurrentDesktop: Budgie:GNOME
> Date: Mon Feb 7 18:47:47 2022
> InstallationDate: Installed on 2022-01-29 (8 days ago)
> InstallationMedia: Ubuntu-Budgie 22.04 LTS "Jammy Jellyfish" - Alpha
> amd64 (20220129)
> SourcePackage: apt
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/1960264/+subscriptions
>
>

[Rolf Leggewie (r0lf)]

@fossfreedom: Thank you for the quick response. I assume you can still reproduce the problem, right? It would be great to see logs, for example /var/log/apt-cacher-ng/apt-cacher.err Would you be so kind to provide those?

[Ricardo Pardini (rpardini)]

Same here. Using Jammy's 3.7.4-1build1 results in "503" errors even when downloading from main archive. Seems to happen only when used with apt in http_proxy mode (when using with localhost:3142/xxx prefix, eg, for debootstrap, that works).

Reverted to debian/3.6.4-1 from salsa and all goes back to normal.

[Chow Loong Jin (hyperair)]

I'm having the same issue, and /var/log/apt-cacher-ng/apt-cacher.err is empty. The only related lines I see are in /var/log/apt-cacher-ng/apt-cacher.log, which look like:

1657167296|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/wireplumber-upstream/ubuntu/pool/main/w/wireplumber/wireplumber_0.4.11-1~ubuntu22.04_amd64.deb [HTTP error, code: 503]
1657167296|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/wireplumber-upstream/ubuntu/pool/main/w/wireplumber/libwireplumber-0.4-0_0.4.11-1~ubuntu22.04_amd64.deb [HTTP error, code: 503]
1657167297|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/pipewire-upstream/ubuntu/pool/main/p/pipewire/pipewire-locales_0.3.53.r23.ge1cb7c6-1~ubuntu22.04_all.deb [HTTP error, code: 503]
1657167297|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/wireplumber-upstream/ubuntu/pool/main/w/wireplumber/wireplumber_0.4.11-1~ubuntu22.04_amd64.deb [HTTP error, code: 503]
1657167297|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/wireplumber-upstream/ubuntu/pool/main/w/wireplumber/libwireplumber-0.4-0_0.4.11-1~ubuntu22.04_amd64.deb [HTTP error, code: 503]
1657167299|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/pipewire-upstream/ubuntu/pool/main/p/pipewire/pipewire-locales_0.3.53.r23.ge1cb7c6-1~ubuntu22.04_all.deb [HTTP error, code: 503]
1657167299|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/wireplumber-upstream/ubuntu/pool/main/w/wireplumber/wireplumber_0.4.11-1~ubuntu22.04_amd64.deb [HTTP error, code: 503]
1657167299|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/wireplumber-upstream/ubuntu/pool/main/w/wireplumber/libwireplumber-0.4-0_0.4.11-1~ubuntu22.04_amd64.deb [HTTP error, code: 503]
1657167303|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/pipewire-upstream/ubuntu/pool/main/p/pipewire/pipewire-locales_0.3.53.r23.ge1cb7c6-1~ubuntu22.04_all.deb [HTTP error, code: 503]
1657167303|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/wireplumber-upstream/ubuntu/pool/main/w/wireplumber/wireplumber_0.4.11-1~ubuntu22.04_amd64.deb [HTTP error, code: 503]
1657167303|E|697|127.0.0.1|ppa.launchpad.net/pipewire-debian/wireplumber-upstream/ubuntu/pool/main/w/wireplumber/libwireplumber-0.4-0_0.4.11-1~ubuntu22.04_amd64.deb [HTTP error, code: 503]
1657167303|E|1394|127.0.0.1|ppa.launchpad.net/pipewire-debian/pipewire-upstream/ubuntu/pool/main/p/pipewire/pipewire-locales_0.3.53.r23.ge1cb7c6-1~ubuntu22.04_all.deb [HTTP error, code: 503]

[Chow Loong Jin (hyperair)]

Retrying apt upgrade multiple times seems to at least allow the upgrade to complete, so it's a pretty transient error.

[Kain (kain-kain)]

You can work around this by redirecting HTTP PPAs to HTTPS PPAs with a remap directive like the following -- maybe theres some interaction with the ppa severs and/or its HTTP backend involving pipelining that doesnt happen running through a SSL reverse proxy or whatever launchpadcontent is served over?

Remap-ubuntuppa: ppa.launchpad.net ; https://ppa.launchpadcontent.net/

[Sebastian Marsching (sebastian-marsching)]

I just wanted to add that this bug does NOT only occur with PPAs. We frequently experienced it when downloading (many) package files from the regular Ubuntu repositories.

It seems like this bug got introduced in Apt-Cacher NG 3.7.x, because switching our Apt-Proxy to Debian 11 (which has Apt-Cacher NG 3.6.4) fixed the problem for us.

[Brian Candler (b-candler)]

I've just seen the same bug with jammy and apt-cacher-ng 3.7.4-1build1

A simple "sudo systemctl restart apt-cacher-ng" fixed it.

========

First try:

$ sudo apt-get update
...
Reading package lists... Done
E: Failed to fetch http://gb.archive.ubuntu.com/ubuntu/dists/jammy-security/restricted/cnf/Commands-amd64 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
W: Failed to fetch http://gb.archive.ubuntu.com/ubuntu/dists/jammy-security/universe/binary-amd64/Packages.gz 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
W: Failed to fetch http://gb.archive.ubuntu.com/ubuntu/dists/jammy-security/universe/i18n/Translation-en.gz 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
W: Failed to fetch http://gb.archive.ubuntu.com/ubuntu/dists/jammy-security/multiverse/binary-amd64/Packages.gz 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
W: Some index files failed to download. They have been ignored, or old ones used instead.

Then:

$ sudo systemctl restart apt-cacher-ng
$ sudo apt-get update
...
Reading package lists... Done
$

[Brian Candler (b-candler)]

Spoke a bit too soon: on the first attempt after updating the package index, "sudo apt-get dist-upgrade" failed to download the packages, with the same 503 error.

...
Fetched 17.0 MB in 8s (2,216 kB/s)
E: Failed to fetch http://gb.archive.ubuntu.com/ubuntu/pool/main/n/netplan.io/netplan.io_0.105-0ubuntu2%7e22.04.3_amd64.deb 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
E: Failed to fetch http://gb.archive.ubuntu.com/ubuntu/pool/main/n/netplan.io/libnetplan0_0.105-0ubuntu2%7e22.04.3_amd64.deb 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
...

But on second attempt it was fine.

So it seems to be intermittent, as described in the Debian bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1031283

[ooshlablu (ooshlablu)]

I want to add that I have experienced this issue with every repo source I have run through apt-cacher-ng 3.7.4-1build1. This includes caching for Raspbian repos, base Ubuntu repos for both 20.04 and 22.04, and PPAs. With the PPA's, I am guaranteed to hit the issue if I have to download more than 2 packages from any given PPA. For the Ubuntu and Raspian repos, if an installation or update requires more than 10 packages to be installed the issue pops up.

As an example, I installed "flowblade" last night, and it took several "apt install flowblade" to get past the dependency install (it has a bunch of deps). When the package is fully installed a machine, and therefore the deps are fully cached in apt-cacher-ng, this issue doesn't happen. It's only on the first pull.

[Ken Sharp (kennybobs)]

I set "DlMaxRetries: 20" in acng.conf as a workaround.

[Chow Loong Jin (hyperair)]

I have "DlMaxRetries: 30" in acng.conf and it's still happening for me.

[Seth Arnold (seth-arnold)]

Hello Chow Loong Jin, I suspect you'll have more success with squid-deb-proxy.

Thanks

[Mate Kukri (mkukri)]

I get similar errors for some things in the Jammy archive too. Disabling apt-cacher-ng resolves everything.

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libp/libpackage-stash-perl/libpackage-stash-perl_0.39-1_all.deb 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libs/libsub-identify-perl/libsub-identify-perl_0.14-1build5_amd64.deb 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libs/libsub-name-perl/libsub-name-perl_0.26-1build3_amd64.deb 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libn/libnamespace-clean-perl/libnamespace-clean-perl_0.27-1_all.deb 503 Connection closed, check DlMaxRetries [IP: 127.0.0.1 3142]
E: Fetched 2269 kB in 7s (304 kB/s)

[Ken Sharp (kennybobs)]

Indeed I have found I need to increase the retries dramatically on occasion (when running a debootstrap) to "DlMaxRetries: 200".