"apt failed to download packages, retrying in 10s..." results in " Unable to acquire the dpkg frontend lock"

[Tim Andersson]

logfile:
https://objectstorage.prodstack5.canonical.com/swift/v1/AUTH_0f9aae918d5b4744bf7b827671c86842/autopkgtest-noble/noble/armhf/b/balsa/20240404_081820_ad372@/log.gz

pertinent log messages:
```
883s Get:204 http://ftpmaster.internal/ubuntu noble-proposed/universe armhf xvfb armhf 2:21.1.11-2ubuntu2 [738 kB]
935s autopkgtest: WARNING: apt failed to download packages, retrying in 10s...
954s E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 763 (apt-get)
954s E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
955s autopkgtest: WARNING: Test dependencies are unsatisfiable - calling apt install on test deps directly for further data about failing dependencies in test logs
957s Reading package lists...
958s Building dependency tree...
```

It perhaps looks like in a setup script, whilst sleeping before a retry, another process starts an apt-get command which causes the script to fail after the sleep. Or something along those lines.

[Paride Legovini (paride)]

"the setup script" is autopkgtest installing test dependencies. The question is: what else is running apt-get? Do we have unattended-upgrades installed on the (armhf) images? That should be removed by setup-testbed, but we should check.

[Paride Legovini (paride)]

This is also one case where we have the auto fallback to all-proposed:

869s autopkgtest: WARNING: Test dependencies are unsatisfiable with using apt pinning. Retrying with using all packages from noble-proposed

which I'm not sure is desirable.

[Tim Andersson (andersson123)]

I'm looking into this.

[Tim Andersson (andersson123)]

So unattended upgrades are disabled here:
https://git.launchpad.net/autopkgtest-cloud/tree/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker-config-production/setup-canonical.sh#n65

but we don't use this script in the setup for lxd workers

I'll propose a fix

[Tim Andersson (andersson123)]

MP attached to bug.

[Paride Legovini (paride)]

The setup-testbed script is called by autopkgtest-build-lxd.

[Tim Andersson (andersson123)]

so, @paride , I checked and unattended upgrades are definitely not installed on our lxc runners:
```
root@autopkgtest-lxd-xxkcjb:~# apt list --installed | grep upgrade

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

root@autopkgtest-lxd-xxkcj
```
and I can see it removed in image creation:
```
Removing unattended-upgrades (2.9.1+nmu4ubuntu1) ...
```
However, I see this failure in prod:
```
Purging configuration files for unattended-upgrades (2.9.1+nmu4ubuntu1) ...
dpkg: warning: while removing unattended-upgrades, directory '/var/log/unattended-upgrades' not empty so not removed
```
I created my own image with autopkgtest-build-lxd.

But that seems to be fine, I boot a container from this image, and unattended-upgrades is 100% not installed.

So, the issue isn't unattended-upgrades. What else could be running apt-get in the background?

[Tim Andersson (andersson123)]

Here's the list of packages installed on the runners:
https://pastebin.canonical.com/p/ZQSX6CjvC4/

[Paride Legovini (paride)]

Thanks for checking. I see nothing obvious.

Is this a failure more we do see from time to time, or was it a one-off thing?

[Paride Legovini (paride)]

*maybe* ubuntu-pro-client?

[Tim Andersson (andersson123)]

I'm not sure if it's something we see often or not. I've only seen it this one time, though I can check to see if it's something we see in lots of logs and get back to you on this bug

[Paride Legovini (paride)]

I think this is a sensible approach.

[Launchpad Janitor (janitor)]

[Expired for Auto Package Testing because there has been no activity for 60 days.]

[Tim Andersson (andersson123)]

as of June 12th, this has occurred 44 times in the last 30 days (armhf only, obviously)

[Tim Andersson (andersson123)]

I set the limit to checking 10000 tests a day, which, if hit, would mean it's:
0.00014666666666666666%

Though I doubt we actually do 10000 tests a day on armhf

[Tim Andersson (andersson123)]

especially recently with the bos03 issues