FTBFS and autopkgtest failure since changes in apt for automatic kernel removal

[Olivier Gayot]

[Impact]
Since the following changes in apt have landed in focal-updates (and is currently in bionic-proposed), unattended-upgrades FTBFS and fails autopkgtest on both series:

  * Revert "Protect currently running kernel at run-time"
  * Backport Determine autoremovable kernels at run-time (LP: #1615381) as of
    2.4.5; including the change to only protect two kernels, not last installed
    one (LP: #1968154)

======================================================================
FAIL: test_remove_unused_dependencies_new_unused_only (__main__.TestRemoveUnused)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_remove_unused.py", line 165, in test_remove_unused_dependencies_new_unused_only
    haystack))
AssertionError: False is not true : Can not find 'Removing unused kernel packages: linux-image-4.05.0-1021-kvm

[Test plan]
1. focal:
  * run autopkgtest against -updates:
    $ autopkgtest unattended-upgrades --apt-upgrade --apt-pocket=updates --test-name run-tests -- <virt-server ...>
  * try to build the package
2. bionic:
  * run autopkgtest against -proposed:
    $ autopkgtest unattended-upgrades --apt-upgrade --apt-pocket=proposed --test-name run-tests -- <virt-server ...>
  * try to build the package

[Where problems could occur]
  * The fix only affects the test-suite (that runs both at build time & autopkgtest time) so the impact should be minimal. Something wrong in the patch would make the unattended package FTBFS or fail autopkgtest but it is already failing ATM.
  * The new tests dependency on apt can trigger more autopkgtest runs. Since unattended-upgrades has a non-trivial autopkgtest suite, this can have a slight impact on the queues.

[Olivier Gayot (ogayot)]

Adding debdiff for bionic

[Olivier Gayot (ogayot)]

Adding debdiff for focal

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "debdiff for bionic" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Julian Andres Klode (juliank)]

looking

[Julian Andres Klode (juliank)]

I ran the tests in lxd, I'm not sure that's sufficient, but:

on bionic:

upgrade-all-security FAIL non-zero exit status 1

focal failed to build for me in autopkgtest:

2022-05-20 10:04:12,150 ERROR Exception happened during upgrade.
Traceback (most recent call last):
  File "/tmp/autopkgtest.7B08MG/build.ZME/src/test/unattended_upgrade.py", line 863, in cache_commit
    res = cache.commit(install_progress=iprogress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 686, in commit
    raise SystemError("installArchives() failed")
SystemError: installArchives() failed
2022-05-20 10:04:12,157 ERROR Auto-removing the packages failed!
2022-05-20 10:04:12,158 ERROR Error message: installArchives() failed
2022-05-20 10:04:12,159 ERROR dpkg returned an error! See ./root.unused-deps/var/log/unattended-upgrades/unattended-upgrades-dpkg.log for details
2022-05-20 10:04:12,159 DEBUG InstCount=0 DelCount=0 BrokenCount=0
2022-05-20 10:04:12,161 DEBUG Extracting content from ./root.unused-deps/var/log/unattended-upgrades/unattended-upgrades-dpkg.log since 2022-05-20 10:04:10

[Robie Basak (racb)]

What's the current status of this please?

Julian's last comment suggests that there's a problem?

I see an upload in the queue for Bionic, but not for Focal. What's the status in Focal?

Is this something that can be staged in proposed, or do you need it in the updates pockets, and if so, why?

[Olivier Gayot (ogayot)]

Hi Robie,

Julian asked the release team to drop the change from focal-proposed because it would fail on production autopkgtest (although the upload did fix the FTBFS).
It requires further investigation for focal (I'm trying to free up some time to work on it).

I'd like Julian to confirm but my gut feeling is that it can stay in -proposed for bionic and be uploaded to -updates when another fix must be delivered. I'm not too familiar with this process myself.

For focal, it's a bit different because unattended-upgrades blocks another SRU: https://bugs.launchpad.net/ubuntu/focal/+source/unattended-upgrades/+bug/1747499

I will try to do what's necessary ASAP.

Thanks,
Olivier

[Olivier Gayot (ogayot)]

I pursued the investigation on this. At the moment I only looked at the failure on focal.
It seems that the .debdiff attached fixes the issue we're trying to address. Unfortunately, there is another (more mysterious) autopkgtest regression that causes the error with installArchives.

This regression has been present since February this year according to autopkgtest reports ; so this is not related to the change in APT for kernel removals.

 2.3ubuntu0.1 migration-reference/0 2022-02-08 19:27:42 UTC 0h 05m 35s - fail log   artifacts   ♻
2.3ubuntu0.1 python-apt/2.0.0ubuntu0.20.04.7 2022-02-08 18:27:10 UTC 0h 03m 18s - fail log   artifacts   ♻

I'm still investigating how to solve this one. I seem to have found a workaround but I'm not happy with it yet.

[Olivier Gayot (ogayot)]

Adding a new debdiff for focal that includes a work around for the autopkgtest failure:

```
  File "/tmp/autopkgtest.BHhHkG/build.sjh/src/test/unattended_upgrade.py", line 863, in cache_commit
    res = cache.commit(install_progress=iprogress)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 686, in commit
    raise SystemError("installArchives() failed")
SystemError: installArchives() failed
Auto-removing the packages failed!
Error message: installArchives() failed
dpkg returned an error! See ./root.unused-deps/var/log/unattended-upgrades/unattended-upgrades-dpkg.log for details
```

when the error occurs, unattended-upgrades-dpkg.log contains the following error:

```
/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import fnmatch, glob, traceback, errno, sys, atexit, locale, imp, stat
apt_pkg.Error: E:Problem closing the gzip file - close (9: Bad file descriptor)

The above exception was the direct cause of the following exception:

SystemError: <built-in method remove of apt_pkg.PackageManager object at 0x7f20c3e894e0> returned a result with an error set
```

During investigation, I observed that the second invocation of test_remove_unused.py (which is executed with python-coverage) would fail if __pycache__/unattended_upgrades.cpython-38.pyc was present.

Removing this __pycache__ file seems to be a good workaround for this bug.

Having said that, I don't have any sensible explanation to why:
 * removing __pycache__ would prevent the error from occurring ;
 * such error would only occur in focal.

This could be caused by a race condition or some sort of undefined behavior (e.g., in unattended-upgrades, in python-apt or in apt itself).

Moving forward, I suggest we continue the investigation to identify the root cause.

[Łukasz Zemczak (sil2100)]

As discussed, let's go with this but, in the meantime, try to figure out the root of the problem anyway.

[Łukasz Zemczak (sil2100)]

Hello Olivier, or anyone else affected,

Accepted unattended-upgrades into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/2.3ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Olivier Gayot (ogayot)]

Tested upload in focal:
Build is successful: https://launchpad.net/ubuntu/+source/unattended-upgrades/2.3ubuntu0.3/+build/24116460
Autopkgtest run is successful: https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/amd64/u/unattended-upgrades/20220627_175649_626c4@/log.gz

[Launchpad Janitor (janitor)]

This bug was fixed in the package unattended-upgrades - 2.3ubuntu0.3

---------------
unattended-upgrades (2.3ubuntu0.3) focal; urgency=medium

  * Fix FTBFS following change in the way apt detects automatically removable
    kernels (LP: #1974223)
  * Work around autopkgtest failure by removing __pycache__ in the test suite
    before the invocation of python-coverage (LP: #1974223)
  * Added new test depends on apt to trigger autopkgtest runs automatically
    when apt is upgraded.

unattended-upgrades (2.3ubuntu0.2) focal; urgency=medium

  [ Brian Murray ]
  * kernel/postinst.d/unattended-upgrades: check to see if canonical-livepatch
    is installed and enabled. In the event that it is and it says a reboot is
    required do not update reboot-required files so that only one reboot
    message appears in the message of the day. (LP: #1747499)

 -- Olivier Gayot <email address hidden> Thu, 19 May 2022 19:38:48 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for unattended-upgrades has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Olivier Gayot (ogayot)]

Based on my tests, the patch 1-1.1ubuntu1.18.04.14__1.1ubuntu1.18.04.15.debdiff for bionic is still relevant.

unattended-upgrades FTBFS in bionic ; and the patch fixes it:

https://launchpad.net/~ogayot/+archive/ubuntu/bionic-proposed/+build/24162373

During autopkgtest run, the error described in comment #9 is not reproduced in the bionic series so there should be no need for clearing __pycache__.

https://autopkgtest.ubuntu.com/results/autopkgtest-bionic-ogayot-bionic-proposed/bionic/amd64/u/unattended-upgrades/20220712_094142_9df41@/log.gz

[Łukasz Zemczak (sil2100)]

Hello Olivier, or anyone else affected,

Accepted unattended-upgrades into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/1.1ubuntu1.18.04.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Julian Andres Klode (juliank)]

Tests have passed on the infra with both old apt and the new apt (I retriggered it with u-u as additional trigger), so this is good to go.

[Chris Halse Rogers (raof)]

Should this be marked as block-proposed-bionic? It looks like there are no user-visible changes, which we'd usually verify and then leave in proposed until it can be folded into a later SRU with user-visible fixes.

[Chris Halse Rogers (raof)]

I'm going to go ahead and mark this as block-proposed-bionic so I don't keep checking this :).

Please ping in #ubuntu-release if this actually needs to be released to users.

[Łukasz Zemczak (sil2100)]

The version of unattended-upgrades in the proposed pocket of Bionic that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.