apt update crashes with "KeyError: 'suite'" in cnf-update-db if index compression is enabled

[Michael Sparmann]

[Impact]
On all Ubuntu releases since at least 19.10, enabling compressed APT package list storage with command-not-found installed leads to this crash in the cnf-update-db hook called by apt update:

Traceback (most recent call last):
  File "/usr/lib/cnf-update-db", line 26, in <module>
    col.create(db)
  File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 93, in create
    self._fill_commands(con)
  File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 127, in _fill_commands
    self._parse_single_commands_file(con, fp)
  File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 165, in _parse_single_commands_file
    suite=tagf.section["suite"]
KeyError: 'suite'

(To enable compressed APT list storage, add Acquire::GzipIndexes "true"; to an apt.conf fragment.)

This appears to be caused by apt downloading an lz4-compressed command list for command-not-found and cnf-update-db passing the /var/lib/apt/lists/*Commands-*.lz4 files to db/creator.py, which does not know about the compression and attempts to parse them as plain text.

[Test plan]
Enable the option, purge old lists, and update to populate lists directory with compressed files; this will run cnf-update-db for you.

[Where problems could occur]
Replacing the open() call with subprocess.Popen for apt-helper seems like a minimal change. If apt-helper has problems, we check the return code and fail, keeping the old db in place.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[sparky3387 (sparky3387)]

As a temporary workaround, you can add to an apt.conf file:
Acquire::GzipIndexes "false";

And do a check to ensure no files have GzipIndexes:
grep -r GzipIndexes /etc/apt

And then delete apt list cache
rm -rf /var/lib/apt/lists

And then finally update
apt-get update

[Michael Sparmann (theseven)]

I'm aware of that, but that workaround defeats the point of enabling compressed list storage.
We're using the patch linked above to solve the real issue without that space overhead, but such local fixes are a maintenance issue. Sadly there appears to be no intention to accept the fix upstream.

[Michael Sparmann (theseven)]

Actually, upon a closer look, it appears like command-not-found has been entirely unmaintained for more than 2 years now, so efforts to get this fixed upstream, let alone backported to focal, are probably futile.
We should probably just remove this package from our base image and let the end user install some fixed fork of it from a PPA, if they need it. Luckily there's no hard dependency to it from ubuntu-standard.

[Anatoly Borodin (anatoly.borodin)]

Standard Docker images contain a file /etc/apt/apt.conf.d/docker-gzip-indexes with

Acquire::GzipIndexes "true"; Acquire::CompressionTypes::Order:: "gz";

in it. So c-n-f is broken there.

[Julian Andres Klode (juliank)]

FWIW, Docker images do not install command-not-found and are intended for machine users, so it's not a place where you'd install it and notice that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package command-not-found - 21.10.0

---------------
command-not-found (21.10.0) impish; urgency=medium

  * Correctly handle compressed APT index files (LP: #1876034)

 -- Julian Andres Klode <email address hidden> Thu, 23 Sep 2021 15:32:30 +0200

[Brian Murray (brian-murray)]

Hello Michael, or anyone else affected,

Accepted command-not-found into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/command-not-found/20.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

The upload in the hirsute queue shows multiple changes outside of the fix for compressed files in db/creator.py, including extensive changes to the debian/ directory. Have these been included by mistake?

[Brian Murray (brian-murray)]

With regards to the multiple changes the Launchpad diff was incorrect in that it was comparing a version of the package that is not in Hirsute.

[Brian Murray (brian-murray)]

Hello Michael, or anyone else affected,

Accepted command-not-found into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/command-not-found/20.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[John Freeman (john-freeman)]

The fix works for me. It's relatively easy to check for yourself with Docker.

To reproduce the issue (version 20.04.4):
docker run --rm ubuntu:focal bash -c 'export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get -y install --no-install-recommends command-not-found && apt-get update ; apt-cache policy command-not-found'

To test the fix (version 20.04.5):
docker run --rm ubuntu:focal bash -c 'echo "deb http://archive.ubuntu.com/ubuntu/ focal-proposed restricted main multiverse universe" > /etc/apt/sources.list.d/ubuntu-focal-proposed.list && export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get -y install --no-install-recommends command-not-found && apt-get update ; apt-cache policy command-not-found'

[John Freeman (john-freeman)]

I checked and the fix also works on Hirsute.

To reproduce the issue (version 20.10.1):
docker run --rm ubuntu:hirsute bash -c 'export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get -y install --no-install-recommends command-not-found && apt-get update ; apt-cache policy command-not-found'

To test the fix (version 20.10.2):
docker run --rm ubuntu:hirsute bash -c 'echo "deb http://archive.ubuntu.com/ubuntu/ hirsute-proposed restricted main multiverse universe" > /etc/apt/sources.list.d/ubuntu-hirsute-proposed.list && export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get -y install --no-install-recommends command-not-found && apt-get update ; apt-cache policy command-not-found'

[Launchpad Janitor (janitor)]

This bug was fixed in the package command-not-found - 20.04.5

---------------
command-not-found (20.04.5) focal; urgency=medium

  * Correctly handle compressed APT index files (LP: #1876034)

 -- Julian Andres Klode <email address hidden> Thu, 23 Sep 2021 15:51:19 +0200

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for command-not-found has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package command-not-found - 20.10.2

---------------
command-not-found (20.10.2) hirsute; urgency=medium

  * Correctly handle compressed APT index files (LP: #1876034)

 -- Julian Andres Klode <email address hidden> Thu, 23 Sep 2021 15:51:19 +0200