apt update crashes with "KeyError: 'suite'" in cnf-update-db if index compression is enabled

Bug #1876034 reported by Michael Sparmann
68
This bug affects 14 people
Affects Status Importance Assigned to Milestone
command-not-found (Ubuntu)
Fix Released
Medium
Unassigned
Focal
Fix Released
Medium
Unassigned
Hirsute
Fix Released
Medium
Unassigned
Impish
Fix Released
Medium
Unassigned

Bug Description

[Impact]
On all Ubuntu releases since at least 19.10, enabling compressed APT package list storage with command-not-found installed leads to this crash in the cnf-update-db hook called by apt update:

Traceback (most recent call last):
  File "/usr/lib/cnf-update-db", line 26, in <module>
    col.create(db)
  File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 93, in create
    self._fill_commands(con)
  File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 127, in _fill_commands
    self._parse_single_commands_file(con, fp)
  File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 165, in _parse_single_commands_file
    suite=tagf.section["suite"]
KeyError: 'suite'

(To enable compressed APT list storage, add Acquire::GzipIndexes "true"; to an apt.conf fragment.)

This appears to be caused by apt downloading an lz4-compressed command list for command-not-found and cnf-update-db passing the /var/lib/apt/lists/*Commands-*.lz4 files to db/creator.py, which does not know about the compression and attempts to parse them as plain text.

[Test plan]
Enable the option, purge old lists, and update to populate lists directory with compressed files; this will run cnf-update-db for you.

[Where problems could occur]
Replacing the open() call with subprocess.Popen for apt-helper seems like a minimal change. If apt-helper has problems, we check the return code and fail, keeping the old db in place.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in command-not-found (Ubuntu):
status: New → Confirmed
Mathew Hodson (mhodson)
Changed in command-not-found (Ubuntu):
importance: Undecided → Medium
Revision history for this message
sparky3387 (sparky3387) wrote :

As a temporary workaround, you can add to an apt.conf file:
Acquire::GzipIndexes "false";

And do a check to ensure no files have GzipIndexes:
grep -r GzipIndexes /etc/apt

And then delete apt list cache
rm -rf /var/lib/apt/lists

And then finally update
apt-get update

Revision history for this message
Michael Sparmann (theseven) wrote :

I'm aware of that, but that workaround defeats the point of enabling compressed list storage.
We're using the patch linked above to solve the real issue without that space overhead, but such local fixes are a maintenance issue. Sadly there appears to be no intention to accept the fix upstream.

Revision history for this message
Michael Sparmann (theseven) wrote :

Actually, upon a closer look, it appears like command-not-found has been entirely unmaintained for more than 2 years now, so efforts to get this fixed upstream, let alone backported to focal, are probably futile.
We should probably just remove this package from our base image and let the end user install some fixed fork of it from a PPA, if they need it. Luckily there's no hard dependency to it from ubuntu-standard.

Revision history for this message
Anatoly Borodin (anatoly.borodin) wrote :

Standard Docker images contain a file /etc/apt/apt.conf.d/docker-gzip-indexes with

Acquire::GzipIndexes "true"; Acquire::CompressionTypes::Order:: "gz";

in it. So c-n-f is broken there.

tags: added: rls-ii-incoming
Revision history for this message
Julian Andres Klode (juliank) wrote :

FWIW, Docker images do not install command-not-found and are intended for machine users, so it's not a place where you'd install it and notice that.

tags: added: fr-1673
Changed in command-not-found (Ubuntu Impish):
status: Confirmed → Triaged
tags: removed: rls-ii-incoming
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package command-not-found - 21.10.0

---------------
command-not-found (21.10.0) impish; urgency=medium

  * Correctly handle compressed APT index files (LP: #1876034)

 -- Julian Andres Klode <email address hidden> Thu, 23 Sep 2021 15:32:30 +0200

Changed in command-not-found (Ubuntu Impish):
status: Triaged → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Michael, or anyone else affected,

Accepted command-not-found into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/command-not-found/20.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in command-not-found (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Steve Langasek (vorlon) wrote :

The upload in the hirsute queue shows multiple changes outside of the fix for compressed files in db/creator.py, including extensive changes to the debian/ directory. Have these been included by mistake?

Changed in command-not-found (Ubuntu Hirsute):
status: New → Incomplete
Mathew Hodson (mhodson)
Changed in command-not-found (Ubuntu Focal):
importance: Undecided → Medium
Changed in command-not-found (Ubuntu Hirsute):
importance: Undecided → Medium
Revision history for this message
Brian Murray (brian-murray) wrote :

With regards to the multiple changes the Launchpad diff was incorrect in that it was comparing a version of the package that is not in Hirsute.

Changed in command-not-found (Ubuntu Hirsute):
status: Incomplete → Fix Committed
tags: added: verification-needed-hirsute
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Michael, or anyone else affected,

Accepted command-not-found into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/command-not-found/20.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
John Freeman (john-freeman) wrote :

The fix works for me. It's relatively easy to check for yourself with Docker.

To reproduce the issue (version 20.04.4):
docker run --rm ubuntu:focal bash -c 'export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get -y install --no-install-recommends command-not-found && apt-get update ; apt-cache policy command-not-found'

To test the fix (version 20.04.5):
docker run --rm ubuntu:focal bash -c 'echo "deb http://archive.ubuntu.com/ubuntu/ focal-proposed restricted main multiverse universe" > /etc/apt/sources.list.d/ubuntu-focal-proposed.list && export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get -y install --no-install-recommends command-not-found && apt-get update ; apt-cache policy command-not-found'

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
John Freeman (john-freeman) wrote :

I checked and the fix also works on Hirsute.

To reproduce the issue (version 20.10.1):
docker run --rm ubuntu:hirsute bash -c 'export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get -y install --no-install-recommends command-not-found && apt-get update ; apt-cache policy command-not-found'

To test the fix (version 20.10.2):
docker run --rm ubuntu:hirsute bash -c 'echo "deb http://archive.ubuntu.com/ubuntu/ hirsute-proposed restricted main multiverse universe" > /etc/apt/sources.list.d/ubuntu-hirsute-proposed.list && export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get -y install --no-install-recommends command-not-found && apt-get update ; apt-cache policy command-not-found'

tags: added: verification-done verification-done-hirsute
removed: verification-needed verification-needed-hirsute
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package command-not-found - 20.04.5

---------------
command-not-found (20.04.5) focal; urgency=medium

  * Correctly handle compressed APT index files (LP: #1876034)

 -- Julian Andres Klode <email address hidden> Thu, 23 Sep 2021 15:51:19 +0200

Changed in command-not-found (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for command-not-found has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package command-not-found - 20.10.2

---------------
command-not-found (20.10.2) hirsute; urgency=medium

  * Correctly handle compressed APT index files (LP: #1876034)

 -- Julian Andres Klode <email address hidden> Thu, 23 Sep 2021 15:51:19 +0200

Changed in command-not-found (Ubuntu Hirsute):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers