The unminimize command fails to reinstall packages with missing files in the `share` and `locale` directories

[Naomi Rose]

[Impact]
========

The `unminimize` command in livecd-rootfs doesn't work out of the box. When `unminimize` is run, it results into the following errors:

    #9 47.54 Reinstalling packages with system documentation in /usr/share/doc/ ..
    #9 48.76 dpkg-query: error: --search needs at least one file name pattern argument
    #9 48.76
    #9 48.76 Use --help for help about querying packages.
    #9 48.80 Reading package lists...
    #9 49.49 Building dependency tree...
    #9 49.62 Reading state information...
    #9 49.76 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    #9 49.76 Restoring system translations...
    #9 50.97 dpkg-query: error: --search needs at least one file name pattern argument
    #9 50.97
    #9 50.97 Use --help for help about querying packages.
    #9 51.01 Reading package lists...
    #9 51.71 Building dependency tree...
    #9 51.85 Reading state information...
    #9 52.00 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    #9 53.18 Documentation has been restored successfully.
    #9 DONE 54.3s

Specifically, the error is:

    dpkg-query: error: --search needs at least one file name pattern argument

The problem is that prior to dpkg/1.21.0, there was a bug where dpkg -V/--verify couldn't list all the correct packages correctly but with that being fix and in archive since Jammy, this works perfectly but the syntax to report the missing files have changed. It just prints 'missing' now. With that new format, we can now fix the regex to simply list the packages.

[Test Plan]
===========

1. Create an image with the changes:
$ ubuntu-bartender --build-provider gce --hook-extras-dir "$(pwd)/cpc_packaging.extra" --livecd-rootfs-branch ubuntu/jammy -- --series jammy --project ubuntu-cpc --image-target gce --subproject minimized

2. Boot into that image. Ensure it's a minimized image.

3. Run update and unminimize.
# apt update
# unminimize

This will result in the above error.

# echo $?
1

Also notes that the above command failed to run successfully.

OTOH, with the fixed package, unminimize will run just fine and "echo $?" would say 0.

[Where Problems Could Occur]
============================

unminimize is already broken so not a lot of things can go wrong at this point. However, if a user has already made amendments to their unminimize script and adapted it to work differently for them, this update could break their existing set up and functionality. But I don't envision that happening a lot, really.

[OG Bug Report]
===============

I am building a custom Docker image based on:

    https://github.com/microsoft/vscode-dev-containers/tree/main/containers/ubuntu

I added a line that runs `yes | unminmize':

    https://github.com/nomirose/devcontainer/commit/1ba51c651c06b9ce71c73e7bf016939d989cfa2c#diff-4c1f94864a9642897e2fa6c1d532830b6d1a2ba9d6b6f2f149178dde32cf0e77R10-R11

Here's from build log :

https://github.com/nomirose/devcontainer/actions/runs/3451129967/jobs/5760151170#step:6:1565

And here are the errors from the `uminmize` command:

    #9 47.54 Reinstalling packages with system documentation in /usr/share/doc/ ..
    #9 48.76 dpkg-query: error: --search needs at least one file name pattern argument
    #9 48.76
    #9 48.76 Use --help for help about querying packages.
    #9 48.80 Reading package lists...
    #9 49.49 Building dependency tree...
    #9 49.62 Reading state information...
    #9 49.76 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    #9 49.76 Restoring system translations...
    #9 50.97 dpkg-query: error: --search needs at least one file name pattern argument
    #9 50.97
    #9 50.97 Use --help for help about querying packages.
    #9 51.01 Reading package lists...
    #9 51.71 Building dependency tree...
    #9 51.85 Reading state information...
    #9 52.00 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    #9 53.18 Documentation has been restored successfully.
    #9 DONE 54.3s

Specifically, the error is:

    dpkg-query: error: --search needs at least one file name pattern argument

I copped `/usr/local/sbin/unminimize` to my local directory to see if I could debug what was doing on.

When looking for packages with missing files in the `/usr/bin/man` directory, the script does this:

    dpkg -S /usr/share/man/ |sed 's|, |\n|g;s|: [^:]*$||' | DEBIAN_FRONTEND=noninteractive xargs apt-get install --reinstall -y

However, for the `/usr/share/docs/` the script does this:

    dpkg --verify --verify-format rpm | awk '/..5...... \/usr\/share\/doc/ {print $2}' | sed 's|/[^/]*$||' | sort |uniq \
        | xargs dpkg -S | sed 's|, |\n|g;s|: [^:]*$||' | uniq | DEBIAN_FRONTEND=noninteractive xargs apt-get install --reinstall -y

The script runs the same command for the `/usr/share/locale/` directory.

In both cases, no packages ever make it to the `xargs dpkg -S` command, so the script throws an error. However, because the error occurs in a pipe, it is lost, and the script continues despite the `set -e` at the start (which is a bug in itself).

However, with a little experimentation, I found that the original command for `/usr/local/bin` works perfectly for the `/usr/share/docs/` and `/usr/share/locale/` directories.

Why not run the same command for all three directories? You could even wrap it in a function.

However, you could improve performance by combining all three into a single command.

The `unminimize` script notes:

    # Reinstallation takes place in two steps because a single dpkg --verified
    # command generates very long parameter list for "xargs dpkg -S" and may go
    # over ARG_MAX. Since many packages have man pages the second download
    # handles a much smaller amount of packages.

I understand the concern about hitting `ARG_MAX`, but I think a better solution to that problem would be to use `xargs -n` to set a limit on the maximum number of arguments. This change would allow xargs to run `dpkg -S` as many times as need (to avoid hitting ARG_MAX).

Doing it this way would also speed up the script.

There are many packages with missing files in multiple minmized directories, meaning that as it stands, even if the bug was fixed, some packages would be re-installed twice, maybe even three times.

(I have seen this for myself because I essentially wrote a shorter version of the script for my own use.)

[John Chittum (jchittum)]

Thank you for reporting the issue and giving suggestions in great detail! Also very timely as myself and another script maintainer were just having conversations about the minimize/unminimize flow.

I've added `livecd-rootfs` package for tracking, as the script lives within those build scripts. We'll take some time to dig into the suggestions and issues and come back with more info.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ferenc Wágner (wferi)]

I think the main problem with the unminimize script in jammy and later is that dpkg 1.21 changed the way it reports missing files: instead of reporting a checksum failure (..5......) it explicitly says "missing" since https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963087 was implemented, breaking the unminimize logic entirely, as demonstrated by /usr/share/doc/libprocps8/NEWS.Debian.gz still being missing at the end of

$ docker run --rm -it ubuntu:jammy sh -c 'yes | unminimize; dpkg --verify libprocps8'

This affects all packages having anything besides their copyright and changelog under /usr/share/doc but having no man pages nor localization.

The above also explains why unminimize uses different logic for different files: the changelog and copyright files are always kept around, while manuals and localization files are excluded entirely.

The ugly but harmless errors from dpkg -S could be avoided by simply using the --no-run-if-empty option of xargs.

xargs should automatically take care of not exceeding the argument length limit of the system, so I assume the comment mentioning that is stale.

[kalvdans (4-launchpad-kalvdans-no-ip-org)]

> However, because the error occurs in a pipe, it is lost, and the script continues despite the `set -e` at the start (which is a bug in itself).

It is not, Posix shell is supposed to behave that way. In bash, you can `set -o pipefail` to get the wanted behaviour.

[Utkarsh Gupta (utkarsh)]

The fix in livecd-rootfs has landed via https://launchpad.net/ubuntu/+source/livecd-rootfs/23.10.30. I've fixed the regex to do the right thing w.r.t the current output dpkg -V emits.

Marking this as done. \o/

[Philip Roche (philroche)]

I don't think we can mark this as Done yet. Only mantic has been fixed

[Philip Roche (philroche)]

I will upload the lunar and jammy SRUs

[Mauricio Faria de Oliveira (mfo)]

This bug has an upload in the unapproved queue, but does not have an SRU bug template [1].
Could you please add it? Thanks!

[1] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

[Andreas Hasenack (ahasenack)]

Marking as incomplete pending the SRU template.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

An upload of livecd-rootfs to jammy-proposed has been rejected from the upload queue for the following reason: "cruft in upload (.idea*/)".

[Philip Roche (philroche)]

Apologies RE jammy upload. I have now re-uploaded Jammy livecd-rootfs 2.765.30 again without any '.idea' directory cruft.

[Steve Langasek (vorlon)]

An upload of livecd-rootfs to lunar-proposed has been rejected from the upload queue for the following reason: ".idea cruft, needs reuploaded".

[Philip Roche (philroche)]

Apologies RE lunar upload. I have now re-uploaded Lunar livecd-rootfs 2.828.2 again without any '.idea' directory cruft - see unapproved queue @ https://launchpad.net/ubuntu/lunar/+queue?queue_state=1&queue_text=livecd-rootfs

[Andreas Hasenack (ahasenack)]

Hello Naomi, or anyone else affected,

Accepted livecd-rootfs into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.828.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Naomi, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.32 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Philip Roche (philroche)]

After some investigation following upcoming changes to LXD and seed the Jammy proposal WRT snap install @ https://code.launchpad.net/~utkarsh/livecd-rootfs/+git/livecd-rootfs/+merge/452742 is not correct as for LTS we need to install targeting a specific LTS track and channel.

Focal has track/channel set to `4.0/stable/ubuntu-20.04` in seed @ https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.focal/server
Groovy has no channel set @ https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/ubuntu/tree/server?h=groovy#n59
Jammy has track/channel set to `5.0/stable/ubuntu-22.04` in seed @ https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.jammy/server
Impish has no channel set @ https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/ubuntu/tree/server?h=impish#n59

As such I propose we block this jammy SRU until this is resolved.

[Philip Roche (philroche)]

I have added tag block-proposed-jammy

[Philip Roche (philroche)]

Comment from @sdeziel in Jammy MP https://code.launchpad.net/~utkarsh/livecd-rootfs/+git/livecd-rootfs/+merge/452742 makes a lot of sense for the unminimzine part related to snap installs.

> Since `lxd-installer` will need to be modified to pull the snap from the right channel when we know it, it might be best to rely on `lxd-installer` to do the right thing.
>
> As such, I'd propose to simply run the `lxd-installer` prior to purging it.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.828.3

---------------
livecd-rootfs (2.828.3) lunar; urgency=medium

  [ Heinrich Schuchardt ]
  * arm: fix console parameter for ARM cloud-images (LP: #2036730)

livecd-rootfs (2.828.2) lunar; urgency=medium

  [ Steve Langasek ]
  * The chroot tmpfs mount should only be /var/lib/apt/lists, not
    /var/lib/apt; the latter breaks changes to /var/lib/apt/extended_states.
    (LP: #2036195).

livecd-rootfs (2.828.1) lunar; urgency=medium

  * Fix unminimize to correctly list packages. (LP: #1996489)
  * Install LXD snap from stable/ubuntu-<version> channel. (LP: #2036725)

 -- Steve Langasek <email address hidden> Thu, 07 Dec 2023 08:51:51 -0800

[Andreas Hasenack (ahasenack)]

I made a mistake releasing this package for lunar, as this bug has not been verified yet.

[Utkarsh Gupta (utkarsh)]

Hey there!

I've tested the update from -proposed pockets of all the releases. The unminimize script works as intended on Lunar and Jammy, marking that as verification-done.

[Łukasz Zemczak (sil2100)]

Hello Naomi, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.33 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Naomi, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.34 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Utkarsh Gupta (utkarsh)]

Hey!

Tested version 2.765.34 from -proposed. unminimize run works and the diff between a base image and a minimized image with unminimize run is small and expected. Therefore, we're good to go for Jammy.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.765.34

---------------
livecd-rootfs (2.765.34) jammy; urgency=medium

  * Remove comments in the debian/control dependencies as it was causing the
    package build to fail.

livecd-rootfs (2.765.33) jammy; urgency=medium

  [ Steve Langasek ]
  * Use losetup instead of kpartx to resolve race conditions in riscv64
    image builds. LP: #2045797.

  [ Utkarsh Gupta ]
  * unminimize: Use lxd-installer to install LXD itself (LP: #2036725)

livecd-rootfs (2.765.32) jammy; urgency=medium

  [ Heinrich Schuchardt ]
  * arm: fix console parameter for ARM cloud-images (LP: #2036730)

livecd-rootfs (2.765.31) jammy; urgency=medium

  [ John Chittum ]
  * fix: add 6.5 kernel apparmor features to livecd-rootfs based on
    features of 6.5 in ubuntu/mantic. This will roll as HWE. (LP: #2037567)

livecd-rootfs (2.765.30) jammy; urgency=medium

  * Enable snap preseeding with ppc64el images where /boot/vmlinux is used
    instead of /boot/vmlinuz. (LP: #2038957)

livecd-rootfs (2.765.29) jammy; urgency=medium

  [ Steve Langasek ]
  * The chroot tmpfs mount should only be /var/lib/apt/lists, not
    /var/lib/apt; the latter breaks changes to /var/lib/apt/extended_states.
    (LP: #2036195).

livecd-rootfs (2.765.28) jammy; urgency=medium

  * Fix unminimize to correctly list packages. (LP: #1996489)
  * Install LXD snap from stable/ubuntu-<version> channel. (LP: #2036725)

livecd-rootfs (2.765.27) jammy; urgency=medium

  [ Thomas Bechtold ]
  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc
    project builds (LP: #1968873)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Mon, 15 Jan 2024 16:41:18 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.