Fresh install of kinetic-preinstalled-desktop-arm64+raspi has packages to autoremove

[Brian Murray]

[Impact]

Immediately after installing hirsute-preinstalled-desktop-arm64+raspi on a Raspberry Pi I noticed that there were packages available for autoremoval.

bdmurray@bdmurray-desktop:~$ sudo apt autoremove
[sudo] password for bdmurray:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  cryptsetup-bin dctrl-tools dmeventd dmraid dpkg-repack efibootmgr gir1.2-timezonemap-1.0 gir1.2-xkl-1.0 grub-common grub-efi-arm64
  grub-efi-arm64-bin grub-efi-arm64-signed grub2-common kpartx kpartx-boot libdebian-installer4 libdevmapper-event1.02.1
  libdmraid1.0.0.rc16 liblvm2cmd2.03 libtimezonemap-data libtimezonemap1 lvm2 os-prober python3-icu python3-pam rdate
  thin-provisioning-tools
0 upgraded, 0 newly installed, 27 to remove and 0 not upgraded.
After this operation, 45.5 MB disk space will be freed.
Do you want to continue? [Y/n] n
Abort.

It seems to me the installation process should run an autoremove at the end.

bdmurray@bdmurray-desktop:~$ cat /var/log/installer/version
ubiquity 21.04.19

[Test case]
* Wait for a daily jammy preinstalled image to be built after ubiquity is accepted and built.
* Install using the image.
* Confirm that after installation, 'sudo apt autoremove' does not offer to remove any additional packages.

[Where things could go wrong]
Because we are now removing additional packages from the system that previously were left installed at the end of oem-config, it is possible that for some installations using oem-config - but *not* the raspi preinstalled images - that some of the packages being removed are needed to access the disk and that removal will break the boot. The correct fix in such a case is to make sure the relevant packages are marked manually installed as part of the installation process; we already do this, but because of this bug with removals it could be that this isn't being done correctly in all cases which would only be discovered after fixing this bug. We should handle that by making any necessary follow-up fixes to ubiquity.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1925265

[Steve Langasek (vorlon)]

This bug has been open now and listed in the release notes for 3 releases, including the latest LTS, which is far from ideal. Has anyone reproduced it with a current release? If this is a preinstalled image, why is this reported against ubiquity, which is the desktop installer?

I don't have a raspi to test on, but if I mount the jammy image and chroot into it (without booting) I see:

# apt autoremove --purge
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
#

[Steve Langasek (vorlon)]

ok, figured this out a bit more. oem-config is installed (because you need a first-boot configurator), which is built from ubiquity source. oem-config-firstboot runs:
                apt-get -y purge ubiquity >>/var/log/oem-config.log 2>&1

But does not do an apt autoremove --purge afterwards. So the code needs changed to do this.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sam Lane (samlane00)]

When creating the Ubuntu Budgie Pi image, we ran into this same issue. The packages are removed through /usr/sbin/oem-config-remove-gtk but leave behind these same packages. If we mount and chroot into the image, and modify oem-config-remove-gtk and change:

    def _on_transaction(trans):
        apt_dialog = AptProgressDialog(trans)
        ...

to:

    def _on_transaction(trans):
        trans.set_remove_obsoleted_depends(remove_obsoleted_depends=True)
        apt_dialog = AptProgressDialog(trans)

this seems to "fix" the issue. After setup, there are no packages to autoremove. I have tested this not only on our Budgie Kinetic Pi image, but also on the Ubuntu Kinetic RasPi Preinstalled Desktop image.

[Julian Andres Klode (juliank)]

That seems correct, fsvo correct. Like if everything else is correctly working, this works correctly. But if something else fails to be marked manually it gets weird obviously. So it's not bug resistant, but not much we can do better without having `apt history undo` right now

I think there's probably a nicer place to put this in, where trans is created.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubiquity - 23.04.8

---------------
ubiquity (23.04.8) lunar; urgency=medium

  * Whenever removing packages, use apt-get autoremove --purge instead of
    apt-get purge so that no-longer-used dependencies are removed together
    with the package in question, and we do not leave behind any packages
    that will be reported as autoremovable later. LP: #1925265.

 -- Steve Langasek <email address hidden> Fri, 14 Apr 2023 11:35:46 -0700

[Dave Jones (waveform)]

Sorry, still appears to be an issue on the raspi preinstalled desktop images (confirmed that ubiquity 23.04.8 was used in their production in the manifest). This was the state immediately after installation:

dave@miss-piggy:~$ sudo apt autoremove
[sudo] password for dave:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED
  dctrl-tools dmeventd dmraid dpkg-repack efibootmgr gir1.2-nma-1.0
  gir1.2-timezonemap-1.0 gir1.2-xkl-1.0 grub-common grub-efi-arm64
  grub-efi-arm64-bin grub-efi-arm64-signed grub2-common kpartx kpartx-boot
  libdebian-installer4 libdevmapper-event1.02.1 libdmraid1.0.0.rc16
  libdpkg-perl libfile-fcntllock-perl liblvm2cmd2.03 libtimezonemap-data
  libtimezonemap1 lvm2 os-prober python3-gi-cairo python3-icu python3-pam
  rdate thin-provisioning-tools tzdata-legacy
0 to upgrade, 0 to newly install, 31 to remove and 0 not to upgrade.
After this operation, 56.4 MB disk space will be freed.
Do you want to continue? [Y/n] n
Abort.

[Steve Langasek (vorlon)]

The remaining packages marked for removal result from oem-config-remove-gtk, which doesn't simply call apt-get purge - it uses aptdaemon.

And aptdaemon looks to me like it's supposed to remove "obsoleted dependencies" (equivalent of apt-get autoremove) via worker.aptworker.AptWorker.commit_packages -> worker.aptworker.AptWorker._check_obsoleted_dependencies. So I think this is now actually an aptdaemon bug?

[Steve Langasek (vorlon)]

ah - no, there's a property we need to set on the client side to "remove obsoleted depends".

[Steve Langasek (vorlon)]

this is fixed in mantic:

ubiquity (23.04.9) mantic; urgency=medium

  * bin/oem-config-remove-gtk uses aptdaemon to remove packages, not
    apt-get, so was overlooked in the previous upload. Set the flag to
    autoremove dependencies here as well. LP:# 1925265.

 -- Steve Langasek <email address hidden> Wed, 26 Apr 2023 21:51:58 +0200

[Robie Basak (racb)]

Hello Brian, or anyone else affected,

Accepted ubiquity into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubiquity/22.04.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dave Jones (waveform)]

Verification done on daily jammy pre-installed desktop image. After oem-config has completed, after first login, I attempted apt autoremove and got the following output:

dave@miss-piggy:~$ sudo apt autoremove
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 to upgrade, 0 to newly install, 0 to remove and 5 not to upgrade.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubiquity - 22.04.20

---------------
ubiquity (22.04.20) jammy; urgency=medium

  [ Steve Langasek ]
  * Whenever removing packages, use apt-get autoremove --purge instead of
    apt-get purge so that no-longer-used dependencies are removed together
    with the package in question, and we do not leave behind any packages
    that will be reported as autoremovable later. LP: #1925265.
  * bin/oem-config-remove-gtk uses aptdaemon to remove packages, not
    apt-get. Set the flag to autoremove dependencies here as well.

  [ Nick Rosbrook ]
  * tests: modify test_city_entry to adapt to tzdata update (LP: #2022965)

  [ Shih-Yuan Lee (FourDollars) ]
  * Use dpkg-divert /sbin/start-stop-daemon unconditionally instead of just
    renaming it conditionally. (LP: #1978931)

 -- Steve Langasek <email address hidden> Mon, 05 Jun 2023 14:36:01 -0700

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for ubiquity has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.