direct dependencies of ubiquity should not be autoremovable

[Chris Rainey]

In a clean install of Xubuntu 18.10, if an unsuspecting user runs 'apt autoremove', it will remove 'cryptsetup' and 'lvm2' making the system non-bootable at next restart if an encrypted(LUKS+LVM) root partition was selected during the ubiquity installer wizard:

$ sudo apt update && sudo apt --auto-remove full-upgrade && cat /run/reboo*
<snip...>
The following packages will be REMOVED:
  cryptsetup cryptsetup-bin cryptsetup-initramfs cryptsetup-run dmeventd libdevmapper-event1.02.1 liblvm2app2.2 liblvm2cmd2.02 libreadline5 lvm2
<...snip>

This _will_ make the system non-bootable upon restart if LUKS+LVM are active on the root partition. This would, by extension, make any auto-mounted partitions(home, etc.) unavailable after boot as well!

WORKAROUND:

$ sudo apt install cryptsetup lvm2
<snip...>
cryptsetup is already the newest version (2:2.0.4-2ubuntu2).
cryptsetup set to manually installed.
lvm2 is already the newest version (2.02.176-4.1ubuntu3).
lvm2 set to manually installed.
<...snip>

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: xubuntu-core 2.227
ProcVersionSignature: Ubuntu 4.18.0-10.11-generic 4.18.12
Uname: Linux 4.18.0-10-generic x86_64
ApportVersion: 2.20.10-0ubuntu13
Architecture: amd64
CurrentDesktop: XFCE
Date: Sun Nov 4 18:20:38 2018
InstallationDate: Installed on 2018-11-04 (0 days ago)
InstallationMedia: Xubuntu 18.10 "Cosmic Cuttlefish" - Release amd64 (20181017.2)
SourcePackage: xubuntu-meta
UpgradeStatus: No upgrade log present (probably fresh install)

[flabnat (flabnat)]

same for ubuntu mate 18.10

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Julian Andres Klode (juliank)]

This can be a regression from the minimize-manual work in livecd-rootfs. I'll do some digging shortly.

[Julian Andres Klode (juliank)]

Rerunning the minimize manual script in an xubuntu image with some extra tracking I see the following chain causing it to be marked as automatically installed:

1. xubuntu-desktop Recommends indicator-messages
2. indicator-messages Recommends ubiquity-frontend-gtk (due to indicator-renderer Provides)
3. ubiquity-frontend-gtk Depends on ubiquity
4. ubiquity Depends on cryptsetup, Recommends lvm2

We later on remove ubiquity in cleanup, but don't mark any remaining autoremovable stuff manually installed.

[Julian Andres Klode (juliank)]

I think we should run

apt-get autoremove -s | sed -nr 's#Remv (.*) \[.*#\1#p' | xargs apt-mark manual

after the install finished otherwise, this way get any packages marked as manually installed that would otherwise be subject to autoremoval.

The alternative here would be to run the minimize-manual script on the "install" chroot, and not the "live" chroot. But that requires further integration with live-build.

[Mario Limonciello (superm1)]

Is this expected to not be an issue in bionic as well? I'm hearing similar reports on people who reinstalled from an OEM image.

[Mario Limonciello (superm1)]

FYI I tested both generic bionic image and OEM image. Generic bionic image does not suffer this problem. It's only OEM image generated by Canonical OEM infrastructure that suffers it.

[Yuan-Chen Cheng (ycheng-twn)]

In generic bionic image (live cd mode), lvm and cryptsetup set mark as manual.
In oem image, it's not mark manual. Plan to mark so.

[Arseny (aruseni-magiku)]

I have this bug on Xubuntu 18.10.

The following packages were automatically installed and are no longer required:
  cryptsetup cryptsetup-bin cryptsetup-initramfs cryptsetup-run
Use 'sudo apt autoremove' to remove them.

[Steve Langasek (vorlon)]

According to Julian, this does not (currently?) affect bionic, so marking invalid. If the "manually installed" minimization work is SRUed to bionic, this will need to be dealt with as part of that work but not before.

[Michael Baker (mjb32803)]

I have this bug on Ubuntu Mate 18.10. Repeated install of standard installation with LUKS LVM on HP 14-DF0023CL. After each install, apt autoremove deletes cryptsetup and lvm2 making computer fail to boot on next reboot. Attempted same install on Acer One netboot with exact same result.

[Julian Andres Klode (juliank)]

Fix in https://git.launchpad.net/livecd-rootfs/commit/?id=b2db5bf3658f33dfaf3261bdc73f41be72acd374

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[lumbricus (lumbricus)]

I just did run a "apt autoremove" without being careful enough and can't boot anylonger. After rebooting it just prints:

gave up waiting for suspend/resume device.
...
ALERT! /dev/mapper/xxx does not exist. Dropping to a shell!

Is there an easy way to unbreak a broken system like this one?

[lumbricus (lumbricus)]

I think I've been successful recovering the installation. Please let me know if this makes sense or if there are any mistakes. People running into this, will be happy to find recovery instructions here.

See also: https://askubuntu.com/a/1120949/27088

# find root partition
sudo fdisk -l

# unencrypt partition
# Note: replace /dev/nvme0n1p3 with your disk
# replace "nvme0n1p3_crypt" with the correct name
# check by running this in chroot:
# $ cat /etc/crypttab | cut -f1 -d " "
# nvme0n1p3_crypt
sudo cryptsetup luksOpen /dev/nvme0n1p3 nvme0n1p3_crypt

# mount root partition
sudo vgscan
sudo vgchange -ay
sudo mount /dev/mapper/xubuntu--vg-root /mnt

# prepare chroot environment
sudo mount /dev/nvme0n1p2 /mnt/boot/
sudo mount -o rbind /dev/ /mnt/dev/
sudo mount -t proc proc /mnt/proc/
sudo mount -t sysfs sys /mnt/sys/

# make dns available in chroot
sudo cp /etc/resolv.conf /mnt/etc/resolv.conf

# enter chroot
sudo chroot /mnt /bin/bash

# re-install missing packages
apt install cryptsetup lvm2

# re-generate (this might be done also by apt in the step before, I'm not sure)
update-initramfs -u -k all

# Leave chroot environment
exit
# Write buffers to disk
sudo sync
# Unmount file systems
sudo umount /mnt/sys
sudo umount /mnt/proc
sudo umount /mnt/boot

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.566

---------------
livecd-rootfs (2.566) disco; urgency=medium

  [ Julian Andres Klode ]
  * Do not mark direct dependencies of ubiquity as auto installed. This caused
    cryptsetup to remain auto on the installed system (LP: #1801629)

  [ Tobias Koch ]
  * When the magic-proxy script could not find a valid InRelease file for the
    configured timestamp, it would fall back to serving the canonical version
    of it. This meant that builds would succeed, even though snap-shotting the
    repository failed.
    .
    This update makes the script return HTTP 404 when an InRelease by-hash
    link for a given combination of mirror, suite and timestamp cannot be
    found.

 -- Julian Andres Klode <email address hidden> Tue, 26 Feb 2019 08:56:02 +0100

[Oliver (oliver-uvman)]

This seems to have affected me even though I never ran autoclean. Maybe apt autocleans after I've used it to install things enough times? No idea. Was saved by this answer: https://askubuntu.com/a/1120949/69057

[Julian Andres Klode (juliank)]

Fix is verified, daily images of xubuntu disco as of today do not produce the issue anymore :)

[Mathieu Trudel-Lapierre (cyphermox)]

Julian verified this as not reproducing on disco now, with the livecd-rootfs change. Closing the ubiquity task as invalid.

[Martin Wimpress (flexiondotorg)]

This issue is still affecting Ubuntu MATE 19.10 beta. See the following for more detail:

  * https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1841672

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-mate-meta - 1.253

---------------
ubuntu-mate-meta (1.253) eoan; urgency=medium

  * Refreshed dependencies
  * Added cryptsetup to core, desktop (LP: #11841672, #1801629)
  * Added libnotify-bin to core-recommends, desktop-recommends

 -- Martin Wimpress <email address hidden> Wed, 25 Sep 2019 00:47:17 +0100

[Wolverine (nareshtechs)]

This bug has also affected me on Ubuntu

[Wolverine (nareshtechs)]

Sorry all. Something happened while I was typing in the comment.

The bug has also affected me on Ubuntu 18.04.3 LTS. I finally fixed it as described in comment #17.

Maybe unrelated but FYI. My installation is using UEFI but I need to keep the bios in legacy mode and then select Ubuntu (UEFI) to boot properly. If I set the BIOS to boot using UEFI directly, the fix still fails for me.