ubiquity offers installation media as an install target

[Michael Hudson-Doyle]

I guess this is actually a bug in some partman component, but well. In both the desktop and ubuntu studio installers for eoan beta, I was offered the installation media as a target disk. This seems unlikely to be correct.

==

partman-auto must not use the _whole_ device, of anything that is mounted on /cdrom.

Our ISOs are multipartition eltorito devices. And one can mount the top-level device, or the first partition, to get the same content (depending if one knows how to read partitions that is). Previously we were mounting the top-level /dev/sda, but now we started to mount /dev/sda1, unbreaking using other partitions of the installer media for purposes of persistence & installer logs.

Partman & parted are actually very confused by our installer media, as they get tricked by -isohybrid-apm-hfsplus into thinking it is MAC based partition table, when in fact it's just MS-DOS or GPT depending how one looks at it. At least blkid/udevadm correctly declare it el-torito hybrid image.

Thus when parted init code tries to walk the partitions, and detect installer media mounts none of that code actually runs, as no partitions are detected partman (and parted_devices declares it unknown partition table).

Hence I guess, the static strict check for the "whole $device mount on /cdrom" was added.

But given the above change to mount just a partition this has now started to fail. Imho we should expand the pattern to match at least optionally match 'p?[0-9]*' - depending on how one re-rolls the iso, it may or may not be multipartition one, with optionally 'p'-prefixed (hello nvme) partition number, such that all of these cases are valid.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1845571

[Dimitri John Ledkov (xnox)]

i wonder if we are missing casper uuids or some things in .disk that causes the installation media to be excluded (detected as install media)

[Sebastien Bacher (seb128)]

@Brian, I'm raising that back to at least High, Desktop consider it as a visible problem that should be resolved for 19.10, could you make sure it's stay on your list?

[Jean-Baptiste Lallement (jibel)]

IMHO, importance is High and fixed for release because the disk selection page is always displayed even if the machine only has one drive and shouldn't. Also if the user selects the USB stick, the installation will fail.

[Michael Hudson-Doyle (mwhudson)]

The reason that I argued for the importance to be lowered is that I'm not super confident about fixing it on a short turnaround without causing other problems. In disco and before, casper would mount the entire device (say /dev/sdb) containing the installation media (i.e. mount /dev/sdb not /dev/sdb1) and that would cause partman base to exclude /dev/sdb entirely (code around https://git.launchpad.net/ubuntu/+source/partman-base/tree/init.d/parted?h=ubuntu/eoan-proposed&id=ac4be1bb9c00f28d8e19f4ec0c6323769085d593#n156). Now casper mounts /dev/sdb1 and so that filtering doesn't remove the disk (well, unless sdb1 fills the device, anyway). It would be possible to cause that filtering to still exclude the disk but that would break the dell use case where the installer is in a recovery partition and expects to install to another partition on the same device (see https://bugs.launchpad.net/bugs/1846762).

So filtering needs to be added to partman-auto, specifically in automatically_partition/{replace,reuse,some_device}. The thing is, it looks like get_auto_disks in lib/auto-shared.sh is _already_ trying to exclude the device with the install media on it:

  # Skip devices containing the installation medium
  [ -e "$dev/installation_medium" ] && continue

This file is supposed to be being created by init.d/parted but apparently isn't. And also fixing this would seem to break the Dell use case.

And now I'm giving up for now :)

[Sebastien Bacher (seb128)]

If we are not confident in fixing the bug this cycle what about reverting the casper change that created the problem for now?

[Sebastien Bacher (seb128)]

(also 'it's hard to fix' isn't really a reason that justify lowering the importance of an issue)

[Jeremy Bícha (jbicha)]

Years ago, I believe I used to be able to use grub to mount an Ubuntu iso on my hard drive as a loopback device and use that to install Ubuntu on the same hard drive. But years ago, that use case broke. (Apologies if I'm way off-topic here.)

[Brian Murray (brian-murray)]

For the record this happens when choosing use "Erase disk and install Ubuntu", not in an advanced partitioning screen. Attached you'll find a screenshot of issue with the install media as the default destination. Although it might be because I used the following "kvm -m 4096 -boot d -hda /misc/isos/ubuntu/eoan-desktop-amd64.iso -hdb /srv/vms/eoan.qemu".

[Dimitri John Ledkov (xnox)]

But looking at brian's screenshot 2.3GB usb stick should not be offered for installation, as it's too small and should have been filtered out by some_device for installation.

[Dimitri John Ledkov (xnox)]

For the dell usecase, the expection is to "use free" or "wipe Ubuntu FOO and reinstall", meaning that only free space is used or an existing Ubuntu partition is wiped & reinstalled.

Meaning that the full device /dev/sdb (the install media) is not offered for install target.

The only time we do support wiping and installing ont hte full /dev/sdb which we booted from, is when casper was booted with toram option. In which case yes, it's expected to wipe the disk one booted from.

[Dimitri John Ledkov (xnox)]

Reading a bit more code....

ubiquity has partman/check.d/partition_too_small which does read /cdrom/casper/filesystem.size (which is 5GiB) and should actually prevent from successfully contining to install on the installation media.

Also there is d-i/source/partman-auto/lib/auto-shared.sh get_auto_disks() which should be skipping /cdrom , ie.

                # Skip devices containing the installation medium
                [ -e "$dev/installation_medium" ] && continue

                # Skip installer disk
                $(mount | grep -qF "$device on /cdrom ") && continue

So we need to check that those pieces are working correctly and that devices smaller than 5GiB are never offered for installation *both* when they are and are *not* installation media.

[Dimitri John Ledkov (xnox)]

Hm, i wonder if

$(mount | grep -qF "$device on /cdrom ") && continue

is broken ever since UEFI was introduced, since now $device1 is mounted not simply $device.

[Dimitri John Ledkov (xnox)]

Hm, I wonder, if this is now the same issue as seen on ppc64le subiquity images, that parted now detects our installer media as partition table `mac' and doesn't deal with it via partman which shows no partitions for the device. and the state directory for it has no partitions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package partman-auto - 134ubuntu11

---------------
partman-auto (134ubuntu11) eoan; urgency=medium

  * lib/auto-shared.sh: skip installer disk when it's mounted as a
    partition too, as the current casper tries to do by default when
    presented with a multi-partition el-torito hybrid image. LP: #1845571

 -- Dimitri John Ledkov <email address hidden> Thu, 10 Oct 2019 02:50:54 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubiquity - 19.10.18

---------------
ubiquity (19.10.18) eoan; urgency=medium

  * Automatic update of included source packages: partman-auto
    134ubuntu11. (LP: #1845571)

 -- Dimitri John Ledkov <email address hidden> Thu, 10 Oct 2019 03:04:14 +0100

[Mario Limonciello (superm1)]

I think this change broke OEM installation from a recovery partition on HDD. Can't seem to install to it anymore.

[Mario Limonciello (superm1)]

Yes; I can confirm that is root cause; IE removing that new line in /lib/partman/lib/auto-shared.sh resolves the issue.

[Michael Hudson-Doyle (mwhudson)]

Argh I was worried that might happen :( Are you in a postition to test the attached patch? (I realize testing installer patches like this isn't the easiest thing)

[Mario Limonciello (superm1)]

Thanks for the quick turnaround! I was able to do some quick testing with it, and it does fix the problem.

A few comments though:
* This line:
$(mount | grep -qF "$device on /cdrom ") && continue

Is what was in 134ubuntu10, so is that what you actually meant? It seems like it might want to be:
$(mount | grep -qE ""$device"p?[0-9]* on /cdrom ") && continue

If you want to use the new logic xnox added.

* The OEM recipe we use doesn't use some_device, it uses biggest_free in case it's relevant.

[Michael Hudson-Doyle (mwhudson)]

Argh you're right of course. Have to run now, don't have time to make a better patch.

And yes, I was assuming your recipe doesn't use some_device -- the intent of my change was essentially to restrict the logic xnox added to some_device and not the others (we should also probably add some filtering to 'replace' to exclude the installer system, but usually that's too small anyway AIUI).

[Mario Limonciello (superm1)]

Ah right, so since we're using biggest_free that corrected line from xnox wouldn't even run.
I opened up another bug for this regression so we don't lose it (bug 1847944), so maybe we should move discussion there.

[Dimitri John Ledkov (xnox)]

@superm1 i want the OEM image.

Imho, i like mwhudson approach. In general, we do want to prevent some_device, if installer is running from the device in question. My change to capture partitions, is because our regular stock installer now mounts as a partition, rather than a whole disk.

[Dimitri John Ledkov (xnox)]

I'm slightly interested to know how you can use biggest_free

  # Skip devices containing the installation medium
  [ -e "$dev/installation_medium" ] && continue

And we have always done that, meaning if one boots from something that looks like installation_medium it will not be offered for autopartitioning.