overlayroot: nofail is not applied, an unplugged removable disk makes it stuck on boot

[Adam Vodopjan]

On a fresh jammy server minimal install I have such fstab entries:

    /dev/sda1 / ext4 defaults 0 1
    /dev/sdb1 /disk1 ext4 nofail 0 2

If the second disk is attached, it boots as supposed with and without overlayroot=tmpfs in the kernel cmdline. In case the disk is unplugged and the kernel cmdline contains overlayroot=tmpfs (without extra :recurse=0), the boot stumbles on such message:

    You are in emergency mode. After logging in, type "journalctl -xb" to ..
    Press Enter for maintenance
    (or press Control-D to continue):

So I press Enter and inspect fstab generated by overlayroot:

    > grep disk1 /etc/fstab
    /dev/sdb1 /media/root-ro/disk1 ext4 ro,nofail 0 2
    /media/root-ro/disk1 /disk1 overlay lowerdir=/media/root-ro/disk1,upperdir=/media/root-rw/overlay/disk1,workdir=/media/root-rw/overlay-workdir/disk1 0 2

Evidently, nofail is not applied to the overlay mount, which makes it non optional.

When trying to replicate, it is important to NOT use FAT formatted sdb1 since overlayroot does bypass FAT.

[Athos Ribeiro (athos-ribeiro)]

Thanks for reporting this issue, Adam!

Have you checked the system logs when the issue occurs? Could you provide them here?

Moreover, would you be able to provide steps for a minimal reproducer from within a VM with a fresh Ubuntu install?

Thank you!

[Adam Vodopjan (grozzly)]

Here I tried to completely cover it:

- create two virtual disks
- on each disk
  - create a single MBR partition over the whole disk
  - format the partition as ext4

- create a virtual machine with the disks attached
- install jammy server minimal onto sda1
- boot the vm

- just in case, disable swap
  - run "sudo swapoff -a"
  - drop the swap line in /etc/fstab
  - delete /swapfile

- run "sudo mkdir /disk1"
- in /etc/fstab mount sdb1 as /disk1 with such options:

    /dev/sdb1 /disk1 ext4 nofail,x-systemd.device-timeout=10 0 2

- run "sudo mount -a"
- checkpoint:

    > mount | grep /dev/sd
    /dev/sda1 on / type ext4 (rw,relatime)
    /dev/sdb1 on /disk1 type ext4 (rw,relatime)

- run "sudo apt install overlayroot -y"

[1] - reboot and keep Shift pressed to stop at grub menu
[2] - in grub, press "e" to edit the boot entry
[3] - in the "linux ..." line append "overlayroot=tmpfs"
[4] - press Ctrl+X to boot

- checkpoint:

    > mount | grep overlay
    overlayroot on / type overlay ..
    /media/root-ro/disk1 on /disk1 type overlay ..

- shutdown the machine
- detach the second disk
- repeat [1]..[4]

- notice there is a 10s countdown waiting for the second disk
- checkpoint:

    You are in emergency mode. After logging in, type "journalctl -xb" to view system logs,
    "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode.
    Press Enter for maintenance
    (or press Control-D to ocntinue):

- press Enter
- run "journalctl -b -o cat | grep local-fs"
- checkpoint:

    local-fs.target: Job local-fs.target/start failed with result 'dependency'.
    local-fs.target: Triggering OnFailure= dependencies.

Full output of "journalctl -xb" from a failed boot attempt attached

[Athos Ribeiro (athos-ribeiro)]

Thanks for the (very) detailed reproducer, Adam.

I am adding this to the server team backlog so we can get to this soon.

[Robie Basak (racb)]

Hi,

Please could you expand on what you're trying to achieve? Why aren't you using our standard installer or standard tooling to create a bootable Ubuntu VM?

> - install jammy server minimal onto sda1

What does this mean exactly? Please could you expand on this?

If you're not using a supported installer and you're also not using one of our standard and supported mechanisms to set up an Ubuntu VM, I'm struggling to understand why you think this is a bug in Ubuntu rather than in the process you are using to set up the system "by hand". Please could you clarify?

For now, I'm setting the bug status to Incomplete. Please set it back to New once you have answered.

Thanks!

[Adam Vodopjan (grozzly)]

@racb When using the standard installer there is an option at some step if it should be a minimal install or not. My point was a minimal install is enough to demo the problem.

> I'm struggling to understand why you think this is a bug in Ubuntu

It is a bug in overlayroot stuff which comes from cloud-initramfs-tools bundled by ubuntu. It should give special treatment to "nofail" mount option.

[Robie Basak (racb)]

OK, so am I right in understanding that this issue only occurs when attempting to use the cloud-initramfs-tools overlayroot function in a custom way that our installers and VM images do not do by default?

If so it may be a valid bug in that package, but not one that we can prioritise fixing. Patches welcome!

[Adam Vodopjan (grozzly)]

I'm not sure what are you talking about. Mby it would be better to tell you how I encountered the problem.

I've got a headless raspberry pi running ubuntu from an sdcard. It has an extra usb disk plugged in. In fstab I mount the disk as nofail. If I detach the disk and reboot, nothing bad happens, the mount silently fails. But when I detach the disk and reboot with overlayroot=tmpfs kernel option (I just wanted to do a backup), it does not boot, as explained above. A headless system does not boot because of a harmless kernel option. Nice, right? :)

In #2 I provided detailed instructions on how to reproduce the problem in a vm with ubuntu installed using the standard installer.

> Patches welcome

I knew you'd say it

[Adam Vodopjan (grozzly)]

Btw, when I say "the standard installer" I mean nothing else but the server iso. Not some cloud stuff. And for virtual machines I use virtualbox.

[Robie Basak (racb)]

Right, so the overlayroot=tmpfs was something you added - not supplied by our images or our installer.

I agree that it would be nice if this worked correctly, including with the onfail option. As I say this is a valid request, and it's in our backlog. But sorry, we have to prioritise requests. This doesn't preclude those with different priorities from working on their specific cases themselves, and we'd be happy to work with you to land a fix if you have a proposed solution (presumably this would be special case handling in overlayroot= handling when nofail is in use).

[Adam Vodopjan (grozzly)]

I don't get it why you keep talking about the installer and some "images". Is this the wrong place to report bugs in overlayroot >>package<< ?

[Robie Basak (racb)]

> I don't get it why you keep talking about the installer and some "images".

Because this affects our assessment of priorities.

> Is this the wrong place to report bugs in overlayroot >>package<< ?

This is the correct place. In both my previous replies I made a point of acknowledging that your report appears to be valid. I just want to set expectations that nobody else can be expected to work on this soon.

[Adam Vodopjan (grozzly)]

Thanks for making it clear. I just wanted to a report a problem, no expectations here.

To those who might look into it later: such patch does fix the issue, but who knows what possible side effects are

    --- initramfs-tools/scripts/init-bottom/overlayroot
    +++ initramfs-tools.1/scripts/init-bottom/overlayroot
    @@ -393,7 +393,7 @@
            else
                ospec="${root_ro}${file}"
                copy_opts=""
    - for copy_opt in nobootwait noauto; do
    + for copy_opt in nobootwait noauto nofail; do
                    case ",$opts," in
                        *,${copy_opt},*)
                            copy_opts="${copy_opts},${copy_opt}";;

[Paride Legovini (paride)]

Hello Adam and thanks for the patch! I was also worried by possible side effects, however I checked what fstab(5) says about nofail, and that's:

  do not report errors for this device if it does not exist.

which seems safe. Moreover the flag would be inherited only if already present in the "parent" fstab entry.

Would to like to submit your patch as a merge request to the cloud-initramfs-tools repository [1]? Authorship will show in the package changelog.

[1] https://code.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/+git/cloud-initramfs-tools/+ref/master

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-initramfs-tools - 0.48

---------------
cloud-initramfs-tools (0.48) noble; urgency=medium

  [ Brian Murray ]
  * debian/control Fix a typo.

  [ grozzly ]
  * overlayroot: add 'nofail' to the list of inherited mount options
    (LP: #2049271)

  [ Paride Legovini ]
  * d/copyright: fix duplicate globbing pattern
  * d/copyright: use secure (https) Format URI
  * d/copyright: trim trailing whitespaces
  * d/control: specify Rules-Requires-Root: no
  * d/control: switch to dh compat level 13 via debhelper-compat.
    No changes needed.
  * d/copyright: coalesce identically licensed debian/ stanzas
  * d/copyright: add Canonical Ltd. to the debian/* stanza
  * d/control: bump Standards-Version to 4.6.2 (no changes needed)
  * d/cloud-initramfs-copymods.l-o: override empty directory warning.
    The empty directory is created on purpose.

 -- Paride Legovini <email address hidden> Fri, 26 Jan 2024 11:49:48 +0100