netbooting the bionic live CD over NFS goes straight to maintenance mode :

[Eric Desrochers]

[Impact]

Mounting manually a network share (NFS) and masking it breaks the state of other units (and their dependencies).
Casper is masking a mounted NFS share, blocking the normal boot process as described in the original description, but the issue comes from systemd.

[Test Case]

- NFS mount point at /media
root@iscsi-bionic:/home/ubuntu# mount | grep media
10.230.56.72:/tmp/mnt on /media type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.127,local_lock=none,addr=10.230.56.72)

- Test mount point (/test) defined in /etc/fstab:
root@iscsi-bionic:/home/ubuntu# cat /etc/fstab |grep test
tmpfs /test tmpfs nosuid,nodev 0 0

1. If media.mount is not masked, everything works fine:

root@iscsi-bionic:/home/ubuntu# mount | grep test
root@iscsi-bionic:/home/ubuntu# systemctl status media.mount | grep Active
   Active: active (mounted) since Thu 2018-11-15 16:03:59 UTC; 3 weeks 6 days ago
root@iscsi-bionic:/home/ubuntu# systemctl status test.mount | grep Active
   Active: inactive (dead) since Thu 2018-12-13 10:33:52 UTC; 4min 11s ago
root@iscsi-bionic:/home/ubuntu# systemctl start test.mount
root@iscsi-bionic:/home/ubuntu# systemctl status test.mount | grep Active
   Active: active (mounted) since Thu 2018-12-13 10:38:13 UTC; 3s ago
root@iscsi-bionic:/home/ubuntu# mount | grep test
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)
root@iscsi-bionic:/home/ubuntu# systemctl stop test.mount
root@iscsi-bionic:/home/ubuntu# systemctl status test.mount | grep Active
   Active: inactive (dead) since Thu 2018-12-13 10:38:32 UTC; 3s ago
root@iscsi-bionic:/home/ubuntu# mount | grep test

2. If media.mount is masked, other mounts are failing:

root@iscsi-bionic:/home/ubuntu# systemctl mask media.mount
Created symlink /etc/systemd/system/media.mount → /dev/null.
root@iscsi-bionic:/home/ubuntu# systemctl start test.mount
Job for test.mount failed.
See "systemctl status test.mount" and "journalctl -xe" for details.
root@iscsi-bionic:/home/ubuntu# systemctl status test.mount | grep Active
   Active: failed (Result: protocol) since Thu 2018-12-13 10:40:06 UTC; 10s ago
root@iscsi-bionic:/home/ubuntu# mount | grep test
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)
root@iscsi-bionic:/home/ubuntu# systemctl stop test.mount
root@iscsi-bionic:/home/ubuntu# systemctl status test.mount | grep Active
   Active: failed (Result: protocol) since Thu 2018-12-13 10:40:06 UTC; 25s ago
root@iscsi-bionic:/home/ubuntu# mount | grep test
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)
root@iscsi-bionic:/home/ubuntu# systemctl start test.mount
Job for test.mount failed.
See "systemctl status test.mount" and "journalctl -xe" for details.
root@iscsi-bionic:/home/ubuntu# mount | grep test
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)
root@iscsi-bionic:/home/ubuntu# systemctl stop test.mount
root@iscsi-bionic:/home/ubuntu# mount | grep test
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)

[Regression potential]

Minimal. Originally, one failing mount point blocked the processing of the rest due to how the return codes were handled for every line in /proc/self/mountinfo. This patch removes this "dependency" and keeps the failure local to the affected mount point, allowing the rest to be processed normally.

[Other Info]

Upstream bug: https://github.com/systemd/systemd/issues/10874
Fixed upstream with commit: https://github.com/systemd/systemd/commit/c165888426ef99440418592a8cdbaff4b7c319b3

[Original Description]

netbooting the bionic live CD[1] over NFS goes straight to maintenance mode :

[1] http://cdimage.ubuntu.com/daily-live/current/

# casper.log
Begin: Adding live session user... ... dbus-daemon[568]: [session uid=999 pid=568] Activating service name='org.gtk.vfs.Daemon' requested by ':1.0' (uid=999 pid=569 comm="" label="unconfined")
dbus-daemon[568]: [session uid=999 pid=568] Successfully activated service 'org.gtk.vfs.Daemon'
dbus-daemon[568]: [session uid=999 pid=568] Activating service name='org.gtk.vfs.Metadata' requested by ':1.0' (uid=999 pid=569 comm="" label="unconfined")
fuse: device not found, try 'modprobe fuse' first
dbus-daemon[568]: [session uid=999 pid=568] Successfully activated service 'org.gtk.vfs.Metadata'

(gvfsd-metadata:580): GUdev-CRITICAL **: 16:28:56.270: g_udev_device_has_property: assertion 'G_UDEV_IS_DEVICE (device)' failed

(gvfsd-metadata:580): GUdev-CRITICAL **: 16:28:56.270: g_udev_device_has_property: assertion 'G_UDEV_IS_DEVICE (device)' failed
A connection to the bus can't be made
done.
Begin: Setting up init... ... done.

[Eric Desrochers (slashd)]

Attaching log from maintenance mode

[Jean-Baptiste Lallement (jibel)]

From the journal

[ 20.311413] ubuntu systemd[1]: sys-kernel-config.mount: Directory /sys/kernel/config to mount over is not empty, mounting anyway.
[ 20.311594] ubuntu systemd[1]: Mounting Kernel Configuration File System...
[ 20.313502] ubuntu mount[813]: mount: /sys/kernel/config: configfs already mounted on /sys/kernel/config.
[ 20.313793] ubuntu systemd[1]: sys-kernel-config.mount: Mount process exited, code=exited status=32
[ 20.313842] ubuntu systemd[1]: sys-kernel-config.mount: Failed with result 'exit-code'.
[ 20.314013] ubuntu systemd[1]: Failed to mount Kernel Configuration File System.
[ 20.325702] ubuntu systemd[1]: Mounting FUSE Control File System...
[ 20.325777] ubuntu mount[814]: mount: /sys/fs/fuse/connections: fusectl already mounted on /sys/fs/fuse/connections.
[ 20.326038] ubuntu systemd[1]: sys-fs-fuse-connections.mount: Mount process exited, code=exited status=32
[ 20.326089] ubuntu systemd[1]: sys-fs-fuse-connections.mount: Failed with result 'exit-code'.
[ 20.326248] ubuntu systemd[1]: Failed to mount FUSE Control File System.

[Taras Prokopenko (tpro)]

For temporary solution see
https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1754777

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Eric Desrochers (slashd)]

The sequence of failure seems to be the following:

-- Unit dev-mqueue.mount has failed.
-- Unit sys-kernel-debug.mount has failed.
-- Unit dev-hugepages.mount has failed.
-- Unit sys-kernel-config.mount has failed.
-- Unit sys-fs-fuse-connections.mount has failed.
-- Unit tmp.mount has failed.
-- Unit local-fs.target has failed.
-- Unit dns-clean.service has failed.
-- Unit systemd-resolved.service has failed.
-- Unit systemd-timesyncd.service has failed.
-- Unit sys-kernel-config.mount has failed.
-- Unit sys-fs-fuse-connections.mount has failed.

# journal -xb:
-- The limits controlling how much disk space is used by the journal may
-- be configured with SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=,
-- RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize= settings in
-- /etc/systemd/journald.conf. See journald.conf(5) for details.
Apr 03 12:59:15 ubuntu systemd-modules-load[758]: Inserted module 'lp'
Apr 03 12:59:15 ubuntu systemd[1]: Failed to set up mount unit: Device or resource busy
Apr 03 12:59:15 ubuntu systemd[1]: dev-mqueue.mount: Mount process finished, but there is no mount.
Apr 03 12:59:15 ubuntu systemd[1]: dev-mqueue.mount: Failed with result 'protocol'.
Apr 03 12:59:15 ubuntu systemd[1]: Failed to mount POSIX Message Queue File System.
-- Subject: Unit dev-mqueue.mount has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit dev-mqueue.mount has failed.

[Eric Desrochers (slashd)]

Seems like this recipe is enough to start gnome (as a potential workaround until this get fix) :

# systemctl mask tmp.mount
# ctrl-d

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[beta-tester (alpha-beta-release)]

is the status "Fix Released" from 2018-04-27 mean,
that the fix is already included in release ubuntu 18.04,
or will it be included to 18.04.1 the first time?

[Stephen Early (steve-greenend)]

I've just checked: a NFS boot image built from the currently released Ubuntu 18.04 still has this bug. I can't see any relevant commits to casper or systemd, either.

[Simone Scisciani (scisciani)]

sorry, by mistake I put the tick "fix released" and I can not put it back in "confirmed". The bug has not yet been fixed

[Eric Desrochers (slashd)]

I set both back to 'Confirmed'.

[Woodrow Shen (woodrow-shen)]

I can confirm that the ubiquity can finish the installation via appending the string of systemd mask services "systemd.mask=dev-hugepages.mount systemd.mask=dev-mqueue.mount systemd.mask=sys-fs-fuse-connections.mount systemd.mask=sys-kernel-config.mount systemd.mask=sys-kernel-debug.mount systemd.mask=tmp.mount" and enter normal Gnome session with user after reboot.

[richud (richud.com)]

Can confirm Woodrow Shen's workaround works for me too, (also tested ok with kubuntu and ubuntu-mate 18.04 automated deploys).

[Woodrow Shen (woodrow-shen)]

I keep doing some experiments (with Dell/HP laptops) and there are some conclusions currently:

1. why does the issue happen (not real root cause)
Due to local-fs.target, the fstab-generator automatically adds dependencies of type Before= to all mount units that refer to local mount points for this target unit. In addition, it adds dependencies of type Wants= to this target unit for those mounts listed in /etc/fstab that have the auto mount option set[1].
Therefore, the emergency shell is triggered by local-fs.target which is dependent on failures of several systemd mounts.

2. provides 2 approaches for workaround fix

1) append "systemd.mask=dev-hugepages.mount systemd.mask=dev-mqueue.mount systemd.mask=sys-fs-fuse-connections.mount systemd.mask=sys-kernel-config.mount systemd.mask=sys-kernel-debug.mount systemd.mask=tmp.mount" into kernel boot options.

2) add "toram" into kernel boot options.
it would completely decompress the filesystem into RAM, which requires 3-4x more RAM, and is hence undesired[2].

3. trade-off between workarounds
Until we find the solution, the better way to workaround is to use "toram" to fix the issue. The reason behind it is we not only get speed-up the installation but also avoid the unstable network with nfs, despite requiring more RAM.

4. The real solution concern
I think the solution may be more complicated if we really want to fix, and ideally we have to consider the cases of normal (e.g. from usb stick) and nfs mount to satisfy the conditions to avoid systemd failure with dependencies or protocol.

[1] https://www.freedesktop.org/software/systemd/man/systemd.special.html
[2] https://wiki.ubuntu.com/BootToRAM

[David Coronel (davecore)]

I confirm the "toram" workaround from Woodrow allows me to PXE netboot the most recent Ubuntu 18.04 Desktop amd64 ISO image.

[Martin Bogomolni (martinbogo)]

The "toram" workaround does not work for me attempting to boot on a SuperMicro X10DRW motherboard with 128GB of ram installed + SATA SSD. I have also added Woodrow Shen's workaround in the command line.

The difference is that I am attempting to install 18.04 server.

Environment is:

Debian "Wheezy" server runnig dnsmasq to provide DHCP and tftp service
Synced/mirrored Ubuntu 18.04 repository being served via nginx

-----

May 29 18:08:13 ubuntu systemd[1]: dev-mqueue.mount: Mount process finished, but there is no mount.
May 29 18:08:13 ubuntu systemd[1]: dev-mqueue.mount: Failed with result 'protocol'.
May 29 18:08:13 ubuntu systemd[1]: Failed to mount POSIX Message Queue File System.
-- Subject: Unit dev-mqueue.mount has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit dev-mqueue.mount has failed.

[Woodrow Shen (woodrow-shen)]

"toram" option only affected desktop with casper.

[beta-tester (alpha-beta-release)]

"toram" works to me and will also fix the hanging at shutdown/reboot.
but "toram" takes more that twice as long to boot into the desktop (~3minutes instead of ~1minute).

[Jon (jmkloz)]

Appears to still be affecting 18.04.1

[Skye K (skyebuddha)]

I am having a similar issue in 18.04 and 18.04.1. I am trying to netboot and retrieve the install files from nfs or http. I have seen a few others have similar issues: https://askubuntu.com/a/1031024/837404 . When I try to install, it says that I am missing files, but when I check the apache logs, nothing has been requested and also no errors show in the console

[thh (thh01217)]

Based on in-depth analysis, I found the cause of the error：

“Apr 03 12:59:15 ubuntu systemd[1]: Failed to set up mount unit: Device or resource busy”

call tree on systemd mount.c & unit.c:
 mount_dispatch_io
  -> mount_load_proc_self_mountinfo
     -> mount_setup_unit
         -> mount_setup_existing_unit
             -> mount_add_extras
                 -> unit_set_default_slice:
                      -> unit_set_slice:
                         if (unit_active_state(u) != UNIT_INACTIVE)
                              return -EBUSY;

"unit_set_slice" return EBUSY always, because of nfsroot always active state in netbooting,

"mount_dispatch_io" give up updating mount state when "mount_load_proc_self_mountinfo" return the error.

finally, all systemd mount service failed and then goto emergency shell.

[lepidas blades rompolos (lepidas)]

This bug is still affecting me, downloaded yesterday iso images from ubuntu,kubuntu,xubuntu and can't boot from pxe server saying that iam in emergency mode.

[Lukas (lukas-wringer)]

is this bug lost or not assigned to someone? It is still broken, of course there are workarounds but no one knows the exact consequences of them?

[Eric Desrochers (slashd)]

I started to look at this problem from scratch since it's been a while since I have reported it....

It seems to go into emergency mode due to a failed attempt to start unit "tmp.mount" :

# /var/log/boot.log
65 emergency.target: Enqueued job emergency.target/start as 159
66 tmp.mount: Unit entered failed state.

Adding "systemd.mask=tmp.mount" in /tftpboot/pxelinux.cfg/default as a parameter did the trick to workaround the behaviour :

APPEND initrd=bionic-desktop-amd64/initrd root=/dev/nfs boot=casper netboot=nfs nfsroot=192.168.100.2:/bionic-desktop-amd64 splash systemd.mask=tmp.mount systemd.debug-shell=1 systemd.log_level=debug systemd.log_target=console console=ttyS0,38400 console=tty1 --

Note:
- I did the test by curiosity w/ Artful/17.10 (systemd-234) and it works, so it's possibly something between v234 and v237 which introduced the behaviour for tmp.mount, a change in mount, ...
- Problem is also reproducible in Cosmic, and journalctl was a little bit more verbose in Cosmic than it was for Bionic in my testing :

$ journalctl -a -u tmp.mount
-- Logs begin at Wed 2018-10-10 20:15:36 UTC, end at Wed 2018-10-10 20:15:43 UTC. --
Oct 10 20:15:36 ubuntu systemd[1]: tmp.mount: Directory /tmp to mount over is not empty, mounting anyway.
Oct 10 20:15:36 ubuntu systemd[1]: Mounting /tmp...
Oct 10 20:15:36 ubuntu systemd[1]: tmp.mount: Mount process finished, but there is no mount.
Oct 10 20:15:36 ubuntu systemd[1]: tmp.mount: Failed with result 'protocol'.
Oct 10 20:15:36 ubuntu systemd[1]: Failed to mount /tmp.

# src/core/mount.c
802 static void mount_enter_dead(Mount *m, MountResult f) {
803 assert(m);
804
805 if (m->result == MOUNT_SUCCESS)
806 m->result = f;
807
808 if (m->result != MOUNT_SUCCESS)
809 log_unit_warning(UNIT(m), "Failed with result '%s'.", mount_result_to_string(m->result));
...
1282 switch (m->state) {
1283
1284 case MOUNT_MOUNTING:
1285 /* Our mount point has not appeared in mountinfo. Something went wrong. */
1286
1287 if (f == MOUNT_SUCCESS) {
1288 /* Either /bin/mount has an unexpected definition of success,
1289 * or someone raced us and we lost. */
1290 log_unit_warning(UNIT(m), "Mount process finished, but there is no mount.");
1291 f = MOUNT_FAILURE_PROTOCOL;
1292 }

and m->result is indeed equalt to "MOUNT_FAILURE_PROTOCOL" ^

1955 [MOUNT_FAILURE_PROTOCOL] = "protocol",

I'll try to instrument things and create a custom ISO for further debugging/testing. This is where am at the moment.

- Eric

[Eric Desrochers (slashd)]

So far I highly suspect this commit[1] to be the possible offending one and it would "fit" with the bionic systemd version in Ubuntu[2] vs upstream introduction of the change :

$ git describe --contains 006aabbd05
v237~47^2~2

[1] 006aabbd0 mount: mountinfo event is supposed to always arrive before SIGCHLD

[2] rmadison
 systemd | 237-3ubuntu10 | bionic | source, amd64, arm64, armhf, i386, ppc64el, s390x
 systemd | 237-3ubuntu10.3 | bionic-updates | source, amd64, arm64, armhf, i386, ppc64el, s390x

[Eric Desrochers (slashd)]

With systemd on Xenial & Artful, there is no instruction in the case of MOUNT_MOUNTING which re-enforce why it is working with these releases :

# src/core/mount.c
1182 case MOUNT_MOUNTING:
1183 case MOUNT_MOUNTING_DONE:
1184 case MOUNT_MOUNTING_SIGKILL:
1185 case MOUNT_MOUNTING_SIGTERM:
1186
1187 if (f == MOUNT_SUCCESS)
1188 mount_enter_mounted(m, f);
1189 else if (m->from_proc_self_mountinfo)
1190 mount_enter_mounted(m, f);
1191 else
1192 mount_enter_dead(m, f);
1193 break;

So most likely, falls in the MOUNT_MOUNTING_SIGTERM and then break the case statement flow.

# src/basic/unit-def.h
MOUNT_MOUNTING, /* /usr/bin/mount is running, but the mount is not done yet. */

[Eric Desrochers (slashd)]

Additionally,

tmp.mount unit configuration :
https://github.com/systemd/systemd/blob/master/units/tmp.mount

# tmp.mount
--
..
ConditionPathIsSymbolicLink=!/tmp
..
--

ConditionPathIsSymbolicLink = Verifies whether a certain path exists and is a symbolic link.
When there is an exclamation mark ("!"), the validation is negated

For "ConditionPathIsSymbolicLink=!/tmp" the unit is making sure /tmp doesn't exist and is not a symbolink link, if it exist and is a symbolic link like then it will fail and create the actual situation.

[Eric Desrochers (slashd)]

The "existing" /tmp come from casper code :

bionic/casper-1.394/scripts/casper-bottom/12fstab
...
cat > $FSTAB <<EOF
${UNIONFS} / ${UNIONFS} rw 0 0
tmpfs /tmp tmpfs nosuid,nodev 0 0
EOF
...

[Eric Desrochers (slashd)]

By reading this article : https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems/

I really start thinking the easiest way to fix it is as describe here :

...
/tmp as location for volatile, temporary userspace file system objects (X)
...
It is possible to disable the automatic mounting of some (but not all) of these file systems, if that is required. These are marked with (X) in the list above. You may disable them simply by masking them:
systemctl mask dev-hugepages.mount
...

I have tested by masking "tmp.mount", but the official documentation recommend "dev-hegepages.mount" instead.

pxe configuration line :
"
APPEND initrd=bionic-desktop-amd64/initrd root=/dev/nfs boot=casper netboot=nfs nfsroot=192.168.100.2:/bionic-desktop-amd64 splash systemd.mask=tmp.mount systemd.debug-shell=1 systemd.log_level=debug systemd.log_target=console console=ttyS0,38400 console=tty1 --
"

I'm starting to think that this may become the final solution to the new systemd behaviour as indicated in official documentation above and not only a workaround.

Thoughts ?

[Eric Desrochers (slashd)]

Does someone impacted can test the dev-hugepages.mount masking within their PXE configuration and let me know how it works ?

systemd.mask=dev-hugepages.mount

[Eric Desrochers (slashd)]

According to the documentation, the recommendation is to mask dev-hugepages.mount, but in my current test doing it was preventing the tmp.mount failure, but not preventing to go into "Emergency mode". In my lab masking tmp.mount had better result.

Feel free to try both and let me know the outcome, results may varies from one setup to another.

- Eric

[Marcel Partap (empee584)]

mmh #32 not enough.. I had to convert it /tmp to an overlay mount, as regardless how early, it seems there'll be some important file (custom_mounts.list, see /var/log/live/boot) on it already..

[Brian Nelson (bhnelson)]

Eric,

The 'recommendation' for masking dev-hugepages you site from that wiki page is clearly just an example of how you could disable one of the various mounts described there. I don't think it's a recommendation to fix anything in particular.

FWIW: Masking dev-hugepages doesn't seem to help much for me. Masking tmp seems to let the system boot up, but still the other mount services fail and systemd status is red 'degraded.'

I've ended up masking all affected mounts (per comment 12 and 14) with the addition of masking run-rpc_pipefs.mount too. This lets the systemd boot up to green 'running' state.

I'm still having problems logging into Gnome with a user with NFS home. I'm not sure if that's related to this issue or something else though. Still looking at that.

I think you're on the right track in comment 27. I get the feeling that somewhere along the line a result of 'this is already mounted' changed from a success to a failure in systemd, possibly due to the change in mount.c you pasted.

[beta-tester (alpha-beta-release)]

@Eric,
i just tested what you suggested at your comment #31 with Ubuntu 18.10 release ...
    KERNEL http://pxe-server/nfs/ubuntu-x64/casper/vmlinuz
    INITRD http://pxe-server/nfs/ubuntu-x64/casper/initrd
    APPEND nfsroot=192.168.1.1:/srv/nfs/ubuntu-x64 ro netboot=nfs file=/cdrom/preseed/ubuntu.seed boot=casper systemd.mask=dev-hugepages.mount -- debian-installer/language=de console-setup/layoutcode=de keyboard-configuration/layoutcode=de keyboard-configuration/variant=German

... but then i again run straight into emergency console.
(not sure, if that information still is helpful)

at the moment "systemd.mask=tmp.mount" still is the best solution.
but with the issues of:
- having few red "FAILED" messages at boot time;
- at reboot/poweroff often run into "stop jobs" are running endless at shutting down;
- or hangs for ever at "Starting Shuts down the 'live' preinstalled system clearly...";

the second best solution is "toram", there i never observed the issues of any "FAILED" message and never hat the issues at reboot/poweroff (maybe there are race conditions - "toram" nothing has to be loaded fron nfs via network).
but "toram" takes a lot of time, because the whole squashfs-image has to be loaded into ram first, before it can be mounted.

BTW: lununtu 18.10 shows the same behavior as ubuntu 18.10, but it does not show up the issues with reboot/poweroff hanging or endles running stop jobs.

[Brian Nelson (bhnelson)]

So I've found a complete work-around for this. I also found that this issue is NOT new in 18.04 as it also affects 16.x (and likely 15 and 17 too). However it is DIFFERENT in 18.04. More details below.

TL;DR:
You need to netboot with an initramfs that doesn't have 'scripts/casper-bottom/25disable_cdrom.mount' in it. This script masks the dynamically-generated cdrom.mount systemd unit (where the NFS mount goes). That causes all the issues described in this bug.

From whatever machine where netboot initramfs is created:

# Disable/block the problem script
mkdir -p /etc/initramfs-tools/scripts/casper-bottom
touch /etc/initramfs-tools/scripts/casper-bottom/25disable_cdrom.mount

# rebuild initramfs
update-initramfs -u

# Move/copy the new file to the netboot server

The issue here is that systemd isn't able to update its mount status properly. In the case of 18.04, all of the 'failed' mounts are actually successfully mounted. This includes /tmp. BUT systemd doesn't recognize that fact and marks them all as red/failed.

In 16.04 this issue is a bit different. When booting, all of the same mounts are again mounted successfully AND systemd shows them all as green/active. BUT if you try to stop/unmount any of them you will see a similar situation. The unmount will actually succeed, but systemd will report an unmount failure and continue to show the unit as green/active.

Per the call trace thh noted in comment #21:
From what I can tell, mount_load_proc_self_mountinfo iterates through every active mount on the system (some perhaps more than once). When it gets to the nfs-mount on /cdrom, it does fail in unit_set_slice and generate the "Failed to set up mount unit: Device or resource busy" error. For whatever reason, that failure seems to completely bork systemd's ability to update its mount status. Thus mounts get 'stuck' either mounted or not from systemd's perspective.

The failure seems to be caused by the fact that the cdrom.mount unit (NFS mount) is masked. Once it's unmasked the failure doesn't occur and all mounts work as expected. You can actually observe this from within a 'broken' boot at the emergency prompt:
rm /lib/systemd/system/cdrom.mount
systemctl daemon-reload
umount /tmp (ensure it's gone, there may be multiple mounts)
systemctl reset-failed tmp.mount
systemctl start tmp.mount
..and it will succeed

I did verify this issue by actually booting from a 'real' DVD and the problem doesn't happen there. It's something specific to having the image mounted over NFS and masking it's unit.

For reference, the disable_cdrom.mount script was the solution for this bug
https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1436715

[Robert Giles (rgiles)]

Brian,

Thanks for sleuthing out a fix for this; I wanted to add that this also seems to work for netbooting 18.10.

[Douglas Hammond (douglas-e-hammond)]

This issue also affected Linux Mint 19.2, and Brian's workaround and fix in #36 work for that distribution as well.

I used the workaround steps to resume the broken boot, then followed the fix steps and rebuilt the initramfs using:

# rebuild initramfs
/usr/sbin/update-initramfs.distrib -u

then copied the resulting .img to my netboot server.

[beta-tester (alpha-beta-release)]

hello Victor (vtapia),
does it mean the next comming ubuntu Live release 18.04.2 sould PXE boot without any tweak?
is it already in the "Ubuntu 19.04 (Disco Dingo) Daily Build"?

[Victor Tapia (vtapia)]

Hi,

I'm working on the patches for Disco, Cosmic, Bionic and Xenial. It's not in Disco yet, but the fix process will be tracked in this bug.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "disco-stop-mount-error-propagation.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[beta-tester (alpha-beta-release)]

i just tried out the daily build of
http://cdimage.ubuntu.com/daily-live/pending/disco-desktop-amd64.iso
from 2019-01-31 07:45 to pxe boot wihout using the "systemd.mask=tmp.mount" workaround...
but i still goes straight to maintenance mode.

is the fix not included to the daily build yet?

[Dimitri John Ledkov (xnox)]

No, it would have been impossible for any desktop image (even pending) to contain the new systemd on the 31st of January.

The first image that contains the new systemd is from 20190204 or later.

[beta-tester (alpha-beta-release)]

just tried http://cdimage.ubuntu.com/daily-live/pending/disco-desktop-amd64.iso from 2019-02-08 without the workaround and it pxe boots just fine.
thank you very much!!!

will the Ubuntu 18.04 LTS (Bionic Beaver) receive that fix in the next point release as well?

[Dimitri John Ledkov (xnox)]

I'm afraid it just missed it, i think. But we can try.

[Guillermo (guillermo-etsetb)]

Using Brian's workaround (#36) I'm able to netboot. However, raising the network interfaces fails. Is it related?

journalctl shows:
ubuntu ifup[3055]: Error: any valid prefix is expected rather than "dhcp/dhcp"

Networking does partially work: pinging IP addresses work, but DNS resolution doesn't.

[beta-tester (alpha-beta-release)]

@Guillermo, i have the same issue with ubuntu 19.04 daily (pending) from 2019-02-14
(without using the workaround) when pxe booting.
/var/log/syslog shows the folowing lines, when i serch for resolv:
```
Feb 14 17:53:27 ubuntu systemd[1]: Starting Restore /etc/resolv.conf if the system crashed before the ppp link was shut down...
Feb 14 17:53:27 ubuntu systemd[1]: Started Restore /etc/resolv.conf if the system crashed before the ppp link was shut down.
Feb 14 17:53:27 ubuntu avahi-daemon[1225]: Failed to open /etc/resolv.conf: Invalid argument
Feb 14 17:53:27 ubuntu kernel: [ 6.498832] Key type dns_resolver registered
Feb 14 17:53:27 ubuntu kernel: [ 6.985154] ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT0._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330)
Feb 14 17:53:27 ubuntu kernel: [ 6.985719] ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT0._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330)
Feb 14 17:53:27 ubuntu kernel: [ 6.986106] ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT2._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330)
Feb 14 17:53:27 ubuntu kernel: [ 6.988112] ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT2._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330)
Feb 14 17:53:27 ubuntu kernel: [ 7.347354] ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT4._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330)
Feb 14 17:53:27 ubuntu kernel: [ 7.357711] ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT4._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330)
Feb 14 17:53:28 ubuntu sh[1400]: grep: /etc/resolv.conf: No such file or directory
Feb 14 17:53:29 ubuntu systemd-resolved[1604]: Positive Trust Anchors:
Feb 14 17:53:29 ubuntu systemd-resolved[1604]: . IN DS 19036 8 2 ...
Feb 14 17:53:29 ubuntu systemd-resolved[1604]: . IN DS 20326 8 2 ...
Feb 14 17:53:29 ubuntu systemd-resolved[1604]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Feb 14 17:53:29 ubuntu systemd-resolved[1604]: Using system hostname 'ubuntu'.
Feb 14 17:53:29 ubuntu systemd-resolved[1604]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Feb 14 17:53:30 ubuntu NetworkManager[1405]: <info> [1550166810.1729] Read config: /etc/NetworkManager/NetworkManager.conf (lib: 10-dns-resolved.conf, 10-globally-managed-devices.conf, 20-connectivity-ubuntu.conf, no-mac-addr-change.conf) (run: netplan.conf) (etc: default-wifi-powersave-on.conf)
Feb 14 17:53:30 ubuntu NetworkManager[1405]: <info> [1550166810.4138] dns-mgr[0x55dc42397170]: init: dns=systemd-resolved rc-manager=symlink, plugin=systemd-resolved
Feb 14 17:54:29 ubuntu systemd-resolved[1604]: Using degraded feature set (UDP) for DNS server 192.168.0.1.
```

PS: long time ago ...

[Guillermo (guillermo-etsetb)]

@beta-tester I can finally boot with network using the workaround in 18.04.2.

The workaround is, however, a bit difficult to apply, unless there is another way to do it. unmkinitramfs will only extract one of the 2 embedded microcodes and thus I'm extracting the initrd manually.

[Alkis Georgopoulos (alkisg)]

A workaround to avoid rebuilding the initramfs is to use TWO initramfs's. The second one contains just a symlink from scripts/casper-bottom/25disable_cdrom.mount to /bin/true. I.e.:

cd $(mktemp -d)
mkdir -p scripts/casper-bottom/
ln -s /bin/true scripts/casper-bottom/25disable_cdrom.mount
find . | cpio -o -H newc | gzip > initrd-1755863.img

Copy the generated initrd-1755863.img to your TFTP/HTTP server, and pass it in the kernel cmdline. In iPXE vernacular:

kernel /images/cdrom/casper/vmlinuz initrd=initrd.lz initrd=initrd-1755863.img boot=casper netboot=nfs nfsroot=10.161.254.11:/cdrom file=/cdrom/preseed/ubuntu.seed --
initrd /images/cdrom/casper/initrd.lz
initrd /liveclone/initrd-1755863.img

[Steve Langasek (vorlon)]

Hello Eric, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Eric, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Eric, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dan Streetman (ddstreet)]

cosmic:

root@systemd-c:~# dpkg -l | grep libsystemd0
ii libsystemd0:amd64 239-7ubuntu10.8 amd64 systemd utility library
root@systemd-c:~# systemctl mask media.mount
Created symlink /etc/systemd/system/media.mount → /dev/null.
root@systemd-c:~# systemctl start test.mount
Job for test.mount failed.
See "systemctl status test.mount" and "journalctl -xe" for details.

root@systemd-c:~# dpkg -l | grep libsystemd0
ii libsystemd0:amd64 239-7ubuntu10.9 amd64 systemd utility library
root@systemd-c:~# systemctl mask media.mount
Created symlink /etc/systemd/system/media.mount → /dev/null.
root@systemd-c:~# systemctl start test.mount
root@systemd-c:~# mount | grep test
tmpfs on /test type tmpfs (rw,relatime)

[Dan Streetman (ddstreet)]

bionic:

root@systemd-b:~# dpkg -l | grep libsystemd0
ii libsystemd0:amd64 237-3ubuntu10.13 amd64 systemd utility library
root@systemd-b:~# systemctl mask media.mount
Created symlink /etc/systemd/system/media.mount → /dev/null.
root@systemd-b:~# systemctl start test.mount
Job for test.mount failed.
See "systemctl status test.mount" and "journalctl -xe" for details.

root@systemd-b:~# dpkg -l | grep libsystemd0
ii libsystemd0:amd64 237-3ubuntu10.14 amd64 systemd utility library
root@systemd-b:~# systemctl mask media.mount
Created symlink /etc/systemd/system/media.mount → /dev/null.
root@systemd-b:~# systemctl start test.mount
root@systemd-b:~# mount | grep test
tmpfs on /test type tmpfs (rw,relatime)

[Łukasz Zemczak (sil2100)]

Hello Eric, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Eric, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dan Streetman (ddstreet)]

cosmic:

root@lp1755863-c:~# dpkg -l | grep systemd0
ii libsystemd0:amd64 239-7ubuntu10.8 amd64 systemd utility library
root@lp1755863-c:~# mount | grep test
root@lp1755863-c:~# systemctl mask media.mount
Created symlink /etc/systemd/system/media.mount → /dev/null.
root@lp1755863-c:~# systemctl start test.mount
Job for test.mount failed.
See "systemctl status test.mount" and "journalctl -xe" for details.

root@lp1755863-c:~# dpkg -l | grep systemd0
ii libsystemd0:amd64 239-7ubuntu10.10 amd64 systemd utility library
root@lp1755863-c:~# mount | grep test
root@lp1755863-c:~# systemctl mask media.mount
Created symlink /etc/systemd/system/media.mount → /dev/null.
root@lp1755863-c:~# systemctl start test.mount
root@lp1755863-c:~# mount | grep test
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)

[Dan Streetman (ddstreet)]

bionic:

root@lp1755863-b:~# dpkg -l | grep systemd0
ii libsystemd0:amd64 237-3ubuntu10.13 amd64 systemd utility library
root@lp1755863-b:~# mount | grep test
root@lp1755863-b:~# systemctl mask media.mount
Created symlink /etc/systemd/system/media.mount → /dev/null.
root@lp1755863-b:~# systemctl start test.mount
Job for test.mount failed.
See "systemctl status test.mount" and "journalctl -xe" for details.

root@lp1755863-b:~# dpkg -l | grep systemd0
ii libsystemd0:amd64 237-3ubuntu10.15 amd64 systemd utility library
root@lp1755863-b:~# mount | grep test
root@lp1755863-b:~# systemctl mask media.mount
Created symlink /etc/systemd/system/media.mount → /dev/null.
root@lp1755863-b:~# systemctl start test.mount
root@lp1755863-b:~# mount | grep test
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)

[Dan Streetman (ddstreet)]

xenial (note, the test case repro is slightly different for x, the systemctl stop fails instead of the start):

root@lp1755863-x:~# dpkg -l | grep systemd0
ii libsystemd0:amd64 229-4ubuntu21.16 amd64 systemd utility library
root@lp1755863-x:~# mount | grep test
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)
root@lp1755863-x:~# systemctl mask media.mount
Created symlink from /etc/systemd/system/media.mount to /dev/null.
root@lp1755863-x:~# systemctl stop test.mount
Job for test.mount failed. See "systemctl status test.mount" and "journalctl -xe" for details.

root@lp1755863-x:~# dpkg -l | grep systemd0
ii libsystemd0:amd64 229-4ubuntu21.17 amd64 systemd utility library
root@lp1755863-x:~# mount | grep test
tmpfs on /test type tmpfs (rw,nosuid,nodev,relatime)
root@lp1755863-x:~# systemctl mask media.mount
Created symlink from /etc/systemd/system/media.mount to /dev/null.
root@lp1755863-x:~# systemctl stop test.mount
root@lp1755863-x:~# mount | grep test

[Dan Streetman (ddstreet)]

autopkgest failures:

xenial:

snapd version 2.34.2ubuntu0.1 autopkgtests always failed. ignore.

linux: tests very flaking, usually fail; ignore.

systemd (s390x): tests always failed, ignore.

linux-oracle (amd64): tests always failed (except once), ignore.

nplan (s309x/armhf): tests failed for long time, ignore.

bionic:

systemd: tests always failed, ignore.

linux-oracle (amd64): tests always failed (except twice), ignore.

linux (i386): tests always failed, ignore.

mariadb-10.1: tests always failed, ignore.

(re-running tests linux-gcp-edge, umockdev, lxc, util-linux, munin)

cosmic:

gvfs (s390x): tests always failed, ignore.

(re-running tests linux, gvfs, upower, suricata, asterisk, systemd, apt)

[Dan Streetman (ddstreet)]

autopkgtest failures:

bionic:

linux-gcp-edge: fails due to bug 1723223, ignore.

(remaining re-run tests all passed)

cosmic:

linux still re-running, re-ran gvfs/amd64, asterisk, systemd again

(remaining re-run tests all passed)

[Dan Streetman (ddstreet)]

autopkgtest failures:

cosmic:

linux (amd64/i386): re-ran, but still failed; tests fail majority of the time; ignore.

systemd (ppc64el/arm64): re-ran, but still failed; tests fail majority of the time; ignore.

all autopkgtest regression failures re-ran to succcess or explained/justified in above comments.

[Dan Streetman (ddstreet)]

note this is listed in pending-sru as being at 5 days aging, however it was actually uploaded 9 days ago, then re-uploaded mid-last-week with the patches for bug 1812760 removed; so the patches for this bug have been in the systemd in -proposed for 9 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 239-7ubuntu10.10

---------------
systemd (239-7ubuntu10.10) cosmic; urgency=medium

  [ Victor Tapia ]
  * d/p/stop-mount-error-propagation.patch:
    keep mount errors local to the failing mount point instead of
    blocking the processing of all mounts (LP: #1755863)

 -- Dan Streetman <email address hidden> Thu, 28 Feb 2019 14:29:48 -0500

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 237-3ubuntu10.15

---------------
systemd (237-3ubuntu10.15) bionic; urgency=medium

  [ Victor Tapia ]
  * d/p/stop-mount-error-propagation.patch:
    keep mount errors local to the failing mount point instead of blocking
    the processing of all mounts (LP: #1755863)

 -- Dan Streetman <email address hidden> Thu, 28 Feb 2019 16:03:40 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-4ubuntu21.17

---------------
systemd (229-4ubuntu21.17) xenial; urgency=medium

  [ Victor Tapia ]
  * d/p/stop-mount-error-propagation.patch:
    keep mount errors local to the failing mount point instead of blocking
    the processing of all mounts (LP: #1755863)

  [ Eric Desrochers ]
  * d/p/fix-egde-case-when-processing-proc-self-mountinfo.patch:
    Mounting any file system to a mount point in a directory
    that is bind mounted to itself will create an inactive
    mount unit. (LP: #1795764)

 -- Dan Streetman <email address hidden> Thu, 28 Feb 2019 17:50:50 -0500