It will prompt "Failed to unmount /oldroot" when shutdown or reboot

[Bin Li]

Summary]
[ThinkPad]It will prompt error message after shutdown or reboot.

[Reproduce steps]
1. Preload Ubuntu OS 20.04.2
2. Boot and enter to the os
3. Shutdown or reboot the machine, it wouldn't show below error by default.
4. Then installed finalrd, and do reboot or shutdown.

Expected Result:
 It will not prompt error message after shutdown or reboot.
Actual result:
 It will prompt error message after shutdown or reboot as the attached 'the page after shutdown or reboot.jpg'.

 sd-umoun[2406]: Failed to unmount /oldroot: Device or resource busy
 sd-umoun[2407]: Failed to unmount /oldroot/dev/pts: Device or resource busy
 sd-umoun[2408]: Failed to unmount /oldroot/dev: Device or resource busy
 shutdown[1]: Failed to finalize file systems, ignoring

 We met this issue on several OEM projects, such as ThinkPad-P17-Gen-1, ThinkPad X13.

[Additional information]
Platform: ThinkPad X1 Titanium Gen1, ThinkPad X13, ThinkPad P17
systemd:245.4-4ubuntu3.11
finalrd: 6~ubuntu20.04.1

[Bin Li (binli)]

https://github.com/systemd/systemd/issues/20516

Get the feedback from upstream, it looks not related to systemd issue.

@xnox,

 Do you have any idea about this issue? Thanks!

[Bin Li (binli)]

When I removed 'splash' parameter from grub, I couldn't find this error any more, is it possible the 'splash' used the /oldroot/dev/pts ? Thanks!

[Bin Li (binli)]

I read the source code of finalrd, when the system shutdown or reboot, it will copy the systemd-shutdown, /bin/sh and needed .so files, but I'm not sure when systemd switch to /run/initramfs/shutdown to finish shutdown.
I thought the error messages is from /run/initramfs/shutdown, it will umount /oldroot/dev/pts, but the device is busy.

I tried to umount /dev/pts before shutdown, the /oldroot/dev/pts is gone, but the /oldroot/dev is still there, there are too much devices node at /dev, I couldn't remove them to verify the error.

[Dimitri John Ledkov (xnox)]

With "splash" in cmdline, on shutdown, desktop images start plymouth with graphical splash on a shutdown TTY such that one should be seeing animation (and any graphical or text messages should be hidden from the user). One should be able to use alt-ctrl-arrowkeys to switch back to tty1 to still see messsages if one desires. Plymouth should continue to run.

normally on shutdown systemd-shutdown binary is called from the regular rootfs. It tries to kill and unmount all the things, and after it stops making any progress, it reexecs systemd-shutdown brinamy from /run/initramfs/shutdown.

That, in turn, is systemd-shutdown binary, _again_. But this time it has been performed with pivot root. Meaning the regular rootfs is now mounted as /oldroot/ and things are held up there.

I wonder if it is plymouth that is hodling up /dev/pts and like udevd holding up /dev. And if killing those in shutdown hooks is appropriate. Or for example running plymouth from /run/initramfs, rather than from rootfs.

[Bin Li (binli)]

@xnox,

 Yes, it looks related to plymouth, I tried the edit the plymouth-reboot.service to let it hide splash when reboot, it won't show error /oldroot messages, it looks like same with no 'splash' parameter.

In /lib/systemd/system/plymouth-reboot.service
ExecStartPost=-/bin/plymouhth hide-splash

I also tried the same way on plymouth-halt.service or plymouth-poweroff.service, but it didn't work.

And I also tried add the killall in /run/initramfs/, and add the hook to call 'killall plymouth', it's very bad, the message on screen is distorted, and a kernel crash. XD
So killing the plymouth is not okay.

Next step, I will try to move the plymouth into /run/initramfs.

[Bin Li (binli)]

@xnox,

 I tried to enable the plymouth and plymouthd in finalrd, after test, it could reboot without error message in low rate, most of time the system will hang at splashing cursor, and sometimes kernel crash. I doubt it might be the race condition between threads when shutdown. The good point is that I never saw /oldroot message again, it might be the right way to fix it, do you have more idea, or did I miss something? Thanks!

--- /usr/bin/finalrd 2020-09-10 22:53:09.000000000 +0800
+++ finalrd 2021-09-02 21:40:43.510819136 +0800
@@ -26,7 +26,7 @@
 # will unmount all the things, and run our hooks

 LD=`grep -Eow '/.*ld-.*.so' /proc/self/maps | head -1`
-for bin in /bin/sh /lib/systemd/systemd-shutdown
+for bin in /bin/sh /lib/systemd/systemd-shutdown /usr/bin/plymouth /usr/sbin/plymouthd
 do
     rm -f /run/finalrd-libs.conf
     touch /run/finalrd-libs.conf
--- /usr/lib/finalrd/finalrd-static.conf 2020-09-10 22:53:09.000000000 +0800
+++ finalrd-static.conf 2021-09-02 21:40:30.104127176 +0800
@@ -15,3 +15,5 @@
 C /run/initramfs/etc/ld.so.conf - - - - /etc/ld.so.conf
 C /run/initramfs/etc/ld.so.cache - - - - /etc/ld.so.cache
 C /run/initramfs/etc/ld.so.conf.d - - - - /etc/ld.so.conf.d
+C /run/initramfs/usr/bin/plymouth - - - - /bin/plymouth
+C /run/initramfs/usr/sbin/plymouthd - - - - /sbin/plymouthd

[Bin Li (binli)]

@xnox,

 I checked the mdadm source code, in systemd it has provided mdadm.shutdown to stop raid service, should we use the mdadm.finalrd now? From the changelog I couldn't find any history, just 'Ship finalrd hook' in each release, :D. Could you give me some background about the mdadm.finalrd? Our customer have a strong request to dismiss these error messages, if we disable finalrd.service, is there any risk? Thanks a lot!

$ cat mdadm.finalrd
mdmon --all --takeover
mdadm -vv --wait-clean --scan
mdadm -vv --stop --scan

$ cat systemd/mdadm.shutdown
#!/bin/sh
# We need to ensure all md arrays with external metadata
# (e.g. IMSM, DDF) are clean before completing the shutdown.
BINDIR/mdadm --wait-clean --scan

[Bin Li (binli)]

+mdadm (4.0-2) unstable; urgency=medium
+
+ * Ship mdadm-shutdown.service and suggest dracut-core. Users of systemd
+ with rootfs on Intel Matrix Raid and DDF external metadata-raid arrays
+ that require mdmon monitoring, may wish to install dracut-core package
+ and enable mdadm-shutdown.service. This will create a shutdown
+ initramfs, that systemd-shutdown can pivot to. This may result in an
+ improved shutdown behaviour with less hangs and synced raid
+ arrays. The generated initramfs will takeover mdmon monitoring, wait
+ for the arrays to be clean before stopping them and unmounting
+ everything and finally executing requested shutdown command.
+ * Bump standards version, no changes required.
+ * Bump debhelper to 9.
+ * Add second email address to avoid NMU warnings.
+
+ -- Dimitri John Ledkov <email address hidden> Tue, 10 Oct 2017 11:55:52 +0100

[Bin Li (binli)]

A solution from Daniel.

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1928200/comments/11

[Bin Li (binli)]

I installed 21.10(impish), I found this issue is gone, there are no error messages.

mdadm 4.2~rc2-2ubuntu1
finalrd 8
systemd 248.3-1ubuntu8

[Bin Li (binli)]

Verified on 22.04, this issue is fixed.
And on 20.04, I installed finalrd 9 and plymouth 0.9.5, this issue is also gone.

Here is the patch for finalrd, but for plymouth, there are so many changes from 0.9.4, so I gave up to find the patches, let's fix this issue on 22.04.

diff --git a/finalrd.service b/finalrd.service
index e67d177..b31705f 100644
--- a/finalrd.service
+++ b/finalrd.service
@@ -6,6 +6,7 @@ Documentation=man:finalrd(1)
 After=local-fs.target boot.mount boot.automount
 Wants=local-fs.target
 Conflicts=shutdown.target umount.target
+Before=shutdown.target
 DefaultDependencies=no

 [Service]
diff --git a/finalrd.sh b/finalrd.sh
index cffa94b..f623f8c 100755
--- a/finalrd.sh
+++ b/finalrd.sh
@@ -25,12 +25,11 @@ mount -o remount,exec /run
 # our shutdown sequence is to be controled by systemd-shutdown which
 # will unmount all the things, and run our hooks

-LD=`grep -Eow '/.*ld-.*.so' /proc/self/maps | head -1`
 for bin in /bin/sh /lib/systemd/systemd-shutdown
 do
     rm -f /run/finalrd-libs.conf
     touch /run/finalrd-libs.conf
- for lib in `$LD --list $bin | grep -Eow "/.* "`
+ for lib in `LD_TRACE_LOADED_OBJECTS=1 $bin | grep -Eow "/.* "`
     do
         if [ "$lib" = '=>' ]
         then

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.