Prompt error message "Failed to unmount /oldroot" when shutdown or reboot

[Bin Li]

[impact]

root fs is not cleanly unmounted on shutdown

[test case]

create a file in /etc/binfmt.d/TestFormat.conf with the following content:

:TestFormat:M::XX::/bin/true:F

reboot the system, and then shutdown or reboot again, and watch the serial console for the error message:

sd-umount[1334]: Failed to unmount /oldroot: Device or resource busy

[regression potential]

any regression would likely cause problems on shutdown and/or reboot, or may cause problems with unmounted root fs on shutdown/reboot

[scope]

this is needed only for f

this is fixed upstream with PR 15566 which is included in v246, so this is fixed already in g and later

this isn't reproducable on b

[original description]

On ubuntu 20.04, with latest kernel and systemd, it will show error on ThinkPad X1.

sd-umount[1334]: Failed to unmount /oldroot: Device or resource busy

systemd 245.4-4ubuntu3.6
kernel: 5.8.0-53

[Bin Li (binli)]

https://github.com/systemd/systemd/issues/14981

[Bin Li (binli)]

Tried to back-port 4 patches from #2, it didn't work for me.

[Brian Murray (brian-murray)]

Hello Bin, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Bin Li (binli)]

Checked the proposed channel, the 3.8 was not ready yet, will check it later.

[Bin Li (binli)]

Upgraded to systemd 245.4-4ubuntu3.8 on ThinkPad P17, but this issue is still exist.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.8) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

asterisk/1:16.2.1~dfsg-2ubuntu1 (armhf)
gvfs/1.44.1-1ubuntu1 (amd64)
linux-oem-5.6/5.6.0-1057.61 (amd64)
munin/2.0.56-1ubuntu1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

ubuntu@lp1928200-f:~$ cat /etc/binfmt.d/TestFormat.conf
:TestFormat:M::XX::/bin/true:F
ubuntu@lp1928200-f:~$ dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.6 amd64 system and service manager

reboot to pick up change...

reboot again to show issue...

[ OK ] Reached target Reboot.
[ 35.164202] sd-umoun[1548]: Failed to unmount /oldroot: Device or resource busy
[ 35.174070] shutdown[1]: Failed to finalize file systems, ignoring
[ 35.219215] reboot: Restarting system

ubuntu@lp1928200-f:~$ dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.8 amd64 system and service manager

reboot to pick up change...

reboot again to check issue...

[ OK ] Reached target Reboot.
[ 22.494535] reboot: Restarting system

[Dan Streetman (ddstreet)]

> Upgraded to systemd 245.4-4ubuntu3.8 on ThinkPad P17, but this issue is still exist.

you may be seeing some other problem, you should open a new bug

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 245.4-4ubuntu3.10

---------------
systemd (245.4-4ubuntu3.10) focal-security; urgency=medium

  * SECURITY UPDATE: DoS via DHCP FORCERENEW
    - debian/patches/CVE-2020-13529.patch: tentatively ignore FORCERENEW
      command in src/libsystemd-network/sd-dhcp-client.c.
    - CVE-2020-13529
  * SECURITY UPDATE: denial of service via stack exhaustion
    - debian/patches/CVE-2021-33910.patch: do not use strdupa() on a path
      in src/basic/unit-name.c.
    - CVE-2021-33910

 -- Marc Deslauriers <email address hidden> Tue, 20 Jul 2021 07:39:51 -0400

[Daniel (daniel-marynicz)]

I still have this problem, but I have solved it.

I create files

file: /etc/systemd/system/kill-plymouthd.service
[Unit]
Description=kill plymouthd
After=umount.target
DefaultDependencies=false

[Service] Type=oneshot
ExecStart=/usr/bin/pkill plymouthd

[Install] WantedBy=final.target

file: /etc/systemd/system/rm-nvidia.service
[Unit]
Description=rm nvidia modules
After=umount.target
DefaultDependencies=false

[Service] Type=oneshot
ExecStart=/usr/sbin/rmmod nvidia_uvm
ExecStart=/usr/sbin/rmmod nvidia_drm
ExecStart=/usr/sbin/rmmod nvidia_modeset
ExecStart=/usr/sbin/rmmod nvidia
ExecStart=/usr/sbin/rmmod i2c_nvidia_gpu

[Install]
WantedBy=final.target

execute:
sudo systemctl daemon-reload
sudo systemctl enable rm-nvidia.service
sudo systemctl enable kill-plymouthd.service

for debug this problem i installed finalrd ane created executable

/etc/finalrd/sleep.finalrd

#!/bin/sh

set -x
if [ "$1" = "setup" ] then
    . /usr/share/initramfs-tools/hook-functions
    copy_exec /usr/bin/sleep
    copy_exec /usr/bin/mount

    exit 0
fi

#your debug script
mount
sleep 30

i have second system with ubuntu and in this second system i do not have nvidia drivers and i only created kill-plymouthd.service and this also work.

 and i did know why plymouthd is not killed befor umount root fs..

[Bin Li (binli)]

@daniel,

 Thanks for the solution, and it sounds good, could you help make merge request to finalrd to ask the maintainer's idea?

 From my side, currently finalrd used by mdadm, I would like to remove the dependency cause mdadm.

 In systemd/mdadm.shutdown from mdadm, it also do "mdadm --wait-clean --scan".

$ cat ./debian/mdadm.finalrd
#!/bin/sh
# SPDX-License-Identifier: GPL-3.0-only
set -e

if [ "$1" = "setup" ]
then
    . /usr/share/initramfs-tools/hook-functions
    copy_exec /sbin/mdadm
    copy_exec /sbin/mdmon

    exit 0
fi

mdmon --all --takeover
mdadm -vv --wait-clean --scan
mdadm -vv --stop --scan

 Thanks!

[Bin Li (binli)]

@daniel,

 I reported this issue in another ticket. Let's focus on below links. Thanks!

https://bugs.launchpad.net/ubuntu/+source/finalrd/+bug/1940514