Failed messages of sd-umount shown with reboot/shutdown

[Woodrow Shen]

We were seeing the same sequence of errors heppened on KVM and ARM device running UC20:

"""
[12450.684692] sd-umoun[5922]: Failed to unmount /oldroot: Device or resource busy
[12450.693515] sd-remou[5923]: Failed to remount '/oldroot/run/mnt/data' read-only: Device or resource busy
[12450.704374] sd-umoun[5924]: Failed to unmount /oldroot/run/mnt/data: Device or resource busy
[12450.714225] sd-umoun[5925]: Failed to unmount /oldroot/run: Device or resource busy
[12450.726073] shutdown[1]: Could not detach loopback /dev/loop0: Device or resource busy
[12450.735481] shutdown[1]: Failed to finalize file systems, loop devices, ignoring
[12451.129165] reboot: Power down
"""

[Steps to reproduce]

$ sudo poweroff or $ sudo reboot

[Additional info]

http://cdimage.ubuntu.com/ubuntu-core/20/stable/current/ubuntu-core-20-amd64.img.xz

[Woodrow Shen (woodrow-shen)]

It looks like systemd issue so probably I will move this to systemd later if someone has any thoughts.

[Steve Langasek (vorlon)]

These messages appear to be coming from finalrd, and leading to an actual unclean shutdown.

[Dimitri John Ledkov (xnox)]

The way uc20 is booted, and how bind mounts are setup means that we do force unmount eventually.

It matters which snaps are installed, and which processes have memory mapped which files.

Many things should probably be marked for lazyunmount.

Please provide details why is held up /oldroot/run/mnt/data

For example output from fuser.

I do not believe any of the linked issues are of relevance here. It matters how snaps have been mounted, as they do get mounted circularly.

[Woodrow Shen (woodrow-shen)]

@Dimitri,

The issue can be reproduced by http://cdimage.ubuntu.com/ubuntu-core/20/stable/current/ubuntu-core-20-amd64.img.xz while I previously fetched the image there to test, so if you think it's snap relevant, I can check with snapd team.

[hugh chao (hugh712)]

I also can reproduce this issue in my amd64 machine.

[Balint Reczey (rbalint)]

IMO it is out of scope for finalrd to unmount mounts which can't be cleanly unmounted.

[Launchpad Janitor (janitor)]

[Expired for snapd because there has been no activity for 60 days.]

[Ian Johnson (anonymouse67)]

I continue to see these messages on all UC20 VM's I use, including the most recently released stable image on cdimage when booted in a VM

[Steve Langasek (vorlon)]

> IMO it is out of scope for finalrd to unmount mounts
> which can't be cleanly unmounted.

Er, this is nonsense. The entire point of finalrd is to ensure we pivot to an initramfs after all other processes have been terminated, so that it is possible to cleanly unmount the root filesystem with no errors. For the root filesystem to be busy and failing to unmount is clearly a bug in either systemd or finalrd.

[Maciej Borzecki (maciek-borzecki)]

Valentin has 2 PRs up to address this on the snapd/core20 side: https://github.com/snapcore/snapd/pull/11119 and https://github.com/snapcore/core20/pull/124

[Dimitri John Ledkov (xnox)]

The choices in core20 pr are good.

systemd-shutdownd should attempt to make things private, and should attempt to move them out of the way, and count that as progress.

[Dimitri John Ledkov (xnox)]

However it already does all that.

Marks things private, tries to move them if they cannot be unmounted.

So not sure what the issue is.