Systemd blocks manual disk mount

[Jerry Quinn]

This is apparently a long running systemd bug: https://github.com/systemd/systemd/issues/1741

jlquinn@cerberus:~$ lsb_release -rd
Description: Ubuntu 20.04.5 LTS
Release: 20.04

jlquinn@cerberus:~$ apt-cache policy mount
mount:
  Installed: 2.34-0.1ubuntu9.3
  Candidate: 2.34-0.1ubuntu9.3
  Version table:
 *** 2.34-0.1ubuntu9.3 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.34-0.1ubuntu9 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages
jlquinn@cerberus:~$ apt-cache policy systemd
systemd:
  Installed: 245.4-4ubuntu3.19
  Candidate: 245.4-4ubuntu3.19
  Version table:
 *** 245.4-4ubuntu3.19 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     245.4-4ubuntu3.15 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     245.4-4ubuntu3 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

I am replacing a raid array with a new larger one. After finishing the migration, I try to mount the new one where the old one was located. This doesn't work due to something stale in systemd.

There is also a bug in mount, as it should at least return a failure when this happens. So there are actually 2 bugs here.

The package is util-linux because that is what `ubuntu-bug mount` chose.

root@cerberus:/home/jlquinn# df -h /space
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root 915G 269G 600G 31% /

root@cerberus:/home/jlquinn# mount /dev/md1 /space

root@cerberus:/home/jlquinn# echo $?
0

root@cerberus:/home/jlquinn# df -h /space
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-root 915G 269G 600G 31% /

root@cerberus:/home/jlquinn# mount /dev/md1 /mnt2/

root@cerberus:/home/jlquinn# df -h /mnt2
Filesystem Size Used Avail Use% Mounted on
/dev/md1 7.2T 2.9T 4.0T 43% /mnt2

from journalctl:

Jan 02 12:48:45 cerberus kernel: EXT4-fs (md1): mounted filesystem with ordered data mode. Opts: (null)
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-ad104abb\x2d9033\x2d4f69\x2daa26\x2d446654f03dc9.device. Stopping, too.
Jan 02 12:48:45 cerberus systemd[1]: Unmounting /space...
Jan 02 12:48:45 cerberus umount[150662]: umount: /space: target is busy.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Mount process exited, code=exited, status=32/n/a
Jan 02 12:48:45 cerberus systemd[1]: Failed unmounting /space.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-ad104abb\x2d9033\x2d4f69\x2daa26\x2d446654f03dc9.device. Stopping, too.
Jan 02 12:48:45 cerberus systemd[1]: Unmounting /space...
Jan 02 12:48:45 cerberus umount[150664]: umount: /space: target is busy.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Mount process exited, code=exited, status=32/n/a
Jan 02 12:48:45 cerberus systemd[1]: Failed unmounting /space.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-ad104abb\x2d9033\x2d4f69\x2daa26\x2d446654f03dc9.device. Stopping, too.
Jan 02 12:48:45 cerberus systemd[1]: Unmounting /space...
Jan 02 12:48:45 cerberus umount[150673]: umount: /space: target is busy.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Mount process exited, code=exited, status=32/n/a
Jan 02 12:48:45 cerberus systemd[1]: Failed unmounting /space.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-ad104abb\x2d9033\x2d4f69\x2daa26\x2d446654f03dc9.device. Stopping, too.
Jan 02 12:48:45 cerberus systemd[1]: Unmounting /space...
Jan 02 12:48:45 cerberus umount[150676]: umount: /space: target is busy.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Mount process exited, code=exited, status=32/n/a
Jan 02 12:48:45 cerberus systemd[1]: Failed unmounting /space.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-ad104abb\x2d9033\x2d4f69\x2daa26\x2d446654f03dc9.device. Stopping, too.
Jan 02 12:48:45 cerberus systemd[1]: Unmounting /space...
Jan 02 12:48:45 cerberus umount[150677]: umount: /space: target is busy.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Mount process exited, code=exited, status=32/n/a
Jan 02 12:48:45 cerberus systemd[1]: Failed unmounting /space.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-ad104abb\x2d9033\x2d4f69\x2daa26\x2d446654f03dc9.device. Stopping, too.
Jan 02 12:48:45 cerberus systemd[1]: Unmounting /space...
Jan 02 12:48:45 cerberus systemd[2575]: space.mount: Succeeded.
Jan 02 12:48:45 cerberus systemd[17926]: space.mount: Succeeded.
Jan 02 12:48:45 cerberus systemd[1]: space.mount: Failed with result 'exit-code'.
Jan 02 12:48:45 cerberus systemd[1]: Unmounted /space.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: mount 2.34-0.1ubuntu9.3
ProcVersionSignature: Ubuntu 5.4.0-135.152-generic 5.4.212
Uname: Linux 5.4.0-135-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.25
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Mon Jan 2 12:58:01 2023
InstallationDate: Installed on 2020-05-29 (947 days ago)
InstallationMedia: Ubuntu 18.04.3 LTS "Bionic Beaver" - Release amd64 (20190805)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: util-linux
UpgradeStatus: Upgraded to focal on 2020-11-04 (789 days ago)
mtime.conffile..etc.apport.crashdb.conf: 2021-06-21T20:18:54.448530

[Jerry Quinn (jlquinn)]

The systemd bug report states that issuing `systemctl daemon-reload` clears the stale state. I confirm that systemctl daemon-reload followed by the mount command then works as expected, without requiring a machine reboot.