Bind mount to NFS mount fails on Ubuntu 22.04

Bug #1979952 reported by Philipp Wendler
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd
Unknown
Unknown
systemd (Ubuntu)
Status tracked in Kinetic
Jammy
Fix Released
Medium
Unassigned
Kinetic
Fix Released
Medium
Unassigned

Bug Description

[Impact]
Some NFS setups that have worked on previous versions of Ubuntu are now broken. Specifically, when creating a bind mount, systemd will fail if the directory already exists on NFS.

[Test Plan]
This test plan requires an NFS server to be in place. The exact setup is not important, but for the purposes of this test plan we will assume the server exports /data, which contains a directory called home.

* Make sure nfs-common is installed:

 $ apt install nfs-common -y

* Check the NFS server exports using showmount:

 $ showmount -e $NFS_SERVER_IP
 Export list for $NFS_SERVER_IP:
 /data *

* Add the appropriate entries to /etc/fstab. In this example, we will make /home2 a bind mount to /data/home:

 # /etc/fstab
 $NFS_SERVER_IP:/data /data nfs defaults,nfsvers=3 0 0
 /data/home /home2 none bind,_netdev,x-systemd.requires-mounts-for=/data 0 0

* Check systemctl status home2.mount. On an affected system, we should see a permission denied error:

 $ systemctl status home2.mount
 x home2.mount - /home2
      Loaded: loaded (/etc/fstab; generated)
      Active: failed (Result: resources)
       Where: /home2
        What: /data/home
        Docs: man:fstab(5)
              man:systemd-fstab-generator(8)

 systemd[1]: home2.mount: Failed to make bind mount source '/data/home': Permission denied
 systemd[1]: home2.mount: Failed to run 'mount' task: Permission denied
 systemd[1]: home2.mount: Failed with result 'resources'.
 systemd[1]: Failed to mount /home2.

* On a patched system, we expect /home2 to be mounted successfully.

[Where problems could occur]
The patch makes it so that systemd will not fail bind mounts immediately if creating the directory fails. Thus, any regressions would be seen during bind mounts with systemd. N.B. that this patch restores previous behavior in systemd.

[Original Description]
I have the following setup:
- /data is an NFS mount.
- /home is a bind mount to /data/home.

Configured in /etc/fstab with these lines:

$FILE_SERVER:/data /data nfs defaults 0 0
/data/home /home none bind,_netdev,x-systemd.requires-mounts-for=/data 0 0

This has worked with the same configuration for me on at least Ubuntu 18.04 and Ubuntu 20.04, but on Ubuntu 22.04 the mount of /home fails when attempted by systemd.
So not only is /home not mounted after boot, but also when I run "sudo systemctl start home.mount", it fails.

The journal entries are this:

sudo[1316]: wendler : TTY=pts/0 ; PWD=/home/wendler ; USER=root ; COMMAND=/usr/bin/systemctl start home.mount
sudo[1316]: pam_unix(sudo:session): session opened for user root(uid=0) by wendler(uid=1000)
systemd[1]: home.mount: Directory /home to mount over is not empty, mounting anyway.
systemd[1]: home.mount: Failed to make bind mount source '/data/home': Permission denied
systemd[1]: home.mount: Failed to run 'mount' task: Permission denied
systemd[1]: home.mount: Failed with result 'resources'.
systemd[1]: Failed to mount /home.

However, when I run "sudo mount /home" it works.

Now the weird thing is that after I have mounted /home manually once and unmounted it again, then "sudo systemctl start home.mount" suddenly also works! But of course only until the next reboot.

And even if I just do "ls /data" once (either as root or as my user), it also makes "sudo systemctl start home.mount" start working!

To be fully clear:

- Booting the system (/home fails to mount)
- sudo systemctl start home.mount # fails
- sudo ls /data # shows correct output
- sudo systemctl start home.mount # works!

Additional information:

The NFS export for /data has root_squash set and if I remove this option, the bind mount of /home works as it should. However, both /data and /data/home have mode rwxr-xr-x, so root is able to enter and read these directories even despite root_squash.

It is not a network or mount-order problem. Not only is the bind mount to /home correctly attempted after /data is mounted during boot, remember that the bind mount also fails when I try it with "sudo systemctl start home.mount" minutes afterwards.

Versions (system is a recent installation with all updates applied):
- Ubuntu 22.04
- Linux 5.15.0-40-generic
- systemd 249.11-0ubuntu3.3
- util-linux 2.37.2-4ubuntu3

This appears on both Ubuntu 22.04 machines that I have (a hardware machine and a VM).

Related branches

Revision history for this message
Philipp Wendler (philw85) wrote (last edit ):

Somebody else has reported this upstream (https://github.com/systemd/systemd/issues/24120) and a fix is being developed as https://github.com/systemd/systemd/pull/24122.

Revision history for this message
Philipp Wendler (philw85) wrote :

The fix https://github.com/systemd/systemd/pull/24122 is now merged upstream. The change should be easy to backport, is it possible that this could be done for Ubuntu?

Revision history for this message
Nick Rosbrook (enr0n) wrote :

I did some testing to confirm this is present in jammy (the offending code is also in kinetic). I will test the upstream fix soon.

tags: added: rls-jj-incoming rls-kk-incoming
Lukas Märdian (slyon)
Changed in systemd (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
tags: added: fr-2591
Lukas Märdian (slyon)
tags: removed: fr-2591 rls-jj-incoming rls-kk-incoming
Revision history for this message
Nick Rosbrook (enr0n) wrote :

This upstream fix for this was backported to 251.4, which I am currently working on merging from Debian.

Changed in systemd (Ubuntu Kinetic):
status: Triaged → In Progress
Nick Rosbrook (enr0n)
description: updated
Revision history for this message
Nick Rosbrook (enr0n) wrote (last edit ):

The fix for kinetic is included in 251.4-1ubuntu1, and the patch is staged in git for the next jammy SRU.

Changed in systemd (Ubuntu Kinetic):
status: In Progress → Fix Released
Changed in systemd (Ubuntu Jammy):
status: New → Fix Committed
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Philipp, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-jammy
Revision history for this message
Philipp Wendler (philw85) wrote :

Thanks! Version 249.11-0ubuntu3.5 of systemd fixes this issue for me: I rebooted the machine and the bind mount to the NFS mount was correctly mounted automatically. Previously this had always failed.

Changed in systemd (Ubuntu Jammy):
status: Fix Committed → Invalid
status: Invalid → Fix Committed
tags: added: verification-done-jammy
removed: verification-needed-jammy
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/249.11-0ubuntu3.5)

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.5) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.48.2-0ubuntu1 (ppc64el)
systemd/249.11-0ubuntu3.5 (ppc64el)
udisks2/2.9.4-1ubuntu2 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Changed in systemd (Ubuntu Jammy):
importance: Undecided → Medium
tags: added: foundations-todo
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Philipp, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed-jammy
removed: verification-done-jammy
Revision history for this message
Nick Rosbrook (enr0n) wrote :

I have verified this fix using systemd 249.11-0ubuntu3.6 from jammy-proposed:

nr@clean-jammy-amd64:~$ apt-cache policy systemd
systemd:
  Installed: 249.11-0ubuntu3.6
  Candidate: 249.11-0ubuntu3.6
  Version table:
 *** 249.11-0ubuntu3.6 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.4 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
nr@clean-jammy-amd64:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/vda3 during installation
UUID=32a6b55f-64d3-421b-a77d-2f9a2c6d68be / ext4 errors=remount-ro 0 1
# /boot/efi was on /dev/vda2 during installation
UUID=C591-26A1 /boot/efi vfat umask=0077 0 1
/swapfile none swap sw 0 0
172.16.99.107:/data /data nfs defaults,nfsvers=3 0 0
/data/home /home2 none bind,_netdev,x-systemd.requires-mounts-for=/data 0 0
nr@clean-jammy-amd64:~$ systemctl status home2.mount
● home2.mount - /home2
     Loaded: loaded (/etc/fstab; generated)
     Active: active (mounted) since Mon 2022-09-12 15:29:19 EDT; 29s ago
      Where: /home2
       What: 172.16.99.107:/data/home
       Docs: man:fstab(5)
             man:systemd-fstab-generator(8)
      Tasks: 0 (limit: 1084)
     Memory: 16.0K
        CPU: 2ms
     CGroup: /system.slice/home2.mount

Sep 12 15:29:19 clean-jammy-amd64 systemd[1]: Mounting /home2...
Sep 12 15:29:19 clean-jammy-amd64 systemd[1]: Mounted /home2.

tags: added: verification-done verification-done-jammy
removed: verification-needed verification-needed-jammy
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/249.11-0ubuntu3.6)

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.6) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

stunnel4/3:5.63-1build1 (amd64)
munin/2.0.57-1ubuntu2 (amd64)
corosync/unknown (s390x)
conntrack-tools/unknown (s390x)
exim4/4.95-4ubuntu2.1 (ppc64el)
umockdev/0.17.7-1 (armhf)
netplan.io/0.104-0ubuntu2.1 (amd64)
initramfs-tools/0.140ubuntu13 (amd64)
dovecot/unknown (s390x)
network-manager/1.36.6-0ubuntu2 (amd64)
cups/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Nick Rosbrook (enr0n) wrote :

The autopkgtest regressions for systemd 249.11-0ubuntu3.6 in jammy-proposed were all resolved with retries.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.9 KiB)

This bug was fixed in the package systemd - 249.11-0ubuntu3.6

---------------
systemd (249.11-0ubuntu3.6) jammy; urgency=medium

  * Deny-list TEST-58-REPART on ppc64el (LP: #1988994)
    File: debian/patches/lp1988994-Deny-list-TEST-58-REPART-on-ppc64el.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d2ed3cc1d223bf35015b15ff83b50156b58f0f38

systemd (249.11-0ubuntu3.5) jammy; urgency=medium

  [ Nick Rosbrook ]
  * Ensure dns_search_domain_unlink_marked removes all marked domains (LP: #1975667)
    File: debian/patches/lp1975667-Ensure-dns_search_domain_unlink_marked-removes-all-marked.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=919d5ddedd5bb8b45ab9437bf42d66c2821bb074
  * core,firstboot: workaround timezone issues on Ubuntu Core (LP: #1981042)
    Thanks to Robert Ancell for preparing the patch.
    File: debian/patches/lp1981042-core-firstboot-workaround-timezone-issues-caused-by-Ubunt.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b15546361b549217908fb6ca5d473be23d7fa757
  * network: do not remove localhost address (LP: #1979951)
    File: debian/patches/lp1979951-network-do-not-remove-localhost-address.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=2cd88391cce9fe95a486ae6dd214c12f236f3881
  * units: remove the restart limit on the modprobe@.service (LP: #1982462)
    File: debian/patches/lp1982462-units-remove-the-restart-limit-on-the-modprobe-.service.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8f0acd1b2fbb8eed1259c34963e5e9b201bef900
  * pstore: do not try to load mtdpstore (LP: #1981622)
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=15225032c3657f5906ee49d48929f9295a8664a0
  * core/mount: downgrade log level about several mkdir failures (LP: #1979952)
    Files:
    - debian/patches/lp1979952-Revert-core-mount-fail-early-if-directory-cannot-be-creat.patch
    - debian/patches/lp1979952-core-mount-downgrade-log-level-about-several-mkdir-failur.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ee8cfcf500698fab2e990de291ecf4c3ab87a4ae
  * debian/control: add Recommends: systemd-hwe-hwdb to udev.
    The systemd-hwe-hwdb brings in additional hwdb rules for HWE, so we want
    those installed with udev by default.
    File: debian/control
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4a7a3258c33201cca305956820fcc6bcd6052d76
  * hwdb: implement --root option for systemd-hwdb query (LP: #1988078)
    Files:
    - debian/libsystemd0.symbols
    - debian/patches/lp1988078-hwdb-implement-root-option-for-systemd-hwdb-query.patch
    - debian/patches/lp1988078-sd-hwdb-add-sd_hwdb_new_from_path.patch
    - debian/patches/lp1988078-sd-hwdb-include-sys-stat.h-in-hwdb-internal.h.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=937fef96c858f2f2042bf71032f315647c14add0

  [ Luca Boccassi ]
  * Enable systemd-repart and ship it in a new s...

Read more...

Changed in systemd (Ubuntu Jammy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.