LXC with r/w sys and udev keeps trying to unmount bind mounts

[Iain Lane]

I recently hit ENOSPC which turned out to be the result of various syslogs from the current and previous boots totalling up to 30G.

They were filled with messages like

Apr 15 11:22:35 vivid systemd[1]: Unit var-lib-schroot-union-underlay-vivid\x2damd64\x2d5d4f7453\x2d9a95\x2d4a6a\x2d9aee\x2d9394923bf734.mount is bound to inactive unit dev-disk-by\x2duuid-980689ca\x2de7d9\x2d4a99\x2d8230\x2d33b8b6e917cd.device. Stopping, too.
Apr 15 11:22:35 vivid systemd[1]: Unmounting /var/lib/schroot/union/underlay/vivid-amd64-5d4f7453-9a95-4a6a-9aee-9394923bf734...
Apr 15 11:22:35 vivid umount[31795]: umount: /var/lib/schroot/union/underlay/vivid-amd64-5d4f7453-9a95-4a6a-9aee-9394923bf734: target is busy
Apr 15 11:22:35 vivid umount[31795]: (In some cases useful info about processes that
Apr 15 11:22:35 vivid umount[31795]: use the device is found by lsof(8) or fuser(1).)
Apr 15 11:22:35 vivid systemd[1]: var-lib-schroot-union-underlay-vivid\x2damd64\x2d5d4f7453\x2d9a95\x2d4a6a\x2d9aee\x2d9394923bf734.mount mount process exited, code=exited status=32
Apr 15 11:22:35 vivid systemd[1]: Failed unmounting /var/lib/schroot/union/underlay/vivid-amd64-5d4f7453-9a95-4a6a-9aee-9394923bf734.
Apr 15 11:22:35 vivid systemd[1]: Unit var-lib-schroot-union-underlay-vivid\x2damd64\x2d5d4f7453\x2d9a95\x2d4a6a\x2d9aee\x2d9394923bf734.mount is bound to inactive unit dev-disk-by\x2duuid-980689ca\x2de7d9\x2d4a99\x2d8230\x2d33b8b6e917cd.device. Stopping, too.
Apr 15 11:22:35 vivid systemd[1]: Unmounting /var/lib/schroot/union/underlay/vivid-amd64-5d4f7453-9a95-4a6a-9aee-9394923bf734...
Apr 15 11:22:35 vivid systemd[1]: Unmounted /var/lib/schroot/union/underlay/vivid-amd64-5d4f7453-9a95-4a6a-9aee-9394923bf734.
Apr 15 11:22:35 vivid systemd[1]: Unit var-lib-schroot-union-underlay-vivid\x2damd64\x2d5d4f7453\x2d9a95\x2d4a6a\x2d9aee\x2d9394923bf734.mount entered failed state.

looping constantly for the duration of any builds in sbuild.

Why is systemd trying to do this?

SRU TEST CASE:
--------------
 - Create a vivid container, and set "lxc.aa_profile = unconfined" and "lxc.mount.auto = sys:rw cgroup" in its config. (Note that the latter is not supported!)
 - Start the container
 - Run "mount -v -o bind /bin /mnt"
 - Observe that it doesn't stay mounted, but "sudo journalctl" says

Mai 06 09:17:08 test systemd[1]: Unit mnt.mount is bound to inactive unit dev-sda3.device. Stopping, too.
Mai 06 09:17:08 test systemd[1]: Unmounting /mnt...
Mai 06 09:17:08 test systemd[1]: Unmounted /mnt.

 - With the fixed package bind mounts stay mounted.

REGRESSION POTENTIAL: This could potentially break cleanup of stale mount points of either hotplug devices which disappear, or media which get forcefully ejected (like CDs). Testing should include that these still work.

laney@vivid> apt-cache policy systemd
systemd:
  Installed: 219-7ubuntu1
  Candidate: 219-7ubuntu1
  Version table:
 *** 219-7ubuntu1 0
        500 http://sherwood/ubuntu/ vivid/main amd64 Packages
        100 /var/lib/dpkg/status

[Tpgxyz (tpgxyz)]

Log story short.

After update to systemd-219, dracut during live iso boot was not able to mount devices for further switch root, whereby booting iso in live mode was not possible. An example:

LOOPDEV=$( losetup -f )
losetup -r $LOOPDEV /live/media/LiveOS/squashfs.img
mount -n -t squashfs -o ro $LOOPDEV /live/distrib

After a deep digging, and getting system to boot into a real root simple mount command was not mounting anything.

mount -n -t squashfs -o ro /media/OpenMandriva_2015.0/LiveOS/squashfs.img /media/test

Applying those patches fixed issue:
https://bugzilla.gnome.org/show_bug.cgi?id=743891

[Tpgxyz (tpgxyz)]

Bump. System is unbootable with 219 release. Raising importance.

[Tpgxyz (tpgxyz)]

Looks like that applying this patch:
https://abf.io/openmandriva/systemd/blob/master/0001-Revert-core-mount-add-dependencies-to-dynamically-mo.patch

allows to mount devices.

[Tpgxyz (tpgxyz)]

Ping anyone ?

[Iain Lane (laney)]

These are going over the bus

signal sender=:1.0 -> dest=(null destination) serial=10268593 path=/org/freedesktop/systemd1/unit/var_2dlib_2dschroot_2dunion_2dunderlay_2dvivid_5cx2damd64_5cx2d422d4deb_5cx2d2a9d_5cx2d4364_5cx2dbba4_5cx2d04d126ec6cf2_2emount; interface=org.freedesktop.DBus.Properties; member=PropertiesChanged
   string "org.freedesktop.systemd1.Mount"
   array [
      dict entry(
         string "What"
         variant string "/dev/disk/by-uuid/980689ca-e7d9-4a99-8230-33b8b6e917cd"
      )
      dict entry(
         string "Options"
         variant string "rw,relatime,discard,errors=remount-ro,data=ordered"
      )
      dict entry(
         string "Type"
         variant string "ext4"
      )
      dict entry(
         string "ControlPID"
         variant uint32 0
      )
      dict entry(
         string "Result"
         variant string "exit-code"
      )
   ]
   array [
      string "ExecMount"
      string "ExecUnmount"
      string "ExecRemount"
   ]
signal sender=:1.0 -> dest=(null destination) serial=10268594 path=/org/freedesktop/systemd1/unit/var_2dlib_2dschroot_2dunion_2dunderlay_2dvivid_5cx2damd64_5cx2d422d4deb_5cx2d2a9d_5cx2d4364_5cx2dbba4_5cx2d04d126ec6cf2_2emount; interface=org.freedesktop.DBus.Properties; member=PropertiesChanged
   string "org.freedesktop.systemd1.Unit"
   array [
      dict entry(
         string "ActiveState"
         variant string "active"
      )
      dict entry(
         string "SubState"
         variant string "mounted"
      )
      dict entry(
         string "InactiveExitTimestamp"
         variant uint64 1429097554602566
      )
      dict entry(
         string "InactiveExitTimestampMonotonic"
         variant uint64 10242667599
      )
      dict entry(
         string "ActiveEnterTimestamp"
         variant uint64 1429097829298497
      )
      dict entry(
         string "ActiveEnterTimestampMonotonic"
         variant uint64 10517363530
      )
      dict entry(
         string "ActiveExitTimestamp"
         variant uint64 1429097829292302
      )
      dict entry(
         string "ActiveExitTimestampMonotonic"
         variant uint64 10517357335
      )
      dict entry(
         string "InactiveEnterTimestamp"
         variant uint64 0
      )
      dict entry(
         string "InactiveEnterTimestampMonotonic"
         variant uint64 0
      )
      dict entry(
         string "Job"
         variant struct {
               uint32 1467033
               object path "/org/freedesktop/systemd1/job/1467033"
            }
      )
      dict entry(
         string "ConditionResult"
         variant boolean false
      )
      dict entry(
         string "AssertResult"
         variant boolean false
      )
      dict entry(
         string "ConditionTimestamp"
         variant uint64 0
      )
      dict entry(
         string "ConditionTimestampMonotonic"
         variant uint64 0
      )
      dict entry(
         string "AssertTimestamp"
    ...

[Zuyi Hu (hzy068808)]

(In reply to Tomasz Paweł Gajc from comment #2)
> Looks like that applying this patch:
> https://abf.io/openmandriva/systemd/blob/master/0001-Revert-core-mount-add-
> dependencies-to-dynamically-mo.patch
>
> allows to mount devices.

Thank you very much, this patch helps me, I am doubt why upstream don't fix this problem.

[Greyson Christoforo (greyltc)]

This is a bad bug.

Please see my report on the Arch Linux bug tracker on it here: https://bugs.archlinux.org/task/44658#comment134665

This report is using systemd from Arch's [testing] repo which seems to be version 219 with some patches applied by the Arch maintainers which try to address this bug: systemd 219-6, https://www.archlinux.org/packages/testing/x86_64/systemd/

In summary, even when the bug has been "fixed" (allowing mounts to be made again) there are cases when systemd is unmounting things it should not be when the user runs umount (seems like it unmounts ALL mounts associated with a device whenever any mount associated with that device is umounted).

A bug that breaks filesystem mouting/unmounting seems particularly critical. Whatever new logic 219 introduced that created this bug should probably be reverted from the systemd release until all the mounting and unmounting corner cases can be tested and covered.

[Bastien-g (bastien-g)]

Another use case to complement the report:

- system is a home server running Arch linux, full disk encrypted with dmcrypt/LUKS and remotely unlocked via dropbear_initrd_encrypt [1]
- it was updated today with the latest systemd packages ({lib}systemd{-sysvcompat} 219-5) and latest linux-lts (3.14.39), therefore the initramfs was rebuilt with mkinitcpio.

What happens: after rebooting I am locked out at the initramfs stage with the following error messages:

- on client trying to ssh in dropbear_initrd:
    Device /dev/disk/by-id/wwn-<redacted_id>-part3 doesn't exist or access denied.

- on server:
   Running systemd 219
(dropbear initialization sequence, everything is ok)
    Starting dropbear
    [123] Apr 23 09:43:56 Running in background
(try to connect remotely via SSH)
    Pubkey auth succeeded for 'root' with key xxx from 192.xxx
    syslogin_perform_logout: logout(pts/0) returned an error: No such file or directory
    Exit (root) Disconnect received
    ERROR: device '/dev/mapper/lvm-archroot' not found. Skipping fsck.
    ERROR: Unable to find rot device '/dev/mapper/lvm-archroot'
    You are being dropped to a recovery shell

If I don't try to login via SSH there is a 15 seconds delay between [123] line and the first ERROR; there is no mean whatsoever to unlock dropbear locally as used to be the case ("enter passphrase for /dev/disk/by-id/wwn-<redacted_id>-part" used to be displayed).

My kernel command line is BOOT_IMAGE=/vmlinuz-linux-lts root=/dev/mapper/lvm-archroot rw quiet cryptdevice=/dev/disk/by-id/wwn-<redacted_id>-part3:crypt ip=192.168.1.xxx::192.168.1.254::arch-medion:eth0:none

[1] https://wiki.archlinux.org/index.php/Dm-crypt/Specialties#Remote_unlocking_of_the_root_.28or_other.29_partition

[Tpgxyz (tpgxyz)]

Created attachment 115298
Proposed patch

With this patch systemd does not umounts manually mounted devices

[Iain Lane (laney)]

Seems the revert in the linked bug stops this happening for me.

[Martin Pitt (pitti)]

Iian, https://launchpad.net/~pitti/+archive/ubuntu/sru-test now has packages which potentially fix this. If they don't, I need either

 - a full journalctl from boot up to the point where this surfaces, preferably with "debug"; or

 - a recipe how to reproduce this in a VM; I (and I'm sure many others) use sbuild daily without getting this, so somehow your config is special.

Thanks!

[Martin Pitt (pitti)]

Not fixed by that commit. Reproducible in ssh ubuntu@10.55.32.96

[Martin Pitt (pitti)]

Simpler reproducer:
 - Create a vivid container, and set "lxc.aa_profile = unconfined"
 - Start the container
 - Run "mount -v -o bind /bin /mnt"

Mai 06 09:17:08 test systemd[1]: Unit mnt.mount is bound to inactive unit dev-sda3.device. Stopping, too.
Mai 06 09:17:08 test systemd[1]: Unmounting /mnt...
Mai 06 09:17:08 test systemd[1]: Unmounted /mnt.

[Martin Pitt (pitti)]

In my local container, http://cgit.freedesktop.org/systemd/systemd/commit/?id=f62009410 (which was the fix in the above PPA) helps already. However, it doesn't in the cloud instance: The main difference there is that in that cloud instance udev is running while in my local container it isn't (it's not usually meant to run in LXC).

Adding these options in the container starts udev and reproduces this issue again:

lxc.mount.auto = sys:rw cgroup
lxc.mount.entry = /etc/schroot /srv/lxc/test/rootfs/etc/schroot none bind 0 0

[Martin Pitt (pitti)]

FTR: pull in http://cgit.freedesktop.org/systemd/systemd/commit/?id=7ba2711d (not sufficient, but should also help)

[Martin Pitt (pitti)]

I proposed a patch for this upstream: http://lists.freedesktop.org/archives/systemd-devel/2015-May/031718.html

Iain, it would be great if you could confirm that it works for you as well? I attach a patched amd64 /lib/systemd/systemd binary, in case that's useful for you.

[Iain Lane (laney)]

On Thu, May 14, 2015 at 10:55:50AM -0000, Martin Pitt wrote:
> I proposed a patch for this upstream:
> http://lists.freedesktop.org/archives/systemd-devel/2015-May/031718.html
>
> Iain, it would be great if you could confirm that it works for you as
> well? I attach a patched amd64 /lib/systemd/systemd binary, in case
> that's useful for you.

Cheers Martin. I applied that patch to Ubuntu's systemd package and
started the container with that.

Happy to report that it seems to work fine for me. The mounts stay
intact as they are supposed to.

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

[Martin Pitt (pitti)]

First patch was nack'ed; second proposal: http://lists.freedesktop.org/archives/systemd-devel/2015-May/031893.html

Attaching an updated /lib/systemd/systemd with those patches (amd64).

[Martin Pitt (pitti)]

This is the command line I'm using for test iteration. It's not that simple, so I want to keep it here in case I ever need it again.

This builds systemd from git, copies it into a "test" container, starts it, runs the bind mount/umount, and shows the debug log:

schroot -r -c session:schroot-wily-systemd -- make -j4 && sudo rm -f /srv/lxc/test/rootfs/lib/systemd/systemd && sudo cp systemd /srv/lxc/test/rootfs/lib/systemd/systemd && clear && sudo lxc-start -n test && sleep 3 && sudo lxc-attach -n test -- sh -c 'systemctl show -p Id,ActiveState,SubState,BindsTo,BoundBy dev-sda3.device etc-schroot.mount; systemd-analyze set-log-level debug; mkdir /tmp/etc /tmp/boot; mount -o bind /etc /tmp/etc; mount -o bind /boot /tmp/boot; sleep 0.5; echo "---- unmounting /tmp/etc ----"; umount /tmp/etc; sleep 0.5; systemctl show -p Id,ActiveState,SubState,BindsTo,BoundBy dev-sda3.device etc-schroot.mount tmp-boot.mount; systemd-analyze set-log-level info; mount | egrep "/tmp/|/etc"; journalctl -b | grep -A10000 "Reached target Graphical"; poweroff'

[Martin Pitt (pitti)]

Refined command line to also verify that the tentative device *does* move to "dead" once the last reference gets unmounted:

schroot -r -c session:schroot-wily-systemd -- make -j4 && sudo rm -f /srv/lxc/test/rootfs/lib/systemd/systemd && sudo cp systemd /srv/lxc/test/rootfs/lib/systemd/systemd && clear && sudo lxc-start -n test && sleep 3 && sudo lxc-attach -n test -- sh -c 'systemctl show -p Id,ActiveState,SubState,BindsTo,BoundBy dev-sda3.device; systemd-analyze set-log-level debug; mkdir /tmp/etc /tmp/boot; mount -o bind /etc /tmp/etc; mount -o bind /boot /tmp/boot; sleep 0.5; echo "---- unmounting /tmp/etc ----"; umount /tmp/etc; sleep 0.5; systemctl show -p Id,ActiveState,SubState,BindsTo,BoundBy dev-sda3.device etc-schroot.mount tmp-boot.mount; echo "---- umounting /tmp/boot ----"; umount /tmp/boot; sleep 0.5; systemctl show -p Id,ActiveState,SubState,BindsTo,BoundBy dev-sda3.device etc-schroot.mount tmp-boot.mount; systemd-analyze set-log-level info; mount | egrep "/tmp/|/etc" || true; journalctl -b | grep -A10000 "Reached target Graphical"; poweroff'

[Martin Pitt (pitti)]

New set of proposed patches:
http://lists.freedesktop.org/archives/systemd-devel/2015-May/032030.html
http://lists.freedesktop.org/archives/systemd-devel/2015-May/032024.html

Attached /lib/systemd/systemd with these applied (amd64), for testing.

[Martin Pitt (pitti)]

I got http://cgit.freedesktop.org/systemd/systemd/commit/?id=2005219 upstream, which is about as far as they want to go. It's not a supported configuration to have a r/w /sys in a container and run udev. This can't work. So if you want to manually unmount bind mounts (the second patch), please drop the r/w mounting of sys.

[Martin Pitt (pitti)]

After some more back and forth we fixed the remaining issues as well now:

  http://cgit.freedesktop.org/systemd/systemd/commit/?id=fcd8b26
  http://cgit.freedesktop.org/systemd/systemd/commit/?id=394763f6 (followup fix for the former)

[Martin Pitt (pitti)]

Cherry-picked to http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=ubuntu-vivid&id=f5d9b13f784.

Uploaded to https://launchpad.net/~pitti/+archive/ubuntu/sru-test .

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 219-10ubuntu1

---------------
systemd (219-10ubuntu1) wily; urgency=medium

  * Merge with Debian experimental branch. Remaining Ubuntu changes:
    - Hack to support system-image read-only /etc, and modify files in
      /etc/writable/ instead.
    - Keep our much simpler udev maintainer scripts (all platforms must
      support udev, no debconf).
    - initramfs init-top: Drop $ROOTDELAY, we do that in a more sensible way
      with wait-for-root. Will get applicable to Debian once Debian gets
      wait-for-root in initramfs-tools.
    - initramfs init-bottom: If LVM is installed, settle udev,
      otherwise we get missing LV symlinks. Workaround for LP #1185394.
    - Add debian/udev.lvm2.init: Dummy SysV init script to satisfy insserv
      dependencies to "lvm2" which is handled with udev rules in Ubuntu.
    - Add debian/udev.lvm2.service to avoid running the dummy lvm2 init
      script.
    - Provide shutdown fallback for upstart. (LP: #1370329)
    - debian/extra/ifup@.service: Additionally run for "auto" class. We don't
      really support "allow-hotplug" in Ubuntu at the moment, so we need to
      deal with "auto" devices appearing after "/etc/init.d/networking start"
      already ran. (LP: #1374521) Also run ifup in the background during boot,
      to avoid blocking network.target. (LP: #1425376)
    - ifup@.service: Drop dependency on networking.service (i. e.
      /etc/init.d/networking), and merely ensure that /run/network exists.
      This avoids unnecessary dependencies/waiting during boot and dependency
      cycles if hooks wait for other interfaces to come up (like ifenslave
      with bonding interfaces). (LP: #1414544)
    - Add Get-RTC-is-in-local-time-setting-from-etc-default-rc.patch: In
      Ubuntu we currently keep the setting whether the RTC is in local or UTC
      time in /etc/default/rcS "UTC=yes|no", instead of /etc/adjtime.
      (LP: #1377258)
    - Put session scopes into all cgroup controllers. This makes unprivileged
      user LXC containers work under systemd. (LP: #1346734)
    - systemctl: Don't forward telinit u to upstart. This works around
      upstart's Restart() always reexec'ing /sbin/init on Restart(), even if
      that changes to point to systemd during the upgrade. This avoids running
      systemd during a dist-upgrade. (LP: #1430479)
    - Drop hwdb-update dependency from udev-trigger.service, which got
      introduced in v219-stable. This causes udev and plymouth to start too
      late and isn't really needed in Ubuntu yet as we don't support stateless
      systems yet and handle hwdb.bin updates through dpkg triggers. This can
      be dropped again with initramfs-tools 0.117.
    - Lower Breaks: to plymouth version which has the udev inotify fix in
      Ubuntu.
    - Lower libappamor dep to the Ubuntu version where it moved to /lib.
    - Lower apparmor Breaks: to the Ubuntu version that dropped $remote_fs.
    - Change systemd-sysv's conflicts to upstart-sysv. (LP: #1422681)
    - Make failure of boot-and-services NSpawn.test_boot non-fatal for now.
      This currently fails when being triggered by Jenkins, but is totally
     ...

[Ross Patterson (rossp)]

How can one get this fix in vivid?

[Martin Pitt (pitti)]

@Ross: I already prepared the fix some two weeks ago, it's waiting in https://launchpad.net/ubuntu/vivid/+queue?queue_state=1 to be processed by the SRU team.

[Brian Murray (brian-murray)]

Hello Iain, or anyone else affected,

Accepted systemd into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/219-7ubuntu6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Iain Lane (laney)]

Confirmed the fix on a cloud VM, thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 219-7ubuntu6

---------------
systemd (219-7ubuntu6) vivid; urgency=medium

  * Fix assertion crash with empty Exec*= paths. (LP: #1454173)
  * systemd-fsckd autopkgtest: Stop assuming that
    /etc/default/grub.d/90-autopkgtest.cfg exists.
  * systemd-fsckd autopkgtest: Add missing plymouth test dependency.
  * debian/tests/boot-smoke: Allow 10 seconds for systemd jobs to settle down.
  * Fix "tentative" state of devices which are not in /dev (mostly in
    containers), and avoid overzealous cleanup unmounting of mounts from them.
    (LP: #1444402)
  * journal: Gracefully handle failure to bind to audit socket, which is known
    to fail in namespaces (containers) with current kernels. Also
    conditionalize systemd-journald-audit.socket on CAP_AUDIT_READ.
    (LP: #1457054)
  * Add sigpwr-container-shutdown.service: Power off when receiving SIGPWR in
    a container. This makes lxc-stop work for systemd containers.
    (LP: #1457321)

 -- Martin Pitt <email address hidden> Thu, 21 May 2015 14:47:46 +0200

[Chris J Arges (arges)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.