systemd-machine-id-commit.service fails on overlayfs

[Martin Pitt]

Boot the current live system with systemd: Go to gfxboot, select F6, press Esc twice, stay in gfxboot, and append init=/bin/systemd to the command line. The live system will come up, but degraded:

● systemd-machine-id-commit.service - Commit a transient machine-id on disk
   Loaded: loaded (/lib/systemd/system/systemd-machine-id-commit.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2015-01-15 08:08:47 UTC; 1min 57s ago
     Docs: man:systemd-machine-id-commit.service(8)
  Process: 871 ExecStart=/lib/systemd/systemd-machine-id-commit (code=exited, status=1/FAILURE)
 Main PID: 871 (code=exited, status=1/FAILURE)

Jan 15 08:08:47 ubuntu systemd[1]: Starting Commit a transient machine-id on disk...
Jan 15 08:08:47 ubuntu systemd[1]: systemd-machine-id-commit.service: main process exited, code=exited, status=1/FAILURE
Jan 15 08:08:47 ubuntu systemd[1]: Failed to start Commit a transient machine-id on disk.
Jan 15 08:08:47 ubuntu systemd[1]: Unit systemd-machine-id-commit.service entered failed state.
Jan 15 08:08:47 ubuntu systemd[1]: systemd-machine-id-commit.service failed.

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: systemd 218-3ubuntu3
ProcVersionSignature: Ubuntu 3.18.0-9.10-generic 3.18.2
Uname: Linux 3.18.0-9-generic x86_64
ApportVersion: 2.15.1-0ubuntu2
Architecture: amd64
CurrentDesktop: Unity
Date: Thu Jan 15 09:06:46 2015
EcryptfsInUse: Yes
InstallationDate: Installed on 2014-11-20 (55 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Alpha amd64 (20141119)
MachineType: LENOVO 2324CTO
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-3.18.0-9-generic.efi.signed root=UUID=f86539b0-3a1b-4372-83b0-acdd029ade68 ro rootflags=subvol=@ quiet splash vt.handoff=7
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/09/2013
dmi.bios.vendor: LENOVO
dmi.bios.version: G2ET95WW (2.55 )
dmi.board.asset.tag: Not Available
dmi.board.name: 2324CTO
dmi.board.vendor: LENOVO
dmi.board.version: 0B98401 Pro
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvrG2ET95WW(2.55):bd07/09/2013:svnLENOVO:pn2324CTO:pvrThinkPadX230:rvnLENOVO:rn2324CTO:rvr0B98401Pro:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 2324CTO
dmi.product.version: ThinkPad X230
dmi.sys.vendor: LENOVO

[Didier Roche-Tolomelli (didrocks)]

For reference:
16:29:53 didrocks | pitti: right, seems it's overlayfs messing with ConditionPathIsMountPoint

That's also why:
        r = path_is_mount_point(etc_machine_id, false);
returns 0, it's considered as a mount point in a overlayfs FS.

[Didier Roche-Tolomelli (didrocks)]

IRC context from today:
didrocks | apw: I just tried t_name_to_handle_at.c from man name_to_handle_at
apw | didrocks, it seems to be returning ENOSUPP for me
apw | didrocks, well it failed, as expected
 didrocks | apw: so, I guess this is expected that overlayfs doesn't support it
 didrocks | hum, I wonder how I can detect if a file is a mount point (meaning, I guess, in that context, being overridden)
 didrocks | apw: the systemd logic is to test a path, test the parent path, and check mount_id
 didrocks | the fallback is using stat()
 didrocks | and comparing st_dev
      apw | which sounds ok ?
      apw | oh perhaps not if the path is a file and the parent a directory
 didrocks | apw: yeah, that exactly what happens here, path is file, parent a dir
      apw | its not clear that name_to_handle_at would work any better if it did work of course
      apw | this seems like an odd way to work out what is a mountpoint to me, then again it is systemd
 didrocks | apw: just run a test, and I confirm that st_dev is different if path == file from parent == dir.
 didrocks | apw: hum, yeah, seems tricky to even think of a workaround…
 didrocks | the easiest way would be to take another file (if any) into the same dir, but doesn't seem efficient…

[Didier Roche-Tolomelli (didrocks)]

Fix proposed at http://lists.freedesktop.org/archives/systemd-devel/2015-March/029163.html

[Martin Pitt (pitti)]

Applied to Debian experimental branch, thanks!

[Martin Pitt (pitti)]

I reverted this for now. Building a package with that patch causes the upgrade to hang (logind and other services don't restart) and boot to fail. Did you test this on an installed desktop as well or just on the live system?

[Martin Pitt (pitti)]

The boot hang reproduces with applying the patch on top of upstream v219 and installing the built "systemd".

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 219-5ubuntu1

---------------
systemd (219-5ubuntu1) vivid; urgency=medium

  * Merge with Debian experimental branch. Remaining Ubuntu changes:
    - Hack to support system-image read-only /etc, and modify files in
      /etc/writable/ instead.
    - Keep our much simpler udev maintainer scripts (all platforms must
      support udev, no debconf).
    - initramfs init-top: Drop $ROOTDELAY, we do that in a more sensible way
      with wait-for-root. Will get applicable to Debian once Debian gets
      wait-for-root in initramfs-tools.
    - initramfs init-bottom: If LVM is installed, settle udev,
      otherwise we get missing LV symlinks. Workaround for LP #1185394.
    - Add debian/udev.lvm2.init: Dummy SysV init script to satisfy insserv
      dependencies to "lvm2" which is handled with udev rules in Ubuntu.
    - Provide shutdown fallback for upstart. (LP: #1370329)
    - debian/extra/ifup@.service: Additionally run for "auto" class. We don't
      really support "allow-hotplug" in Ubuntu at the moment, so we need to
      deal with "auto" devices appearing after "/etc/init.d/networking start"
      already ran. (LP: #1374521) Also, check if devices are actually defined
      in /etc/network/interfaces as we don't use Debian's net.agent.
    - ifup@.service: Drop dependency on networking.service (i. e.
      /etc/init.d/networking), and merely ensure that /run/network exists.
      This avoids unnecessary dependencies/waiting during boot and dependency
      cycles if hooks wait for other interfaces to come up (like ifenslave
      with bonding interfaces). (LP: #1414544)
    - Add Get-RTC-is-in-local-time-setting-from-etc-default-rc.patch: In
      Ubuntu we currently keep the setting whether the RTC is in local or UTC
      time in /etc/default/rcS "UTC=yes|no", instead of /etc/adjtime.
      (LP: #1377258)
    - Put session scopes into all cgroup controllers. This makes unprivileged
      user LXC containers work under systemd. (LP: #1346734)
    - systemctl: Don't forward telinit u to upstart. This works around
      upstart's Restart() always reexec'ing /sbin/init on Restart(), even if
      that changes to point to systemd during the upgrade. This avoids running
      systemd during a dist-upgrade. (LP: #1430479)
    - Lower Breaks: to plymouth version which has the udev inotify fix in
      Ubuntu.
    - Lower libappamor1 dep to the Ubuntu version where it moved to /lib.
    - Change systemd-sysv's conflicts to upstart-sysv. (LP: #1422681)
    - Make failure of boot-and-services NSpawn.test_boot non-fatal for now.
      This currently fails when being triggered by Jenkins, but is totally
      unreproducible when running this manually on the exact same machine.

    Upgrade fixes, keep until 16.04 LTS release:
    - systemd Conflicts/Replaces/Provides systemd-services.
    - Remove obsolete systemd-logind upstart job.
    - Clean up obsolete /etc/udev/rules.d/README.

systemd (219-6) UNRELEASED; urgency=medium

  [ Martin Pitt ]
  * Import patches from v219-stable branch (up to 85a6fab).

  [ Didier Roche ]
  * Fix mount point detection on overlayfs and similar file systems without
    n...