udev sometimes doesn't apply systemd tag to devices

[Seth Forshee]

Since upgrading this machine to yakkety I've seen periodic hangs during booting. If I wait long enough systemd give up and drops to maintenance mode. Looking in the journal I see events like this:

Sep 19 08:24:22 ubuntu-xps13 systemd[1]: dev-disk-by\x2duuid-63cd6483\x2ddf95\x2d4426\x2d916e\x2d8c28becbfd8f.device: Job dev-disk-by\x2duuid-63cd6483\x2ddf95\x2d4426\x2d916e\x2d8c28becbfd8f.device/start timed out.

Searching on the internet indicates that this happen if devices reported by udev don't have the "systemd" tag, and I find that they do not have the tag when the boot times out but they do have the tag when boot does not time out.

This happens once for every 3 or 4 boots and with both 4.4 and 4.8 kernels. I'm attaching the output of 'journalctl -xb' and 'udevadm info --export-db' from a timed-out boot.

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: udev 231-6
ProcVersionSignature: Ubuntu 4.8.0-11.12-generic 4.8.0-rc6
Uname: Linux 4.8.0-11-generic x86_64
ApportVersion: 2.20.3-0ubuntu7
Architecture: amd64
CurrentDesktop: Unity
Date: Mon Sep 19 09:27:08 2016
EcryptfsInUse: Yes
InstallationDate: Installed on 2015-02-19 (577 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Alpha amd64 (20150110)
MachineType: Dell Inc. XPS 13 9343
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.8.0-11-generic root=UUID=bdad03f3-06c0-439d-8869-32fad0b1851b ro quiet splash
SourcePackage: systemd
UpgradeStatus: Upgraded to yakkety on 2016-01-08 (254 days ago)
dmi.bios.date: 05/15/2015
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A04
dmi.board.name: 0TM99H
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 9
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA04:bd05/15/2015:svnDellInc.:pnXPS139343:pvr01:rvnDellInc.:rn0TM99H:rvrA00:cvnDellInc.:ct9:cvr:
dmi.product.name: XPS 13 9343
dmi.product.version: 01
dmi.sys.vendor: Dell Inc.

[Seth Forshee (sforshee)]

This is still happening, and it feels like a race. I've enabled debug logging and captured the journal from good and bad boots. Comparing them hasn't told me a whole lot; on the whole the udev debug logging isn't very applicable. I do see that in both cases udev starts up and reads the exact same set of rule files, and the statistics it prints about the rule files are identical. Both have read in /lib/udev/rules.d/99-systemd.rules which contains the rule for adding the systemd tag. Nothing else has really stood out yet, but they are pretty difficult to compare.

I also find that if I run 'udevadm trigger -c add /sys/devices/pci0000:00/0000:00:1f.2/ata4/host3/target3:0:0/3:0:0:0/block/sda' from recovery mode that udev does add the systemd tag at that point.

I'll attach my logs and udev databases from the good and bad boots.