dev-sda2.device job running and times out

[Scott Moser]

$ lxc launch ubuntu-daily:yakkety y4
$ lxc exec y3 -- systemctl list-jobs
JOB UNIT TYPE STATE
  5 dev-sda2.device start running

1 jobs listed.

$ lxc exec y3 -- cat /etc/cloud/build.info
build_name: server
serial: 20160902

$ lxc exec y3 -- systemd-analyze
Startup finished in 1min 30.373s (userspace) = 6d 2h 27min 41.963s

$ lxc exec y3 -- journalctl > yakkety-lxd-journalctl.txt

$ lxc exec y3 -- systemctl status dev-sda2.device
● dev-sda2.device
   Loaded: loaded
   Active: activating (tentative) since Tue 2016-09-06 18:38:31 UTC; 4min 39s ago

Sep 06 18:40:01 y3 systemd[1]: dev-sda2.device: Job dev-sda2.device/start timed out.
Sep 06 18:40:01 y3 systemd[1]: Timed out waiting for device dev-sda2.device.
Sep 06 18:40:01 y3 systemd[1]: dev-sda2.device: Job dev-sda2.device/start failed with result 'timeout'.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: systemd 231-5
ProcVersionSignature: Ubuntu 4.4.0-9136.55-generic 4.4.16
Uname: Linux 4.4.0-9136-generic x86_64
ApportVersion: 2.20.3-0ubuntu7
Architecture: amd64
Date: Tue Sep 6 18:38:48 2016
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-9136-generic.efi.signed root=UUID=f897b32a-eacf-4191-9717-844918947069 ro quiet splash vt.handoff=7
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)

[Martin Pitt (pitti)]

In Scott's local instances he gets "systemctl is-system-running" == "starting" with

JOB UNIT TYPE STATE
  2 dev-sda2.device start running

So there is some impedance mismatch between is-system-running and Type=idle -- apparently the idle unit already starts while the above sda2.device is still running (or, perhaps, it flips *back* to starting while it is already running).

In both Scott's VM and local LXD I never get dev-sdaX.device as "running", only as "activating tentative" (and that does not time out); consequently is-system-running does not say "starting" after reaching multi-user.target, and invoke-rc.d DTRT.

Scott's debug journal: http://paste.ubuntu.com/23153994/

So currently I lack a way of reproducing that "running" state.

[Martin Pitt (pitti)]

I actually can reproduce this in a yakkety LXD container locally; in about 1/3 of the cases there is a "running" job for dev-sda2.device, otherwise not.

[Martin Pitt (pitti)]

I cannot reproduce this in xenial LXD containers, only yakkety. Do you confirm?

[Scott Moser (smoser)]

right. i only see this in yakkety.

[Martin Pitt (pitti)]

There is some chance that this got fixed with https://github.com/systemd/systemd/commit/c5a97ed1

[Martin Pitt (pitti)]

Indeed this does not happen any more with the upstream master build from https://launchpad.net/~pitti/+archive/ubuntu/systemd/+packages, so 233 will fix this.

[Scott Moser (smoser)]

I can verify that in a lxc container of zesty i installed ppa above (yakkety) and dist-upgraded.
rebooted and no dev-sda2 job.