systmd-tmpfiles-setup sometimes not run on boot

[Rafał Cieślak]

This is happening on an up-to-date 15.04,with systemd=219-7ubuntu6.
Ocasionally, when booting up the system, network-manager service fails to start, reporting in logs that it is missing /run/sendsigs.omit.d directory.
That file is indeed missing, although it is listed in /usr/lib/tmpiles.d/debian.conf
The issue is reproducible when starting the system in recovery mode; in normal mode it happens occasionally (about 50%).

Output of "journalctl -b": http://paste.ubuntu.com/11745993/
Contents of /usr/lib/tempfiles.d/debian.conf: http://paste.ubuntu.com/11746045/

Running "systemd-tmpfiles --create" after such unsuccesfull boot creates the missing directory (and some other too).

The problem is similar to #1431110, but that bug was found to be a duplicate of another problem that is now fixed. Even with that fix, I am experiencing a correlated issue. I considered commenting on that bug, but marking it as non-incomplete (so that it would gain any attention) would require unlinking the duplicate, which wouldn't be right.

[Martin Pitt (pitti)]

Indeed your journal shows that systemd-tmpfiles-setup.service wasn't called at all. After that happens, what is the output of

  sudo systemctl status -l systemd-tmpfiles-setup.service

? Can you please boot with "systemd.log_level=debug" on the kernel command line and reproduce this, then attach the journal again? Thanks!

[Rafał Cieślak (rafalcieslak256)]

Hello Martin,
here are the requested outputs:

systemctl status -l systemd-tmpfiles-setup.service
http://paste.ubuntu.com/11795241/

journalctl -b
http://paste.ubuntu.com/11795245/

This log does not include network-manager complaining about the missing directory (because it was not selected to start in this boot - recovery mode), but a manual check confirmed that the directory /run/sendsigs.omit.d was indeed missing.
Apparently, the tmpfiles service was run, but it chose to not create the needed files.
Clearly, the "Entry <file> does not match any include prefix, skipping" messages seem interesting.

[Martin Pitt (pitti)]

So systemd-tmpfiles-setup-dev.service ran fine (don't worry about the "does not match any include prefix", that's debugging output from setup-dev), but systemd-tmpfiles-setup.service never actually ran, it seems its waiting for local-fs.target to get active, which never actually happens. But if you ran this from rescue mode that's no surprise -- all this runs way before it. Can you please give me a journal output for a normal (non-rescue) boot? After such a boot, please log in, do

  systemctl list-jobs > jobs.txt
  systemctl --failed > failed.txt

and attach jobs.txt and failed.txt here as well. Thanks!

[Rafał Cieślak (rafalcieslak256)]

Here are the requested logs:

Normal boot:
journal http://paste.ubuntu.com/11801096/
jobs.txt http://paste.ubuntu.com/11801103/
failed.txt http://paste.ubuntu.com/11801107/

During a normal boot, I observe no issues. The directory is created as it should, and network-manager starts successfully.
You noticed that the logs I sent previously (from recovery mode) claim that systemd-tpmfiles is not started, because it waits for local-fs, which makes perfect sense, as [usually] filesystems are not mounted in recovery mode.
However, my original problem was that I was not able to start network-manager in recovery mode. When enabling networking from the recovery menu, file systems get mounted, so systemd-tmpfiles should start afterwards. The logs I have attached to my previous comment are from a boot into recovery mode *without* networking, which is why fs are not mounted. I have repeated the procedure when booting into recovery mode *with* networking enabled from the recovery menu (in such case the /run/sendsigs.omit.d directory is still missing, even though filesystems should be mounted) - below I attach logs from such boot.

Recovery mode with networking enabled:
systemctl status -l systemd-tmpfiles-setup.service http://paste.ubuntu.com/11801094/
journalctl -b http://paste.ubuntu.com/11801088/

I hope this makes a bit more sense now.

[Martin Pitt (pitti)]

Ah, if NM fails to start *only* in recovery mode, this would make much more sense. It seems to work fine without the /run/sendsigs.omit.d dir in vivid in my testing, but maybe some plugins need that. But that's not what you said in the original description: "in normal mode it happens occasionally (about 50%)". So if this is still the case, I need a journal output from "normal" mode when NM fails to start.

So can you please clarify: Does this still affect a normal boot, or only recovery?

[Launchpad Janitor (janitor)]

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]