Cannot boot with newly installed systemd if /tmp/ is filled with files

[Arthur Lutz]

On a new lubuntu 14.10 install, after installing a bunch of new packages, I rebooted the machine and it stalls on startup. It stays on the four dots of plymouth (not the graphical version).

After trying various options in rescue mode, I end up understanding that the boot system has switched to systemd by looking at /var/log/dpkg.log (attached).

I then tried init=/lib/systemd/systemd in grub without quiet and splash and found that it was blocking on "a start job is running for Create Volatile files and directories". A search on the internet later, I found that the problem was solved by this approach : http://forums.debian.net/viewtopic.php?f=10&t=118008

So, in rescue mode, I did a mv /tmp/ to /fulltmp/ (an ls wouldn't return so I'm guessing the /tmp/ is really full and the disk is not rocket fast). I recreated /tmp and did a chmod 1777 /tmp, reboot and it works!

While describing this, am not entirelly sure upstart is exempt from this bug (how do I check which init was used after I've booted ?)

This is a very frustrating bug since it doesn't appear on startup even when removing quiet or splash.

[Martin Pitt (pitti)]

You are still running upstart. However, it'd be good to check this under systemd as well. Under either init system, a tmpfs should be mounted on /tmp/ if the disk is full.

[Martin Pitt (pitti)]

Under upstart, /etc/init/mounted-tmp.conf is supposed to do that tmpfs mounting, but there's a chance that it's failing on trying to write to /var or another directory. So we should test this under both.

[Didier Roche (didrocks)]

Here is a patch to get under that situation /tmp mounted as tmpfs and tagged appropriatly

[Martin Pitt (pitti)]

Thanks Didier! Some remarks:

+ * Add systemd-emergency-tmpfs to force tmp.mount (tmpfs) enablement if the
 ... "generator"?
Also, s/enablement/startup/? (We don't permanently enable the unit, that could be misleading)

+avail=`df -BM -P /tmp/ | awk 'NR==2 { print substr($4, 0, length($4)-1) }'`

This needs guarding against /tmp not existing. Strange, I know, but let's better be correct. tmpfiles.d/tmp will create it later on if it's missing.

Also, this generator will trigger if tmp.mount is already (manually) enabled, right? In this case you wouldn't have an overflow in /tmp as it's overmounted later on, and that unit should stay inert. You could check for enablement symlinks in /etc, /lib, and /run, but that gets a bit fiddly... Perhaps the "After=tmp.mount" emergency-tmp.service was less intrusive after all?

[Didier Roche (didrocks)]

It seems the current plan of record (after long discussion on IRC and upstream ML) would be to enable /tmp on tmpfs on vivid+1 (it's quite late to do it this cycle, even if we switch to systemd by default).

Upstream ML reference after another approach with a service to disable tmp.mount: http://lists.freedesktop.org/archives/systemd-devel/2015-March/028966.html