boot fails if /dev/cgroup fstab entry is present

[Robert Hardy]

I recently upgraded a server from 14.10 to 15.04. The upgrade claim to have worked but the system could not boot afterwards.

Basically when you attempt to reboot you end up at an emergency maintenance prompt early in the boot process.
After a lot of digging one of the messages in the logs gives:
systemd[1]: /usr appears to be on its own filesystem and is not already mounted. This is not a supported setup. Some things will probably break (sometimes even silently) in mysterious ways. Consult http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken

Basically systemd cannot deal with /usr being on a separate partition. That needs to be fixed or systemd should not be used for upgrades with a separate /usr partition.

This is related to bug 1460790 but asking for systemd to actually be fixed.
Please see bug 1460790 if you are looking for a work around on this issue.

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: systemd 219-7ubuntu5
ProcVersionSignature: Ubuntu 3.19.0-18.18-generic 3.19.6
Uname: Linux 3.19.0-18-generic x86_64
ApportVersion: 2.17.2-0ubuntu1.1
Architecture: amd64
Date: Mon Jun 1 15:29:32 2015
InstallationDate: Installed on 2010-05-23 (1835 days ago)
InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
MachineType: Supermicro X8STi
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_CA.UTF-8
 SHELL=/bin/tcsh
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-18-generic root=UUID=676235cd-93b1-426e-8336-fa0078d96145 ro quiet acpi=off pci=noacpi
SourcePackage: systemd
UpgradeStatus: Upgraded to vivid on 2015-06-01 (0 days ago)
dmi.bios.date: 09/17/10
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 2.0
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: X8STi
dmi.board.vendor: Supermicro
dmi.board.version: 1234567890
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 1234567890
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.0:bd09/17/10:svnSupermicro:pnX8STi:pvr1234567890:rvnSupermicro:rnX8STi:rvr1234567890:cvnSupermicro:ct3:cvr1234567890:
dmi.product.name: X8STi
dmi.product.version: 1234567890
dmi.sys.vendor: Supermicro

[Robert Hardy (rhardy)]

Please note attempting to mitigate this by switching back to upstart failed as well.

Doing so did make it so the system would boot past mounting /usr however for no obvious reason the system stopped automatically mounting the /var partition.

With no network services running, the system would stop the boot process on a screen and complain /var hadn't mounted and prompt me to press m to enter emergency mode. Once I did that and logged in, /var was in fact not mounted and a mount -av would immediately mount it and then exiting emergency mode would cause a normal boot to complete. This should be a blocking issue for 16.04 as it will cause a lot of grief to end users when servers fail during upgrades. If I didn't have an IPMI remote console device I would have been really stuck.

To be clear the only fix that worked was connecting using an IPMI remote console device, booting my server from a rescue iso, taring up each of my partitions with tar cpf and dumping them on a disk outside my raid array. I then wiped all the partitions off my main raid array and switched to a configurations where I had 1 primary boot, 1 primary swap and a root and home partition in a LVM physical volume. I then formated boot ext2, root btrfs and home xfs and extracted all the tars. I then follow a process similar to the one below to reinstall grub: https://help.ubuntu.com/community/Grub2/Installing#via_ChRoot
At the end of the process, inside the chroot, I also did an: update-initramfs -u

The result was a working system but was way way more effort than should have been required.

[Martin Pitt (pitti)]

The "/usr appears to be on its own filesystem ..." is just a warning. Booting with separate /usr is supported, as long as it's a local partition and not a remote one (the latter could work, but really nobody tests this). I just did a 15.04 installation with separate /usr, and it works fine.

Could there be something wrong with your /etc/fstab perhaps? Maybe some UUID mismatch or so? In the emergency shell, can you please do

  journalctl -b > /root/journal.txt
  blkid > /root/blkid.txt

and attach /etc/fstab, /root/blkid.txt, and /root/journal.txt here? Thanks!

[Robert Hardy (rhardy)]

On 2015-06-10 08:59, Martin Pitt wrote:
> The "/usr appears to be on its own filesystem ..." is just a warning.
> Booting with separate /usr is supported, as long as it's a local
> partition and not a remote one (the latter could work, but really nobody
> tests this). I just did a 15.04 installation with separate /usr, and it
> works fine.
>
> Could there be something wrong with your /etc/fstab perhaps? Maybe some
> UUID mismatch or so? In the emergency shell, can you please do
>
> journalctl -b > /root/journal.txt
> blkid > /root/blkid.txt
>
> and attach /etc/fstab, /root/blkid.txt, and /root/journal.txt here?
> Thanks!
>
> ** Summary changed:
>
> - systemd will not work with a separate /usr partition
> + systemd does not boot with a separate /usr partition
>
> ** Changed in: systemd (Ubuntu)
> Status: New => Incomplete
>

I've been working with Linux for 20 years, I really doubt there was
anything wrong with my fstab. The system configuration had been working
perfectly since it was installed using Ubuntu 10.04 several years ago.

The upgrade from 14.04 to 14.10 worked fine. It was the 14.10 to 15.05
upgrade that blew up.

It was not just a warning. The system and emergency mode were both
unusable until I switched back to upstart.

Even after the switch to upstart the 15.04 system would only boot only
with manual intervention on the console during every boot as /var
stopped being automatically mounted.

The system sat there with an error not booting until I manually did a
mount -av from emergency mode and then continued the boot process. This
happened on each and every boot.

In my honest opinion to test this properly you would have to test a
system which had been upgraded from 14.04 with a similar layout with
separate ext4 partitions. You might get away with starting at 14.10...

This was a total system failure on a production server over ten days
ago. I couldn't just leave it failed state for two weeks. The journalctl
and blkid commands would only show the current system state after all
was rebuilt, where everything is working fine. Do you still want them?

As I indicated previously in my comments I was forced to wipe all my
partitions from a rescue disk and switch to a boot, root, home partition
layout abandoning my previous system layout. I then restored from backups.

I was able to restore my old /etc/fstab from a backup. Before the
upgrade it was as follows:
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda1 during installation
UUID=676235cd-93b1-426e-8336-fa0078d96145 / ext4 errors=remount-ro
0 1
/dev/mapper/Guru2VG1-home /home xfs defaults 0 2
/dev/mapper/Guru2VG1-tmp /tmp ext4 defaults 0 2
/dev/mapper/Guru2VG1-usr /usr ext4 defaults 0 2
/dev/mapper/Gu...

[Martin Pitt (pitti)]

>I've been working with Linux for 20 years, I really doubt there was anything wrong with my fstab.

systemd reports errors which sysvinit and upstart just ignore (i. e. silently ignoring wrong UUIDs, and just not pointing that partition, etc.). We got a lot of bug reports due to that, which is why I now routinely ask for that. (Sorry if you felt offended in some way, it certainly wasn't intended).

> All a journalctl -b would gather now would be the boot log data 9 days after this was all fixed. The blkid would
capture uuids for the new partitions.

Ack, that wouldn't be of use indeed. We'd need the logs from when it failed.

So what I can do is to install a 14.04 system with LVM and separate /home, /tmp/, /usr, and /var, upgrade that to 14.10 and 15.04, and see if I can reproduce this.

[Martin Pitt (pitti)]

> So what I can do is to install a 14.04 system with LVM and separate /home, /tmp/, /usr, and /var, upgrade that to 14.10 and 15.04, and see if I can reproduce this.

I just did that, and the upgrade worked without a hitch. So I'm afraid we still don't have a reproducer for this..

/dev/vda1 on /boot type ext2 (rw,relatime)
/dev/mapper/ubu-ubu--usr on /usr type ext4 (rw,relatime,data=ordered)
/dev/mapper/ubu-ubu--tmp on /tmp type ext4 (rw,relatime,data=ordered)
/dev/mapper/ubu-ubu--home on /home type ext4 (rw,relatime,data=ordered)
/dev/mapper/ubu-ubu--var on /var type ext4 (rw,relatime,data=ordered)

[Robert Hardy (rhardy)]

I dug into this more on the server that failed to upgrade properly to see if the server would work properly after switching back to systemd. Please recall this was after a backup and restore with a much simpler partitioning scheme. It still would not boot.

I ended up finding this line in my fstab:
cgroup /dev/cgroup cgroup defaults 0 0

This was not a line I added. I believe it may be a remenant from an old release which the upgrader did not comment out.
The server began its life with Ubuntu 10.04.

Once I commented that cgroup line out the system booted with systemd in place. Since I doubt this will ever get to a complete state please close the bug.

[Martin Pitt (pitti)]

> cgroup /dev/cgroup cgroup defaults 0 0

Erk -- that's what broke the boot after all? Indeed, this was in comment 4 already, but I missed that. So the journal should have a complaint/timeout on that, as /dev/cgroup is a thing of the distant past (google still has a few hits). Thanks for finding!

I'm going to poke around in 10.04's and 12.04's packages to see which one would possibly have added that fstab line.

[Robert Hardy (rhardy)]

I am not convinced that the cgroup issue was the only issue here, but
since this is a single physical I can't easily go back in time to test
further.

Either way the changed summary is backwards. Once repartitioning was
completed systemd fails to boot if the /dev/cgroup fstab entry IS
present.

The one obvious thing we have is a case where do-release-upgrade should
have removed the /dev/cgroup fstab entry and did not.

A better summary might be something like:
+ boot fails with systemd when /dev/cgroup fstab entry is present

On Thu, July 2, 2015 2:44 am, Martin Pitt wrote:
>> cgroup /dev/cgroup cgroup defaults 0 0
>
> Erk -- that's what broke the boot after all? Indeed, this was in comment
> 4 already, but I missed that. So the journal should have a
> complaint/timeout on that, as /dev/cgroup is a thing of the distant past
> (google still has a few hits). Thanks for finding!
>
>
> I'm going to poke around in 10.04's and 12.04's packages to see which
> one would possibly have added that fstab line.
>
> ** Summary changed:
>
>
> - systemd does not boot with a separate /usr partition
> + boot fails on missing /dev/cgroup fstab entry
>
>
> ** Changed in: systemd (Ubuntu)
> Status: Incomplete => Confirmed
>
>
> --
> You received this bug notification because you are subscribed to the bug
> report. https://bugs.launchpad.net/bugs/1460794
>
>
> Title:
> boot fails on missing /dev/cgroup fstab entry
>
> Status in systemd package in Ubuntu:
> Confirmed
>
>
> Bug description:
> I recently upgraded a server from 14.10 to 15.04. The upgrade claim to
> have worked but the system could not boot afterwards.
>
> Basically when you attempt to reboot you end up at an emergency
> maintenance prompt early in the boot process. After a lot of digging one
> of the messages in the logs gives: systemd[1]: /usr appears to be on its
> own filesystem and is not already mounted. This is not a supported setup.
> Some things will probably break (sometimes even silently) in mysterious
> ways. Consult
> http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken
>
>
> Basically systemd cannot deal with /usr being on a separate partition.
> That needs to be fixed or systemd should not be used for upgrades with
> a separate /usr partition.
>
> This is related to bug 1460790 but asking for systemd to actually be
> fixed. Please see bug 1460790 if you are looking for a work around on this
> issue.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 15.04
> Package: systemd 219-7ubuntu5
> ProcVersionSignature: Ubuntu 3.19.0-18.18-generic 3.19.6
> Uname: Linux 3.19.0-18-generic x86_64
> ApportVersion: 2.17.2-0ubuntu1.1
> Architecture: amd64
> Date: Mon Jun 1 15:29:32 2015
> InstallationDate: Installed on 2010-05-23 (1835 days ago)
> InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64
> (20100427)
> MachineType: Supermicro X8STi
> ProcEnviron:
> TERM=xterm
> PATH=(custom, no user)
> XDG_RUNTIME_DIR=<set>
> LANG=en_CA.UTF-8
> SHELL=/bin/tcsh
> ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-18-generic
> root=UUID=676235cd-93b1-426e-8336-fa0078d96145 ro quiet acpi=off
> pci=noacpi SourcePackage: systemd
> UpgradeStatus: Upgraded to vivid on 2015-06-01 (0 da...

[Martin Pitt (pitti)]

Sure, I adjusted the bug title. My "grep the whole archive for /dev/cgroup" search is still running (so far it didn't turn up anything), I'll follow up here once it's finished. If there's a package which sets it, its postinst should clean it up on upgrade, so that this works with apt-get too. If there is no package which sets it, we shouldn't mess around with fstab entries which Ubuntu didn't set by itself.

[Martin Pitt (pitti)]

The archive grep for 12.04 (precise) is done. I see a bunch of (now outdated) documentation wrt. /dev/cgroup, particularly in the kernel sources and in libvirt. ltp also uses it heavily in its test suite. But aside from that there is no maintainer script, installer, or other thing which would create this fstab entry.

archive grep for 10.04 (lucid) is still ongoing, but I don't expect a match there (there's none so far). So I figure this fstab line was added by some third-party package or manually?

Note that in the case of nonexisting fstab entries you are supposed to get a friendly-recovery menu or an emergency shell. This doesn't currently happen, which is perhaps the main issue here? This is being tracked in bug 1471258.

[Martin Pitt (pitti)]

lucid archive grep finally finished, too. It's basically Documentation/cgroups/cgroups.txt from teh various kernel sources, plus libvirt documentation/config file, and the ltp tests again.

[delijati (delijati)]

I had the same problem, thanks for this ticket and the provided fix:

$ cat /etc/fstab
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/vda1 during installation
UUID=8b76c1d0-c6b8-46d7-9cf8-0c5af739bc60 / ext4 errors=remount-ro 0 1
# swap was on /dev/vda5 during installation
UUID=85bb4129-a290-4854-aab6-8ffe14be3b84 none swap sw 0 0
# cgroup /cgroup cgroup

[Robert Hardy (rhardy)]

This shouldn't have been marked invalid. It has caused me and apparently others grief. It wouldn't be hard to have the upgrade tool check for uncommented cgroup lines, given they blow up an upgrade.
I'm glad my experience and work around helped someone else who ran into the same things.