fails to boot with broken or missing root fs fstab entry

[Eugene]

I have just recently done a sudo apt-get upgrade, noting something about an included 219 update -- after consequent reboot, the system now hangs at starting 219. No light on Caps Lock. I have since been using upstart.
Kernel using - I am on: Linux kubuntu 3.19.0-9-generic #9-Ubuntu SMP Wed Mar 11 17:50:03 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Same error when I try the 3.19.0.7-generic kernel.
I have updated Grub, no change.

[Martin Pitt (pitti)]

Can you boot in rescue mode? Can you boot with upstart?

Please boot without "quiet splash $vt_handover", and see how far you get. (Make a photo of the screen or so, if you don't get far enough to get a terminal). It would also help to boot with a debug shell, and get the list of pending jobs there; for that, see /usr/share/doc/systemd/README.Debian.

Thanks!

[Vasco Alves (vascofalves)]

Confirming this. I can boot with upstart, no problem. Will try what you suggested Martin, but thought I'd confirm OP is not the only one.

[Ricard (jeanmi-ricard)]

To fix this problem, do:
sudo systemctl enable sddm.service -f
sudo reboot

[Martin Pitt (pitti)]

What does "hang" mean? what do you see exactly? can you switch VTs with Ctrl+Alt+F1 to F7? can you log into the text console? If it's sddm hanging, that should be possible?

[Vasco Alves (vascofalves)]

Okay, when I don't remove the "quiet splash $vt_handover" flags from grub, all I see after the Plymouth splash screen disappears is "starting version 219," white text on black screen as usual for booting. I can switch screens, but when I log in, it tells me that the file-system has been mounted read only.

[Vasco Alves (vascofalves)]

By screens I mean VTs, of course.

[Vasco Alves (vascofalves)]

Trying to boot without "quiet splash $vt_handover" results in this screen.

[Vasco Alves (vascofalves)]

I can change VTs here as well. I still can't login properly, but switching back to F7 then gives this screen.

[Vasco Alves (vascofalves)]

Richard's fix didn't work btw.

[Martin Pitt (pitti)]

hang1.jpg looks like it didn't even start booting into the system and it's stuck in initramfs during kernel boot. I had such a case this morning on my computer, it would consistently fail to boot with the new 3.19.0-8 kernel. Booting with the previous 3.19.0-7 or upgrading again to 3.19.0-9 worked, though. To test that, can you choose the previous kernel in grub (-7) and see if that looks different? You can add "break=bottom" to the kernel command line to get an initramfs-tools shell right before it would start to boot the real system. If it doesn't even get that far, it's very likely a regression in the kernel.

hang2.jpg did get into the real system. For that, can you boot with a debug shell (see /usr/share/doc/systemd/README.Debian), do "journalctl -b > /root/journal.txt, and attach /root/journal.txt here? Thanks!

[Ricard (jeanmi-ricard)]

Did you do CTRL + ALT + F1? and log in?
Then, sudo systemctl enable sddm.service -f
sudo reboot

you can also try to do:
sudo systemctl disable plymouth.service
sudo apt-get remove plymouth
sudo reboot

[Vasco Alves (vascofalves)]

Here's the journal file. I'm pretty sure it's not a kernel regression as it happened with the previous version too, I was just waiting to see if a fix came around without me having to report it :) . I can try and get a mainline kernel from kernel-ppa and try with it if you think that would be helpful, though.

[Martin Pitt (pitti)]

Indeed, the journal is full of errors due to "read-only file system". I don't really see an error why that is (systemd-remount-fs.service succeeded), except perhaps this one:

Mar 13 14:41:27 Vasco-netbook systemd[1]: /etc/mtab is not a symlink or not pointing to /proc/self/mounts. This is not supported anymore. Please make sure to replace this file by a symlink to avoid incorrect or misleading mount(8) output.

and debian-fixup.service which is supposed to fix that doesn't work because it's readonly:

Mar 13 14:41:28 Vasco-netbook debian-fixup[230]: ln: cannot remove ‘/etc/mtab’: Read-only file system
Mar 13 14:41:28 Vasco-netbook systemd[1]: debian-fixup.service: main process exited, code=exited, status=1/FAILURE

So in the debug shell, can you please do the following:

  * Verify that the root partition is read-only; something like "touch /test" should fail with "read-only file system".
  * run "mount -o remount,rw /" ; does that work, i. e. can you run "touch /test" after that? If so, "rm /test" again.
  * If mounting rw works, please do "mv /etc/mtab /etc/mtab.old; ln -s /proc/mounts /etc/mtab", and see if that cures things?

Thanks!

[Lee (k-ubuntuone-4)]

Also affects me. Boots with upstart but hangs at starting 219. After this message it tried to access every removable device and errored on each one with a "no medium found on dev/sdx" message. After unplugging my card reader from the motherboard these errors disappeared but the 219 error remained.

[Vasco Alves (vascofalves)]

Remounting from the debug shell works, and allowed me to link mtab to /proc/mounts. However, on reboot the error manifested again! I attach the journal from after the reboot.

[Vasco Alves (vascofalves)]

Also, here are the contents of /proc/mounts, in case that helps.

[Martin Pitt (pitti)]

OK, thanks. I ran out of ideas why the file system is readonly. I'll continue here in 1.5 weeks when I'm back from vacation. In the meantime I suggest you install "upstart" again to get a working system.

[Martin Pitt (pitti)]

@Vasco: I have infinite trouble with the latest -8 and -9 kernels, resulting in quite a similar situation as in your "hang1.jpg". Could you try booting -7 again? That was the last kernel that worked for me, perhaps it's similar to your situation?

[Vasco Alves (vascofalves)]

Just tried that, and unfortunately it doesn't work. :( I also tried the most recent mainline kernel from kernel ppa (4.0 rc3 IIRC), and it didn't work either. I don't think it's a kernel issue.

I also tried running "mount -o remount,rw /" off a normal VT rather than the debug shell, and it worked and gave me write access to the file system as the normal user. However, I wasn't able to start sddm from the VT or access the GUI at all.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Eugene (eugene-vanderwatt)]

Hi, sry for being late. I have been using upstart all along.
I am happy to say I tried the workaround Ricard suggested and it worked for me:" Did you do CTRL + ALT + F1? and log in?
Then, sudo systemctl enable sddm.service -f
sudo reboot"

Not sure what this means for things going forward, at least it gets me in. Is there anything I need to do further?

[Didier Roche (didrocks)]

It seems that they are 2 independent issues that were mixed in this bug report. As Vasco seems to have something closer to the initial issue, let's use if for it. i'll try ssdm and poke Eugene if more info were needed.

[Vasco Alves (vascofalves)]

I think I worked it out. I added my root partition to /etc/fstab, and that seems to have fixed the problem, though I'll try a few more runs to make sure!

[Martin Pitt (pitti)]

So the "sddm not enabled" issue is a but in sddm, and the "root partition missing from /etc/fstab" was a local problem. Hence I think this can be closed.

[Vasco Alves (vascofalves)]

But upstart managed to boot without the root partition being written in /etc/fstab. So it's a regression. I think it should be possible to write an upgrade script to check if there's a root file-system in /etc/fstab and add it if there isn't.

[Toni Helenius (toni-helenius)]

I upgraded from Kubuntu 14.10 to 15.04 release and was greeted with this problem. I don't think also that "Invalid" flag is correct here. A real problem. Black screen on boot after update is not nice. Fortunately I found this -> selecting upstart boot I was able to get the system working.

I haven't checked my /etc/fstab yet.

[Toni Helenius (toni-helenius)]

What I am missing on my fstab? And is there a way to regenerate it?

[Martin Pitt (pitti)]

Toni, please don't follow up on already closed bugs that you did not report. Please open a new one (ubuntu-bug systemd), and attach fstab and "journalctl -b" output. Thanks!

[ironstorm (ironstorm-gmail)]

This should be reopened.

This is a potentially serious regression that broke on my machine upgrading from 14.10 to 15.04. The way in which it broke left me looking at a shell that told me despite my FS being mounted "rw" according to "mount", all commands were failing because of "Read-only file system".

It is in no way intuitive why it happens, nor the solution that adding an entry to /etc/fstab for the root would fix it, given that that fstab entry was not required in 14.10 and prior.

I'm fortunate enough to be experienced enough to find this report and execute follow the steps, many non-power/novice users will not be.

This issue could have been avoided altogether by implementing a post-install check that examines the fstab and corrects the issue or at least notifies the user to correct the issue before rebooting.

[Vasco Alves (vascofalves)]

As one of the original reporters, I agree completely with ironstorm.

[ibob63 (james-tuthill)]

I agree with ironstorm. This bug clearly needs to be reopened.

I just upgraded my server and it became read only. I can confirm that the problem for me was that after upgrade the fstab had the wrong UUID in it. Bizarrely, it was changed from:

UUID=18254707-08e8-494e-b456-938592928a5e / ext4 errors=remount-ro 0 1

to:

UUID=815063a9-c956-44a6-ab11-05e1d0bb3a58 / ext4 errors=remount-ro 0 1

The server was virtual and luckily I took a backup before upgrading. I therefore restored the backup, upgraded again and then dropped into the command prompt and corrected the fstab before rebooting the server.

As Ironstorm suggests, this should be a post install check or perhaps the bug which is causing this issue can be resolved. I'm sure that this issue will affect lots of other people and it should be reopened as an issue.

[ibob63 (james-tuthill)]

I agree with ironstorm. This bug clearly needs to be reopened.

I just upgraded my server and it became read only. I can confirm that the problem for me was that after upgrade the fstab had the wrong UUID in it. Bizarrely, it was changed from:

UUID=18254707-08e8-494e-b456-938592928a5e / ext4 errors=remount-ro 0 1

to:

UUID=815063a9-c956-44a6-ab11-05e1d0bb3a58 / ext4 errors=remount-ro 0 1

The server was virtual and luckily I took a backup before upgrading. I therefore restored the backup, upgraded again and then dropped into the command prompt and corrected the fstab before rebooting the server.

As Ironstorm suggests, this should be a post install check or perhaps the bug which is causing this issue can be resolved. I'm sure that this issue will affect lots of other people and it should be reopened as an issue.

[Hans-Gregor Gehrke (b-u6untu-r)]

sorry for half finished post above!

Hi,
I think I have a related problem, that I am not able to solve.
I have the following setup:
sda=SSD (encrypted root luks)
sdb=HDD (encrypted data in encrypted lvm2)

I checked the UUIDs they are correct. The very same files work with Upstart. With systemd I end up in the emergency console after entering my passphrases. What does work, is removing the hdd-home and hdd-root from the fstab. Leaving the swap makes no trouble at all with systemd. Unfortunately I am not so familiar with either upstart or systemd.

Here are the crypttab and fstab files:

crypttab:
ssd-root UUID=565ec7d6-ecd4-47d4-b3b7-e6d09a3e45f4 none luks,discard
lukslvm UUID=bf8cb0cf-f216-4c02-a3fe-f53624e0e653 none luks

fstab:
/dev/mapper/ssd-root / ext4 defaults,noatime,nodiratime,errors=remount-ro 0 1
/dev/mapper/vgubuntu-home /home/hdd-home auto defaults,noatime 0 2
/dev/mapper/vgubuntu-root /home/hdd-root auto defaults,noatime 0 2
/dev/mapper/vgubuntu-swap none swap sw 0 0
UUID=333fc063-e0c7-43ce-bc9f-388554cbb8b1 /boot ext3 defaults 0 2

In case I can help. I would provide more information. For now, I will stick to upstart.

regards Gregor

[Martin Pitt (pitti)]

Hans-Gregor, this looks like bug 1453738 (broken swap with LVM+home dir encryption).

[suoko (suoko)]

I confirm that this bug is still there.
On a workstation I could not boot with systemd but could successfully boot with upstart.
I had to manually edit fstab and add the / line to have systemd working again.

[Uday singh (12u)]

I have a solution but it is right or wrong i am not sure but thank god it works for me ☺️
My system has stucked in maintenance mode ctrlD and blah blah blah...

#nano /etc/fstab
Comment which has dump and pass value = 0
If this is unwritable then
#cp -r fstab fstb1
#rm fstab
#mv fstab1 fstab
Now you will be able to write fstab if not try #chmod
Now comment Line#
And just reboot he machine