© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
e3ace39
(Get the code!)
I can confirm this affects all Ubuntu >= 15.04 installations, including Ubuntu 16.04 (LTS).
> I am reasonably sure that this is not a kernel issue but a boot script dependency issue...
That's absolutely the case. Your intuition has not led you astray.
> Hopefully someone can point me in the right direction?
To no one's surprise, this is a high-level system.d issue rather than a low-level kernel issue. Ergo Ubuntu >= 15.04, the release at which Ubuntu switched from Upstart to system.d.
This issue is a significant show-stopper, particularly for users on newer UEFI-based systems. Triggering this issue is disturbingly trivial on such systems. Explicitly editing the "/etc/fstab" file with superuser permissions is *NOT* required to trigger this issue. Formatting a single device with the builtin GUI-driven disk utility ("Disks") is all it takes. And when it takes, most users will have no recourse but to format their root filesystem and reinstall.
Here's the use case my better half stumbled into this morning. During Ubuntu installation, users may elect to automount devices to arbitrary mountpoints when selecting a custom partition scheme. When this is done on UEFI-based systems, the resulting "/etc/fstab" entries resemble:
UUID=050e1e34-
If any such device is subsequently reformatted (e.g., via the stock "Disks" utility), that device's UUID will be arbitrarily changed, invalidating the UUID previously recorded for that device in "/etc/fstab". Everything will superficially appear to behave as expected. On the next reboot, however, the end user will be presented with the now-infamous Purple Screen of Death. No error messages (...human-readable or otherwise) will appear, obscuring the underlying issue.
Attempting to login with "safe mode" fails. Attempting to login to a root terminal succeeds, albeit only after a few dispiriting rounds of graphical corruption and terminal cursors disappearing. If the user successfully navigates this shameful gauntlet of pain *AND* is sufficiently familiar with low-level system administration to navigate the crucible by geek fire that is the CLI, there still exists no explicit indication of the core issue. (Command-line-fu: why have you betrayed me?)
No relevant warnings or errors appear in either "dmesg" or "journalctl" output *OR* in "/var/log" logfiles. The only clear indicator that I could grep across was a single line of "systemctl -a" output showing the status of some ignorable system service unit with a non-human-readable name to be "loaded inactive dead", which seemed dimly suspicious. Further hours of cursing and grepping yielded the final culprit. My precious sanity. I have less of it now.
Non-tech-savvy users confronted with this problem will probably just vomit, format, and reinstall. As a Gentoo-hardened Disciple of the Command-line Faith with fifteen caffeine-addled years experience in Silicon Valley startup monoculture, I feel overly confident in betting that even the most battle-weary code warriors will hand in their Richard M. Stallman fan club memberships when face-planting into this epic fail. Consider escalating this issue's importance to at least "High" – possibly even "Critical."
System.d's "NetworkManager
System.d's "/usr/lib/
To be fair, this issue is the product of numerous cascading failures – including:
* The failure of the stock "Disks" utility to detect and handle attempts to format UUID-style devices automounted by "/etc/fstab". On detecting such an attempt, this utility should either (in descending order of preference):
* Globally search-and-replace the UUID field of all entries in "/etc/fstab" matching the offending device's prior UUID with the new UUID allocated for that device. In other words, "sed -i -e 's/^UUID=
* Display a non-fatal warning that formatting this device will probably render the entire system unbootable unless the user manually modifies "/etc/fstab", but permissively permit them to do it anyway. (Non-ideal, but this is Linux. Every user should have the right to catastrophically betray themselves.)
* Display a fatal error prohibiting the user from formatting this device. (Don't do this.)
* The failure of the Ubuntu startup process to display coherent warnings or errors. I (and possibly other experienced users bludgeoned by this) would happily settle for a single line in either "dmesg" or "journalctl" output indicating the underlying issue. Ideally, however, boot failures should be detected by the startup process and:
* Rendered into human-readable GUI-based error dialogues or CLI-based error messages. In either case, the user should be presented with some digestible snippet of text for subsequent feeding to Google. Boot failures shouldn't necessitate a degree in post-graduate Computer Science.
* The user should be presented with the option of dropping into a CLI-based login prompt or (failing that) at least into a rescue terminal with superuser privileges. For safety, all currently mounted filesystems should be remounted as read-only. Because something is better than no thing. No thing and the smugness of the Purple Screen of Death is all we currently got.
* The failure of system.d. Oh, the complete failure of system.d. I don't even. On the bright side, you can only go up from rock-bottom destitution. Let's begin by belabouring the hopefully obvious:
* The failure to mount optional automounts should *NOT* render the system unbootable. (This goes without saying. Apparently, it needs saying.)
* The system.d journal ("journalctl") should explicitly log failing units. Bizarrely, it doesn't in this case. The only indication of imminent badness is a discontinuity after system.d attempts to start the "NetworkManager
* To beat the dead horse, all autogenerated system service units mounting "/etc/fstab"-listed mountpoints should be strictly optional. Exceptions might include the obvious system prerequisites (e.g., "/usr", "/var"). Since the root filesystem has (by definition) already been mounted and system.d has (by definition) already been exec-ed into the init process via PID 1, the system should at least be nominally bootable. There's little to no justification for automount failure to completely destroy bootability. Indeed, I'd probably advise all such units to be optional – regardless of whether they appear to be prerequisites or not. If "/usr" isn't mountable, for example, error messages to that effect will hopefully already be logged to "dmesg" and/or "journalctl" and/or printed to the current terminal. Moreover, subsequent units will implicitly fail by virtue of "/usr" going AWOL. System.d prematurely halting the startup process at the first unmountable automount adds no tangible benefit to anything. Let the startup process proceed with fingers and tentacles crossed in the event of automount failure and may the succubus take the hindmost.
* To that effect, all other units currently hard-requiring automount units ("local.target", possibly?) should be revised to use "Wants=
In short, I mourn the untimely death of Upstart.
It'd be awesome-sauce if either the original submitter (Mark) or another contributor with sufficient privileges could:
* Change the affected packages to at least "system.d" (and possibly "gnome-disks" as well, if that indeed be the disk utility that Ubuntu currently leverages).
* Escalate the importance to at least "High" (and possibly "Critical").
Life in the code trenches. It doesn't get uglier than this.