difficult to recover from filesystem errors

[Rocko]

Binary package hint: upstart

On two recent occasions Karmic has failed to boot after upgrading, saying that there are errors on the root partition and I needed to manually run fsck on it.

I got Karmic running again by rebooting into Jaunty to do the manual fsck, but I would have expected Karmic to be able to do this automatically, like earlier versions of Ubuntu.

I'm assuming this is something to do with the change to upstart.

ProblemType: Bug
Architecture: amd64
Date: Fri Sep 18 11:05:37 2009
DistroRelease: Ubuntu 9.10
NonfreeKernelModules: nvidia
Package: upstart 0.6.3-3
ProcEnviron:
 LANG=en_AU.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-10.34-generic
SourcePackage: upstart
Uname: Linux 2.6.31-10-generic x86_64

[Rocko (rockorequin)]

I think this might be a dup of the ext4 bug where it incorrectly thinks the file system is modified in the future. The message I just got in a VM looks like the one I was getting on the primary machine:

/dev/sda1: Superblock last mount time (Sat Sep 19 16:29:10 2009,
    now = Sat Sep 19 08:40:08 2009) is in the future

/dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.

[Scott James Remnant (Canonical) (canonical-scott)]

Do you have cryptsetup installed?

Normally mountall will return you to a shell, but it seems cryptsetup steals keyboard input from it

[Rocko (rockorequin)]

'dpkg -l | grep crypt' doesn't show cryptsetup, at least on the VM where I last saw the bug.

[Scott James Remnant (Canonical) (canonical-scott)]

I've had enough reports of this kind of failure to believe that the current approach of dealing with errors isn't ideal.

[Shrini (kshrinivas-2000)]

I had this problem too. What I figured out was that, after installation, (irrespective of timezone you might have selected) the system time is not set correctly.

But while karmic gets booted, it tries to check superblock time against timezone you had selected, obviously gets to conclusion that the superblock was modified in future. This results in warning & booting to root shell to run fsck manually.

My experience was, next time when I again setup karmic and made sure that timezone is set correctly before I reboot after first install, it worked & I no more get into fsck issue.

[Rocko (rockorequin)]

I've seen it happen after an 'apt-get upgrade' and reboot cycle - not just after installation.

[Felix Geyer (debfx)]

Here is a screenshot of fsck asking me to run it manually because the last mount time is in the future.

[Steve Langasek (vorlon)]

Didn't make it for beta, deferring to final.

[david (davidelizondo2006)]

I nose you fucking hell's going on but the primary file system is failing
  grub2 ext4 librase might not get along
  This is serious since missing 28 days for this in Ubuntu 9.10 stable version is above 9.04 when vergueza demaciado was stable in its beta version is going to stop this poniewndo new things and see what happens bug # 432,237 <----! !

[Mtt.Castelli (mtt.castelli)]

I installed today Karmic beta and had this bug.
Not the first, but the second time i reboot after some upgrade (obviusly, I'm _not_ considering live boot to install!).

What should I do for now? Running fsck from gparted of live cd do not resolve the problem, why?

[Philipp Merkel (plippo)]

One workaround I found was using the magic SysRq keys. With SysRq+R SysRq+E I got to a root shell from where I could run fsck and reboot.

[Mtt.Castelli (mtt.castelli)]

So, the packet we should not update is 'upstart'?

[Boblemur (boblemur)]

here is the data iv gathered:

:~# date ( wrong, in the root shell that i was given after the system failed to boot)
Sat Oct 3 22:50:39 EST 2009

real time:
@desktop:~$ date (correct, on my other computer)
Sat Oct 3 23:53:58 EST 2009

bios reports:

System time: 23:54:50 (correct, bios on the computer that failed to boot)
System Date: 10/03/2009

on a healthy system:

root@desktop:~# date
Sun Oct 4 00:09:33 EST 2009
root@desktop:~# hwclock
Sun 04 Oct 2009 00:09:36 EST -0.555512 seconds
root@desktop:~# date
Sun Oct 4 00:09:40 EST 2009

on a broken system:

root@nick-laptop:~# date
Sat Oct 3 23:07:57 EST 2009
root@nick-laptop:~# hwclock
Sat Oct 4 00:08:23 2009 -0.953406 seconds
root@nick-laptop:~# date
Sat Oct 3 23:08:30 EST 2009

however if i leave my laptop for an hour, it will boot again. i cannot have more than 1 reboot per hour as while booting the time is wrong and the current time will appear in the past.

POSSIBLE WORK AROUND:

i can boot if i change the time 1 hour into the future (in bios) so when ubuntu runs hwclock and then saves the time wrong it is still in the "past".

please confirm if this workaround is effective for you.
oddly enough however the time is still right once its fully booted.

[Boblemur (boblemur)]

also, heres a log of times etc from me booting my system and looking at the clocks more closely. thank god for the boot time speedups thats all i have to say :D

so far what i think the problem is, ubuntu is reading the time wrong and then "correcting" the hardware clock when it successfully boots, this is fine under normal circumstances, however when it loads the time incorrectly at boot and then updates the clock it is losing an hour when it "fixes" the time. so fix the problem at boot and everything should be fine.

again this is just a theory i could have completely missed the mark hope this helps :)

[Glenn Brumfield (gbrumfield)]

When I installed Karmic beta, I selected "Log-in and encrypt HDD". Now the boot sequence complains about file system time in the future, demands a manual fsck, bypasses the command prompt, then hangs attempting to access the encrypted HDD. Keyboard inputs have no effect.

I've attached a screenshot of the hang point.

[FriedChicken (domlyons)]

I wre affected by this bug, too. I manually started fsck which worked without problems. But on trying to continue booting by pressing CTRL+D (as the error message says) mountall block the furter boot process:

   Filesystem check failed.
   A Maintenance shell will now be started.
   CONTROL-D will terminate this shell and retry.

   # fsck /boot
time in superblock gets corrected as aexpected, but to be shure:
   # fsck /boot
   /dev/sda8: clean, x/y files, x/y blocks

Now pressing CTRL+D:
   # exit
   montall start/starting
=> Everything is blocked, ALT+SysRq+R/E/I/S/U/B was the only way out I found.

Is this another bug? Or is it a direct consequence of this bug?

[David Hardstone (dhardstone)]

It is kind of another bug but I think it's all being tracked here.

[Scott James Remnant (Canonical) (canonical-scott)]

This has nothing to do with the "GNU R package for bootstrapping functions from Davison and Hinkley"

[Anders Kaseorg (andersk)]

For me, when fsck requests manual intervention and drops you to a shell, usplash now stays open so that all you see is an Ubuntu logo on a black screen. I had to guess what the problem is, then switch virtual consoles a few times to find the shell.

[Scott James Remnant (Canonical) (canonical-scott)]

I've uploaded a new mountall package to the ubuntu-boot PPA:

https://launchpad.net/~ubuntu-boot/+archive/ppa

I would appreciate it if you could install this and try it out. *BEFORE* you reboot though, could you run "sudo mountall --debug > mountall.log 2>&1" and attach that to this bug - then after you reboot, let me know whether it worked or not.

Thanks

[Rocko (rockorequin)]

Is there a recommended way to readily reproduce the problem? I'm happy to test the new package but I haven't seen the problem in a while.

[Ernst Zlo (ernst-zlo)]

Scott
I've installed your update:
VirtualBox now hangs with the mounting of the shared folder. forever.

hehehe I've to reinstall as I suppose
(because I was silly enough to forget to make a snapshot. and yes this IS my fault <g>)

IMHO :^Þ

[Dave Martin (dave-martin-arm)]

Before installing your update, running sudo mountall --debug on a booted system (this is mountall version 0.1.8) produces the attached output and appears to deadlock. The mountall process does not terminate.

[Ernst Zlo (ernst-zlo)]

Scott
I've removed the line
sudo mount -t vboxsf -o uid=1000,gid=1000 extras /media/xtra

After that the boot resulted in this output (trying 3 rounds with no better result)

[Ernst Zlo (ernst-zlo)]

Nr 2

[Ernst Zlo (ernst-zlo)]

No 3
this is the point of no return only STRG+ALT+DEL helps out and leads back to here.

[Dave Martin (dave-martin-arm)]

After installing the updated mountall package on armel, I can't boot at all...

By default, mountall throws a lot of "Read-only filesystem" errors while trying to clean /tmp, and drops to a root shell.

Alternatively, if I remount the root filesystem read-write before running init, init (or possibly mountall) appears to deadlock; no messages appear, and the platform does not boot.

[Scott James Remnant (Canonical) (canonical-scott)]

On Thu, 2009-10-08 at 11:17 +0000, Ernst Zlo wrote:

> I've installed your update:
> VirtualBox now hangs with the mounting of the shared folder. forever.
>
Thanks. I've fixed this issue, new upload to the PPA is pending.
Again, I would appreciate testing.

Scott
--
Scott James Remnant
<email address hidden>

[Scott James Remnant (Canonical) (canonical-scott)]

On Thu, 2009-10-08 at 11:24 +0000, Dave Martin wrote:

> Before installing your update, running sudo mountall --debug on a booted
> system (this is mountall version 0.1.8) produces the attached output and
> appears to deadlock. The mountall process does not terminate.
>
> ** Attachment added: "mountall.log"
> http://launchpadlibrarian.net/33292692/mountall.log
>
This log is not from the PPA, I'd prefer to see the log from the PPA.

Scott
--
Scott James Remnant
<email address hidden>

[Scott James Remnant (Canonical) (canonical-scott)]

On Thu, 2009-10-08 at 11:53 +0000, Dave Martin wrote:

> By default, mountall throws a lot of "Read-only filesystem" errors while
> trying to clean /tmp, and drops to a root shell.
>
> Alternatively, if I remount the root filesystem read-write before
> running init, init (or possibly mountall) appears to deadlock; no
> messages appear, and the platform does not boot.
>
I believe I have fixed this bug, new upload pending in the PPA - again I
would appreciate testing.

Scott
--
Scott James Remnant
<email address hidden>

[Dave Martin (dave-martin-arm)]

> > ** Attachment added: "mountall.log"
> > http://launchpadlibrarian.net/33292692/mountall.log
> >
> This log is not from the PPA, I'd prefer to see the log from the PPA.

Apologies... looks like I misunderstood your instructions there...

[...]

> > By default, mountall throws a lot of "Read-only filesystem" errors
> > while trying to clean /tmp, and drops to a root shell.
> >
> > Alternatively, if I remount the root filesystem read-write before
> > running init, init (or possibly mountall) appears to deadlock; no
> > messages appear, and the platform does not boot.
> >
> I believe I have fixed this bug, new upload pending in the
> PPA - again I would appreciate testing.

OK... currently under way. I will take a short while since PPAs don't
magically build for armel yet...

--
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[Dave Martin (dave-martin-arm)]

Hi there...

with ~boot3, I still seem to get the "read-only filesystem" errors, and I still get dropped to a root shell.

I run mountall --debug towards the end of the log, in case that helps.

[Scott James Remnant (Canonical) (canonical-scott)]

On Thu, 2009-10-08 at 15:14 +0000, Dave Martin wrote:

> > > By default, mountall throws a lot of "Read-only filesystem" errors
> > > while trying to clean /tmp, and drops to a root shell.
> > >
> > > Alternatively, if I remount the root filesystem read-write before
> > > running init, init (or possibly mountall) appears to deadlock; no
> > > messages appear, and the platform does not boot.
> > >
> > I believe I have fixed this bug, new upload pending in the
> > PPA - again I would appreciate testing.
>
> OK... currently under way. I will take a short while since PPAs don't
> magically build for armel yet...
>
Still having issues, but I'll have those licked by tomorrow ;-)

Scott
--
Scott James Remnant
<email address hidden>

[Scott James Remnant (Canonical) (canonical-scott)]

Sorry about the issues with the previous PPA versions, as usual things worked just fine when I tested it in the various rigs I have here - of course it flatly failed when installed on normal systems because I hadn't actually tested that ;)

I've uploaded a new ~boot4 version, this one feels much better (and I'm running it on my laptop now :p)

As before, after installing the package but *before* you reboot, please run with --debug and attach the log to the bug - then after rebooting, let me know how it works out.

Thanks for all your help with testing, this is a big change and it's good to know that it's now working for 95% of people and your help getting it work for the final 5% is greatly appreciated!

[Dave Martin (dave-martin-arm)]

Sure, this is a definite step forward; some teething problems are inevitable.

Output from mountall (~boot4) --debug before reboot on armel attached.

Note that mountall doesn't appear to terminate in this case... is that the expected behaviour?

I'll post further results after I reboot.

[Dave Martin (dave-martin-arm)]

Hi... rebooted again on armel:

Now, boot does resume after the mountall-shell exits *if* the shell exits with status zero.

However, I still have a couple of issues:

1) A "filesystem last mounted in future" error is still treated as serious enough to interrupt the boot process and require interaction via mountall-shell. My view is that this is not appopriate because this may well happen when Windows fiddles with the clock on a dual-boot system, or when there's some other RTC problem. The novice user may not have a good idea what to type in the maintenenace shell anyway. The problem will persist across boot until the system is sufficiently convinced to mount the fs read-write My suggestion would be that if the only apparent error with the root filesystem is that it was mounted in the future, boot should proceed, although a warning is appropriate. This appears to have been the Jaunty behaviour.

(My ulterior motive is that we still have a lot of RTC problems with armel development hardware so I hit the problem on virtually every boot anyway...)

2) If the mountall-shell exits with non-zero exit status, boot does not resume:

init: mountall-shell main process (749) terminated with status 1
<...hang...>

 This is unlikely to be appropriate: the default exit status of a shell is simply the exit status of the last command executed... which was not necessarily fsck. I believe fsck can exit with non-zero status for various non-fatal situations anyway? It's necessary to quit the shell with "exit 0" to work around this... again, the novice user will have no clue about that.

[Ernst Zlo (ernst-zlo)]

Hello Scott,
I'm one of the 5%.

VirtualBox shared folder is to blame
(which I mount in my /etc/fstab with the line
extras /media/xtra vboxsf uid=1000,gid=1000,rw 0 0
stops the photon torpedos ;-)

When I remove the line everything's smooth and perfect.

I've included a tar.gz where you see:

mountall_before.log (after Install but before boot)
mountall.log (after removing the virtualbox drive and after booting where everything worked)
mountall+virtualbox.log (after inserting the virtualbox and mount -a)
Bildschirmfoto-1.png where you see the screen output during the first attempt where it hangs

If this is of any interest at this point:
When trying to reboot (after all above) it hangs with the white Ubuntu logo and no errormessage till the moment when (as I suppose) the scrensaver switches of the light and the VirtualBox window goes dark = the monitor is black.

Yep this time I was so clever to make a snapshot so I'm almost eagerly awaiting beta5

[Scott James Remnant (Canonical) (canonical-scott)]

On Fri, 2009-10-09 at 13:45 +0000, Dave Martin wrote:

> However, I still have a couple of issues:
>
> 1) A "filesystem last mounted in future" error is still treated as
> serious enough to interrupt the boot process and require interaction via
> mountall-shell. My view is that this is not appopriate because this may
> well happen when Windows fiddles with the clock on a dual-boot system,
> or when there's some other RTC problem.
>
The filesystem upstream author disagrees unfortunately; which is why
that error is there. Apparently the inconsistency causes problems for
journalled filesystems.

Now, you might say that this is one of those errors that fsck -a should
fix automatically - but this issue has become a bit politically charged
and the ext3/4 upstream is currently holding out having it a
boot-critical error.

At the same time, we have had bugs in the past with the system and
hardware clock in Ubuntu being inconsistent - so we've wanted the hard
error to collect the bug reports.

For the Windows case - the hardware clock is in localtime, and our
installer automatically sets that up when it detects the partition.

> The novice user may not have a
> good idea what to type in the maintenenace shell anyway. The problem
> will persist across boot until the system is sufficiently convinced to
> mount the fs read-write My suggestion would be that if the only
> apparent error with the root filesystem is that it was mounted in the
> future, boot should proceed, although a warning is appropriate. This
> appears to have been the Jaunty behaviour.
>
Indeed, but sadly for the wrong reasons.

You can get the jaunty behaviour by creating an /etc/e2fsck.conf file
with the contents:

 [options]
 buggy_init_scripts = 1

(see what I mean about political? <g>)

> 2) If the mountall-shell exits with non-zero exit status, boot does not
> resume:
>
> init: mountall-shell main process (749) terminated with status 1
> <...hang...>
>
> This is unlikely to be appropriate: the default exit status of a shell
> is simply the exit status of the last command executed... which was not
> necessarily fsck. I believe fsck can exit with non-zero status for
> various non-fatal situations anyway? It's necessary to quit the shell
> with "exit 0" to work around this... again, the novice user will have no
> clue about that.
>
Good catch, I've committed a fix for that.

Scott
--
Scott James Remnant
<email address hidden>

[Scott James Remnant (Canonical) (canonical-scott)]

On Fri, 2009-10-09 at 14:02 +0000, Ernst Zlo wrote:

> VirtualBox shared folder is to blame
> (which I mount in my /etc/fstab with the line
> extras /media/xtra vboxsf uid=1000,gid=1000,rw 0 0
> stops the photon torpedos ;-)
>
> When I remove the line everything's smooth and perfect.
>
I'd already fixed this one, and ~boot5 is uploading right now. Hope
that one works out for you!

Scott
--
Scott James Remnant
<email address hidden>

[Ernst Zlo (ernst-zlo)]

In the meantime I've upgraded to the new linux 2.6.31.13
And now I have the hanging without your repository as described above:

When trying to reboot it hangs with the white Ubuntu logo and no errormessage till the moment when (as I am sure now because it fades away) the scrensaver switches of the light and the VirtualBox window goes dark = the monitor is black.

Dead end.

[Dave Martin (dave-martin-arm)]

> > 1) A "filesystem last mounted in future" error is still treated as
> > serious enough to interrupt the boot process and require
> interaction
> > via mountall-shell. My view is that this is not appopriate because
> > this may well happen when Windows fiddles with the clock on a
> > dual-boot system, or when there's some other RTC problem.
> >
> The filesystem upstream author disagrees unfortunately; which
> is why that error is there. Apparently the inconsistency
> causes problems for journalled filesystems.
>
> Now, you might say that this is one of those errors that fsck
> -a should fix automatically - but this issue has become a bit
> politically charged and the ext3/4 upstream is currently
> holding out having it a boot-critical error.

OK... I don't know enough about the fs design to comment, but it sounds plausible.

Would it be possible simply to do a full filesystem check instead of the maintenance shell? I can foresee problems with portable devices where the console text console is not easily to access or use. If the full fs check fails, then dropping to a shell is more appropriate.

This will slow down boot in most circumstances when RTC problems occur, rather than exploding every time.

[...]

> For the Windows case - the hardware clock is in localtime,
> and our installer automatically sets that up when it detects
> the partition.

OK, fair enough. Otherwise, it seemed a good argument :)

>
> You can get the jaunty behaviour by creating an
> /etc/e2fsck.conf file with the contents:
>
> [options]
> buggy_init_scripts = 1
>

If there's a workaround that may be good enough for me. Is this documented somewhere? Otherwise, I'd have no clue to enable this option to work around a buggy RTC :)

[...]

> > for various non-fatal situations anyway? It's necessary to
> quit the
> > shell with "exit 0" to work around this... again, the
> novice user will
> > have no clue about that.
> >
> Good catch, I've committed a fix for that.

OK, thanks. I'll try it... is it in the PPA?

[Ernst Zlo (ernst-zlo)]

I went back to the point before upgrading to 2.6.31.13

I installed your repository
I did
sudo apt-get -f install
sudo apt-get --fix-missing install
sudo apt-get clean
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get clean
sudo apt-get autoremove

I checked if the VirtualBox drive is in /etc/fstab
groan I now call it VBd

Then I booted:
Same hang.

How can I go shure, that the new update to 2.6.31-13 is made before your ~beta5 is installed?

BTW I removed the VBd from /etc/fstab and it was o.k.
which is IMHO a slightly different scenario, because when upgrading the VirtualBoxGuestadditions are not active any more.
So I booted again with Guestadditions installed (and no VBd) -> o.k.

I activated the VBd and booted -> let me say it this way: I stared that long on the ubuntu logo that I can see it on this page as dark violett halo.
The good news for you: I can't bother you with screenshots because they are all black. ;-)

So I tried it with 2.6.31-12 generic (and your mountall, I did NOT go back this time) -> same freeze.

And by typing this essay of #@!* bugs it becomes clear to me: please give me ~beta6 because without ~beta5 nothing goes, with ~beta5 nothing goes either.

[Scott James Remnant (Canonical) (canonical-scott)]

On Fri, 2009-10-09 at 15:16 +0000, Ernst Zlo wrote:

> And by typing this essay of #@!* bugs it becomes clear to me: please
> give me ~beta6 because without ~beta5 nothing goes, with ~beta5 nothing
> goes either.
>
You haven't supplied your debug.log

Scott
--
Scott James Remnant
<email address hidden>

[Ernst Zlo (ernst-zlo)]

Sorry, Scott.

With or without your ~beta5: kernel 2.6.31-12 will hang with a VBd in the /etc/fstab.

Is the mountall.log of any interest for you, when the VBd is NOT mounted (which I am forced now, because otherwise my kernel won't work)?

And afterwards I only can supply you with a mountall.log when I have unmounted the VBd again?

[Ernst Zlo (ernst-zlo)]

After a walk with my dog my brain is either complete damaged or working better:

IMHO Ubuntu is trying now to start a VBd wich only can work if the VirtualBox Guestadditions are allready working, which are working AFTER all harddisks are mounted.

And you can't start the VirtualBox Guestadditions without having mounted the / partition.

I can't see a logical way out of this except mounting all harddrives except the VBd then start the VirtualBox Guestadditions and after they are up and running mount the VBd.

[Scott James Remnant (Canonical) (canonical-scott)]

On Fri, 2009-10-09 at 16:28 +0000, Ernst Zlo wrote:

> With or without your ~beta5: kernel 2.6.31-12 will hang with a VBd in
> the /etc/fstab.
>
> Is the mountall.log of any interest for you, when the VBd is NOT mounted
> (which I am forced now, because otherwise my kernel won't work)?
>
> And afterwards I only can supply you with a mountall.log when I have
> unmounted the VBd again?
>
I obviously need to see the log *with* the VBd in /etc/fstab of mountall
failing, as it's the failure I need to diagnose.

Scott
--
Scott James Remnant
<email address hidden>

[Ernst Zlo (ernst-zlo)]

Am Samstag, den 10.10.2009, 10:41 +0000 schrieb Scott James Remnant:
> On Fri, 2009-10-09 at 16:28 +0000, Ernst Zlo wrote:
>
> > With or without your ~beta5: kernel 2.6.31-12 will hang with a VBd in
> > the /etc/fstab.
> >
> > Is the mountall.log of any interest for you, when the VBd is NOT mounted
> > (which I am forced now, because otherwise my kernel won't work)?
> >
> > And afterwards I only can supply you with a mountall.log when I have
> > unmounted the VBd again?
> >
> I obviously need to see the log *with* the VBd in /etc/fstab of mountall
> failing, as it's the failure I need to diagnose.
>

And just this is impossible at the moment <sigh>
I'll go and get me a daily snapshot to see, if it's better with that
(this will last till monday, because I'm off now)

> Scott
> --
> Scott James Remnant
> <email address hidden>
>

[Ernst Zlo (ernst-zlo)]

Next round <g>
I went back to a version of Karmic where I could boot with the VBd.
I added your repository and manually made a
sudo apt-get install mountall

I made a ez_mt_before.log

I booted (hang)

I removed the VBd from fstab via life CD

I booted again. (no VBd mounted)
I did NOT make a mout -a this time but instantly
sudo mountall --debug > ez_mt_after.log 2>&1
and the VBd was mounted without a problem - of course, because the VirtualBox Guestadditions are running.

[Ernst Zlo (ernst-zlo)]

Scott,
I'm totally impressed, with the today's update the problem is solved.
As there was an upgrade to kernel (2.6.31-13.45) there would have been a Superblock issue. - and was not.
And even my VBd is mounted correctly.

Congratulations and thank you verrrrrry much.

I still miss a VWD button (Very Well Done) in the alpha and beta environements, so that you don't always only hear complaints ;-)

[Scott James Remnant (Canonical) (canonical-scott)]

Since we haven't heard off the original reporter in a little while, and his issue was fixed separately (the last mount time in future thing), and other comments have been positive, I'm going to mark this as fixed.

You do get a shell in case of bad fsck, but now exiting that shell retries everything and can continue the boot.

[Glenn Brumfield (gbrumfield)]

@Scott - Today's updates cured my boot problem (the last mount time in future), but I wanted to thank you while I was thinking of it. I also miss the nice "Thank You!" button; I firmly believe the old adage that one heart-felt "Well Done!" wipes out a hundred "Aw s*%#" 's."

Thank You!