can't fsck / from friendly-recovery

[ski]

Fresh install of jaunty/x86 using one big ext4 partition.

If you select the fsck option, the following message (very) briefly appears at the bottom of the screen:
  mount: / is busy
... and no filesystem is checked.

Similarly, if you start a root shell and try to fsck by hand, you're stymied quite early on:
# mount -o ro,remount /
mount: / is busy.

The only way i can find that works is to tune2fs -c 28 -C 28 /dev/sda1 and reboot.

This is not exactly "friendly" ...

[ski (skibrianski)]

PS - by jaunty, i mean jaunty RC1 with all updates from the net applied as of Sat Apr 18 08:16:32 GMT 2009

[Brian Murray (brian-murray)]

Confirmed using friendly-recovery version 0.2.8.1 on Karmic.

[Brian Murray (brian-murray)]

This worked fine for me on a Jaunty system with the same version of friendly-recover however the filesystem was ext3. I wonder if that has something to do with it.

[ldm (ldmaidens)]

When recovering from a lockup or when system crashes, if you select the recovery option and then select fsck the screen blanks for a second and then returns to the original recovery screen. This also happens any time you select the this combination. Thus, there is no current way other than the live cd to do a fsck on the disk. I am running with ext4 and a encrypted home partition and amd64 beta version of karmic.

[ski (skibrianski)]

As a sanity check, I've reconfirmed the problem on a jaunty/ext4/amd64 system. When I select 'fsck' from the recovery menu, I am told: "mount: / is busy" (at the bottom of the screen, it goes away very quickly as the menu seems to refresh immediately after fsck errors out. When I select a root shell prompt and try "mount -o ro,remount /" I receive the same message: "mount: / is busy"

ski@portege:~$ dpkg -l | grep friendly-recovery
ii friendly-recovery 0.2.8.1 Make recovery more user-friendly

Attached is a log of what happens when I tinker a bit to try to umount other filesystems to see if that helps (short version: it doesn't).

... And here's some tune2fs -l output (from the system booted multi-user, forgot to include it in the log above)
ski@portege:~$ sudo tune2fs -l /dev/disk/by-label/root #multi-user boot
tune2fs 1.41.4 (27-Jan-2009)
Filesystem volume name: root
Last mounted on: <not available>
Filesystem UUID: b25363cd-d0c0-4d6b-ad32-efada5d29825
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 1313280
Block count: 5242880
Reserved block count: 262160
Free blocks: 1928070
Free inodes: 1153802
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 1022
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8208
Inode blocks per group: 513
Flex block group size: 16
Filesystem created: Sun Aug 16 18:01:52 2009
Last mount time: Wed Oct 14 20:45:34 2009
Last write time: Wed Oct 14 20:45:34 2009
Mount count: 20
Maximum mount count: 28
Last checked: Mon Oct 5 04:32:13 2009
Check interval: 15552000 (6 months)
Next check after: Sat Apr 3 04:32:13 2010
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
First orphan inode: 3865
Default directory hash: half_md4
Directory Hash Seed: 802ed7e2-cde7-4eef-8bdf-72922e387abc
Journal backup: inode blocks

[James Westby (james-w)]

Hi,

Could you please also provide the output of

sudo fuser -v -m /

when you get the "busy" message?

Thanks,

James

[Brian Murray (brian-murray)]

screenshot of fuser output

[Scott James Remnant (Canonical) (canonical-scott)]

On Thu, 2009-10-15 at 11:26 +0000, James Westby wrote:

> Could you please also provide the output of
>
> sudo fuser -v -m /
>
> when you get the "busy" message?
>
At a guess, init, udev, dbus, etc. All the usual system services.

Scott
--
Scott James Remnant
<email address hidden>

[emarkay (mrk)]

#6 - James, that message only appears for about 1/4 second!

[James Westby (james-w)]

On Thu Oct 15 18:02:09 UTC 2009 Scott James Remnant wrote:
> At a guess, init, udev, dbus, etc. All the usual system services.

So what changed that this no longer works? Brian confirmed that it
worked on Jaunty on IRC.

Thanks,

James

[emarkay (mrk)]

James (#10) - It does work in Jaunty, but NOTHING happens in Karmic, aside from the aforementioned short message.

[ski (skibrianski)]

James & emarkay - it does NOT work in jaunty, at least with ext4 and amd64 architecture, as in the initial bug report (which i reconfirmed yesterday).

[Scott James Remnant (Canonical) (canonical-scott)]

James: only luck makes it work on Jaunty, usually you only tend to go to "friendly recovery" when you have a problem - and that problem usually means the root filesystem is already read-only.

This needs to be reimplemented so that when any fsck fails, or any critical boot component, you get a friendly-recovery style helper - not just in a special GRUB menu that abuses "single user mode"

[ski (skibrianski)]

Scott - I disagree strongly that the root filesystem is usually already read-only when friendly-recovery is invoked. I don't just go to friendly recovery only when there is a problem (in fact I'm not sure I ever have in reaction to a problem). I admit I'm not a usual user, but there are plenty of cases where a cautious user might select to boot to friendly recovery because s/he wants to make sure there isn't a problem. That is to say, it's just as important to have single user mode work for preventative maintenance as for reactive maintenance. Having a friendly recovery option in grub's menu.lst instead of plain vanilla single user mode is a definite improvement. Adding the triggering of friendly-recovery to other places (failed fsck's, etc.) would be great, I just want to make sure we aren't considering losing the option of booting to single user mode.

[Scott James Remnant (Canonical) (canonical-scott)]

On Tue, 2009-10-20 at 08:40 +0000, ski wrote:

> I just want to make sure we aren't considering losing the option of
> booting to single user mode.
>
You seem to not understand what single user mode is.

In System V, the only services missing in single user mode compared to
multi-user mode are login services, such as getty, gdm, sshd, etc. It's
not intended as a "recovery" mode.

A modern Linux system needs running services to function even at a basic
level. This, and the misuse of "single user mode", is where the current
friendly recovery design falls over and fails.

For example, the recovery menu is on /usr. If /usr is on a network
filesystem mount, you need a large number of services: udev, dbus, hal,
network-manager, portmap, rpc.statd, etc.

These will all be running off the root filesystem, and thus prevent it
from being checked.

Scott
--
Scott James Remnant
<email address hidden>

[ski (skibrianski)]

> In System V, the only services missing in single user mode compared to
> multi-user mode are login services, such as getty, gdm, sshd, etc. It's
> not intended as a "recovery" mode.

Well, I'll admit that I come from a mostly BSD background. I had no idea that sysV used runlevel 1 like that, or that ubuntu wanted to be like sysV for the sake of being like sysV. AFAIK, It's been a long time since debian or ubuntu used runlevels 3-5 for anything, which leaves us with runlevel 2 = multiuser, runlevel 1 = single user / root shell / whatev.

To me, single user mode / init 1 / whatever is where you go to kick users off the machine and do maintenance, repairs, etc. In the BSDs (even darwin) I can boot to single user mode and fsck / without issue. I essentially just get a root shell, and that's how I likes it. I'm willing to accept that I'm atypical, but having a way to offer the user recovery options proactively, not just after something is hosed, is a useful innovation which should be preserved.

And I would point out that with the same type of brush that you declare most people wind up in friendly recovery when root is read only, I could much more easily declare that people with /usr on a network mount (or even any seperate mountpoint from /) are so few and far between that they are hardly worth considering.

But your point about having daemons running does have some truth to it, and to answer that, I don't see why the functionality of friendly recovery couldn't be whittled down to things that can be put in a single binary located on /, statically linked with ncurses and whatever else necessary to get these sorts of common tasks done.

I just think it's a very useful innovation to display a little ncurses menu with common maintenance options before dropping a potentially new user, unguided, at a root shell.

Cheers,
Brian

[Scott James Remnant (Canonical) (canonical-scott)]

On Tue, 2009-10-20 at 10:50 +0000, ski wrote:

> But your point about having daemons running does have some truth to it,
> and to answer that, I don't see why the functionality of friendly
> recovery couldn't be whittled down to things that can be put in a single
> binary located on /, statically linked with ncurses and whatever else
> necessary to get these sorts of common tasks done.
>
This was pretty much my point ;-)

You want a recovery mode to be a bit more interactive, perhaps
interceding before each mount to manually do the filesystem check and
attempt to mount - and then help you diagnose mount errors.

And then step through each of them, until those are down.

Then spool forwards (maybe offering an interactive "Start foo? y/n" type
approach) until it's ready to start the display server.

And obviously if the display server doesn't start, help to diagnose
that, offer a fallback, etc.

This should all be done in the normal boot too - if fsck fails, friendly
recovery should pop up, not a shell.

Scott
--
Scott James Remnant
<email address hidden>

[emarkay (mrk)]

I think we all missed the point. The point is - it DOES NOT WORK AT ALL.
All I see is a flash of "Mount is busy" and nothing more.
This is on 2 PC's here; it's RC time, now!

[ski (skibrianski)]

emarkay, true. But bear in mind, it's been broken with ext4 since jaunty. I'd love to see one sooner, but I'm guessing we won't see a fix until after karmic's release.

[emarkay (mrk)]

#19: Actually it was working in early betas of Karmic, and I had my Jaunty systmes converted to ext4 and they all worked; or at least "appeared" to work.

This is insane, IMHO; that something fundamental, a GRUB2 entry selection has a feature that is BROKEN, and is not being addressed BEFORE release!

[Michael Vogt (mvo)]

I don't think we can fix the fsck entry before release, there is dbus, udevd, dhclient, rsyslog running and preventing to mount the filesystem ro. For karmic we will have to remove the fsck entry.

[elrond (elrond.)]

I can't fsck / with ext3 filesystem on Karmic, summary should be changed

[Launchpad Janitor (janitor)]

This bug was fixed in the package friendly-recovery - 0.2.8.2

---------------
friendly-recovery (0.2.8.2) karmic; urgency=low

  * usr/share/recovery-mode/options/fsck:
    - Drop fsck option because we can not reliable remount the
      filesystem readonly for a fsck (LP: #363271). In karmic
      at least dbus, udevd, dhclient and rsyslog are running
      in single user mode.

 -- Michael Vogt <email address hidden> Fri, 23 Oct 2009 22:43:21 +0200

[elrond (elrond.)]

After update, in recovery mode selecting "dpkg" appears this message, in red:

mountall: Cancelled. General error mounting filesystem. A maintenance
shell will be now started. CONTROL-D will terminate this shell and re-try.
Press enter for maintenance (or type Control-D to continue):

but neither Enter nor Control-D produce effects. I have to restart system whit Alt + SysRq + R E I S U B

[ski (skibrianski)]

Two points:

1 - The bug in jaunty IS specific to ext4, and has existed since jaunty betas. Looks like in the end this bug has been "hijacked" by a related but more critical bug, so I'm not complaining just want to make sure we don't overlook that forever.
2 - I'd like to echo what others have said about not having any way to fsck root w/o either a) boot from a different media or b) doing a normal boot and praying for the best, is a very bad thing.

[Michael Vogt (mvo)]

Lucid has re-added fsck now in the recovery menu we should consider backporting it.

[Brett Randall (javabrett)]

+1 for backport to karmic.

[ski (skibrianski)]

I agree a backport is needed here.

[Tim O'Brien (timob)]

I used initctl to see services running.
Stopped them using stop.
Then mount -o remount,ro /
On karmic