needs to load (or wait for) filesystem modules e.g. xfs

Bug #432620 reported by sbodo on 2009-09-18
50
This bug affects 7 people
Affects Status Importance Assigned to Milestone
mountall (Ubuntu)
Medium
Scott James Remnant (Canonical)
Karmic
Medium
Scott James Remnant (Canonical)

Bug Description

Binary package hint: mountall

booting latest karmic fails - even with mountall-0.1.6 - with the following:

mount: unknown filesystem type 'xfs'
mountall: mount /home [2055] terminated with status 32
mountall: Filesystem could not be mounted: /home
init: mountall main process (1601) terminated with status 3

Then it drops into a shell, where mount -a just works fine. After remounting / to rw and exiting the shell continues the boot process.
The fstab entry for /home is:

/dev/mapper/main-home /home xfs noatime,nodiratime 0 0

I have exactly the same problem, except with jfs. Also using latest Karmic. It's only happened once in about 30 boots, so I'm guessing it's a race condition somewhere.

zoolook (nbensa) wrote :

I get this every boot. Sometimes it can't mount /var, sometimes it can't mount /home.

Once I got it to boot, I see a strange thing with /var:

zoolook@venkman:~$ mount | grep var
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
/dev/mapper/venkman-var on /dev/.var type xfs (rw,relatime)

zoolook@venkman:~$ grep var /proc/mounts
none /var/lock tmpfs rw,nosuid,nodev,noexec,relatime 0 0
none /var/run tmpfs rw,nosuid,relatime,mode=755 0 0
/dev/mapper/venkman-var /var xfs rw,relatime,attr2,noquota 0 0
none /var/lock tmpfs rw,nosuid,nodev,noexec,relatime 0 0
none /var/run tmpfs rw,nosuid,relatime,mode=755 0 0

my fstab:

zoolook@venkman:~$ cat /etc/fstab
proc /proc proc defaults 0 0
/dev/mapper/venkman-root / reiserfs relatime 0 0
/dev/md0 /boot ext2 relatime 0 0
/dev/mapper/venkman-home /home xfs relatime 0 0
/dev/mapper/venkman-usr /usr reiserfs relatime 0 0
/dev/mapper/venkman-var /var xfs relatime 0 0
/dev/mapper/venkman-opt /opt reiserfs relatime 0 0
/dev/mapper/venkman-virt /usr/local/share/virt reiserfs relatime 0 0
/dev/mapper/venkman-music /usr/local/share/music xfs relatime 0 0
/dev/mapper/venkman-swap none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0

If I change RAMRUN and RAMLOCK to "no" in /etc/default/rcS, my machine boots every time.

Regards,
Norberto

sbodo (sbodomerle) wrote :

I have this on every boot also. Disabling RAMRUN and RAMLOCK does not help either.
It seems more of a race as pointed out, because sometimes i get the shell almost instantly, while other times it takes pretty long to get into the shell (more then a minute).

zoolook (nbensa) wrote :

Yes sbodo, changing RAMRUN and RAMLOCK doesn't always help. My box didn't want to boot today. It took me like 10 reboots. (BTW, grub2 also doesn't help at all. Why is it so sllloooooowww?)

Is there a way to disable event-driven services and go back to serial (one after another) service startup?

Santiago Otero (siryurian) wrote :

I had the same problem but with a xfs and a reiserfs partition. I think the problem is that mountall parses /proc/filesystem for the supported file systems but some file systems are implemented as modules (see bug #433672). I have modified /etc/mkinitramfs/modules with the entries for my file systems and updated initramfs with "sudo update-initramfs -k all -u" and now my system boots again.

Santiago Otero (siryurian) wrote :

The file for the initramfs modules is /etc/initramfs-tools/modules, not /etc/mkinitramfs/modules.

summary: - unknown filesystem type 'xfs'
+ needs to load (or wait for) filesystem modules e.g. xfs
Changed in mountall (Ubuntu):
importance: Undecided → Medium
status: New → Triaged

This one's a little strange, from what I can tell the kernel should auto-load the module itself

Changed in mountall (Ubuntu Karmic):
assignee: nobody → Scott James Remnant (scott)
milestone: none → ubuntu-9.10
status: Triaged → Incomplete
tags: added: ubuntu-boot
sbodo (sbodomerle) wrote :

It seems to be related to fs_passno being 0 instead of 2.
While all the fs_passno fileds were set to 0 in my fstab - latest karmic failed to boot.
Setting fs_passno on all xfs entries to 2 solved the problem ....

Huh, that's strange.

Setting the pass number just means that fsck will be run. I wonder whether there's some race condition going on here, and the extra delay introduced by fsck is enough to lose it

sbodo (sbodomerle) wrote :

Yes - you are right. But i had a succesfull boot right after changing the pass number.
Today boot was the same: unknown xfs filesystem, status 32 .... except that i can see the fsck.xfs started to all the volumes, before i saw only one instance fired (probably for the first volume)

darthanubis (darthanubis) wrote :

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# / was on /dev/sda4 during installation
UUID=67904705-b186-4d51-96e3-0dae4399346c / ext4 errors=remount-ro 0 1
# /boot was on /dev/sda2 during installation
UUID=ba9c30de-9d4e-4ff1-b50a-2b3c3c23ad2b /boot ext2 defaults 0 2
# /home was on /dev/mapper/wolfdale-home during installation
UUID=a5b5d14f-e2c2-49a4-b1d4-05b0309534e8 /home ext4 defaults 0 2
# swap was on /dev/sda6 during installation
UUID=9a1989f9-c0e2-4746-ba08-b0869d9d209a none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0
# /home/anubis/backup was on /dev/mapper/wolfdale-backup during installation
UUID=ba830f6e-f634-42fd-9fb3-61a9d5ca6d15 /home/anubis/backup ext4 defaults 0 2
# /home/anubis/mythrec was on /dev/mapper/wolfdale-mythtv during installation
UUID=90ede06f-40f4-40bf-9cb1-8e623e00ce4d /home/anubis/mythrec xfs defaults 0 2
# /home/anubis/storage was on /dev/mapper/wolfdale-storage during installation
UUID=e09b01df-35dd-4f86-b870-1c59dc6c2266 /home/anubis/storage xfs defaults 0 2
# /home/anubis/terabyte was on /dev/mapper/wolfdale-terabyte during installation
UUID=9d865078-8835-4d2d-9601-34b75b3c393f /home/anubis/terabyte xfs defaults 0 2

Confirmed same here with my XFS partitions. Mythrec and Storage do not automount, but Terabyte does. Weird?
http://ubuntuforums.org/showthread.php?p=8039064

Amedee Van Gasse (amedee) wrote :

Confirmed with the following fstab:

# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# / was on /dev/mapper/lvmvolume-root during installation
UUID=42ea9aa1-9ec5-45df-8e5f-b02e014906a8 / ext4 relatime,errors=remount-ro 0 1
# /boot was on /dev/sda1 during installation
UUID=6f608f1c-3e37-44b2-9f92-6695453a40ce /boot ext2 relatime 0 2
# /home was on /dev/mapper/lvmvolume-home during installation
UUID=3ca8d92c-c04a-433a-825d-57edc77f014f /home ext4 relatime 0 2
# swap was on /dev/mapper/lvmvolume-swap during installation
UUID=0b92ab6a-7dfe-40b6-b227-833c9f136204 none swap sw 0 0
LABEL=vm /media/vm xfs noatime 0 0
LABEL=backup /media/backup xfs noatime 0 0

/media/vm and /media/backup give a short error at boot (but continues), but when the OS is completely loaded, I can do ls /media/vm and see what's in there, so the filesystems are mounted.
Could it be that mountall is running twice during the boot process, and that it doesn't pick up xfs (or jfs, or any other filesystem that is not in kernel but in a module) the first time, but it does the second time?

It's just my pet theory, I don't know.
The only (minor) nuisance is that the error message prevents usplash from displaying properly.

description: updated
Changed in mountall (Ubuntu Karmic):
status: Incomplete → Confirmed
Dejan (dejan-rodiger) wrote :

Confirming bug in beta version of Karmic. I am using jfs on my /home. Santiago Otero's instructions helped me.

gator (waltons4) wrote :

confirmed on mythbuntu 9.10 beta i386.
XFS partitions fail to mount at boot.
sudo mount -a ; works as expected

I think that what's actually happening here is that we're running two mounts for XFS in parallel, the first one stops to load the module but the second one rushes ahead.

We don't need to do this really, in fact it's a bit error-prone, so changing to just running one mount at a time seems sane except for remote filesystems

Changed in mountall (Ubuntu Karmic):
status: Confirmed → Fix Committed

I've uploaded a new mountall package to the ubuntu-boot PPA:

https://launchpad.net/~ubuntu-boot/+archive/ppa

I would appreciate it if you could install this and try it out. *BEFORE* you reboot though, could you run "sudo mountall --debug > mountall.log 2>&1" and attach that to this bug - then after you reboot, let me know whether it worked or not.

Thanks

darthanubis (darthanubis) wrote :

I'll reboot in an hour. I'll keep you guys posted.

BTW. Thank you so much for the quick work on this issue!

darthanubis (darthanubis) wrote :

Well, sadly, after reboot, the / filesystem would only mount read only. The system is now sitting at terminal until I can come up with a way to mount the filesystem, alter my source.list file (which does not show the PPA I added to use your mountall.deb), and reinstall the good, but broken mountall, so that I can get back to the desktop. I only have a alternative disk. I think I'll download the regular beta disk, chroot in and see if I can get this mountall off the system.

Yeah, Yeah I know it's still Beta and I should ....etc.

darthanubis (darthanubis) wrote :

Well, I got the Beta iso. I put the ISO on a USB stick, and booted, but all I got was a black screen with the spinning mouse pointer. The desktop never came up after waiting for 5+ minutes. The Alphas were better than this Beta. Fail all the way around. This is shaping up to be the most buggy release for me yet.

On Thu, 2009-10-08 at 11:41 +0000, darthanubis wrote:

> Well, sadly, after reboot, the / filesystem would only mount read only.
> The system is now sitting at terminal until I can come up with a way to
> mount the filesystem, alter my source.list file (which does not show the
> PPA I added to use your mountall.deb), and reinstall the good, but
> broken mountall, so that I can get back to the desktop. I only have a
> alternative disk. I think I'll download the regular beta disk, chroot in
> and see if I can get this mountall off the system.
>
> Yeah, Yeah I know it's still Beta and I should ....etc.
>
I've fixed this and it should be available in the PPA soon, I'd
appreciate testing.

Scott
--
Scott James Remnant
<email address hidden>

James Crow (the-crowbar) wrote :

I was experiencing the problems described in this bug.

I installed mountall 0.2.0 from the ppa and restarted my system. The system would not start. The last message I saw on the screen was regarding stopping NTP. It would just sit there indefinitely. I could Ctrl-Alt-Del and the system would shutdown. I had to use the beta cd to boot and downgrade back to mountall 0.1.8.

I can only do limited testing because this system is heading to a different location.

I had a total of four hard drives.
sda1 - / - ext4
sad5 - /home - ext4
sda6 - swap
sda7 - /var - ext4
sdb1 /myth1 - xfs
sdc1 /myth2 - xfs
sdd1 /myth3 - xfs

I may have time to build another test box in the next day or so, but this update failed to fix my problem.

Thanks,
James

darthanubis (darthanubis) wrote :

Ok, I wasted my time with the BETA disk. Recovered with the Alpha6 alt. disk. Now will try the latest mountall from the PPA. Here goes nothing:-P.

darthanubis (darthanubis) wrote :

This version of mountall from the PPA leave me stuck at the "warning running a fsck on a mounted system could fsck your system". If I chose 'Y' it just sits there and who knows what it is doing. If I answer 'N', I get nothing as well.

After using rescue disk to downgrade mountall, twice each time after reboot, all partitions mount perfectly. If I reboot a second time after recovery, the same behaviour of not mounting all xfs filesystems returns.

Can we get more help on this besides Scott and I?

sbodo (sbodomerle) wrote :

mountall from ppa get stucked after displaying the fsck messages.
I could recover by booting with init=/bin/bash,
mount the filesystems, then exec /sbin/init, and then go back to version 0.1.8.

Sorry about the issues with the previous PPA versions, as usual things worked just fine when I tested it in the various rigs I have here - of course it flatly failed when installed on normal systems because I hadn't actually tested that ;)

I've uploaded a new ~boot4 version, this one feels much better (and I'm running it on my laptop now :p)

As before, after installing the package but *before* you reboot, please run with --debug and attach the log to the bug - then after rebooting, let me know how it works out.

Thanks for all your help with testing, this is a big change and it's good to know that it's now working for 95% of people and your help getting it work for the final 5% is greatly appreciated!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mountall - 0.2.0

---------------
mountall (0.2.0) karmic; urgency=low

  [ Colin Watson ]
  * Always check the root filesystem if --force-fsck is used, regardless of
    passno. LP: #435707.

  [ Johan Kiviniemi ]
  * Have each fsck instance create a lock for each underlying physical device.
    If you have a single disk or RAID, all filesystem checks will happen
    sequentially in order to avoid thrashing. On more complex configurations,
    you’ll benefit from the parallel checks mountall has been doing all along.
    LP: #434974.

  [ Scott James Remnant ]
  * Flush standard output and error before spawning processes to make
    capturing logs easier (otherwise we end up repeating things still in
    the buffer), and before calling exec().
  * Turn the code upside down so that each mount knows what it's waiting
    for, and allow multiple dependencies. This makes the code much more
    readable putting the "policy" in a single function, and will make it
    much easier in future when this is done by Upstart.
  * For kernel filesystems listed in fstab, honour the order that they
    are listed in fstab. LP: #432571, #433537, #436796
  * Always create new swap partition mounts for each fstab entry, don't
    treat them as updating the same. LP: #435027.
  * Virtual filesystems under local or remote filesystems (and local under
    remote) don't delay the virtual or local events. LP: #431040.
  * Simplify event emission, this has the advantage that we can now output
    what mount points we're waiting for and what they are waiting for as
    well.
  * Fixed issue with trailing slashes. LP: #443035.
  * Only run hooks if the filesystem was not already mounted. LP: #444252.
  * Don't clean up /tmp when run without --tmptime argument.
  * Ignore loop and ram devices until ready. LP: #441454.
  * Add options to binfmt_misc filesystem, which will probably cause it to
    be mounted on boot as well.
  * Synchronously mount local and virtual filesystems, I suspect this is
    the real cause behind the XFS races as one will modprobe and the other
    will not (and fail). LP: #432620.
  * Synchronously activate swap to avoid out of memory issues when checking
    the root filesystem.
  * Enumerate existing udev devices on startup, so we don't always have to
    see udev be coldplugged.
  * Don't break on general errors for non-essential filesystems.
    LP: #441144.
  * Don't repeat attempts to mount a filesystem without having first
    succeded to mount another.
  * Still restart mountall even if the recovery shell fails.
  * Don't queue filesystem check when device is "none", or missing, or the
    filesystem is marked nodev.
  * Generate a "mount" event before mounting a filesystem, and wait for its
    effects to complete.

 -- Scott James Remnant <email address hidden> Fri, 09 Oct 2009 16:50:46 +0100

Changed in mountall (Ubuntu Karmic):
status: Fix Committed → Fix Released
darthanubis (darthanubis) wrote :

YEAH!

Added the PPA installed mountallv5. Took the debug report. Rebooted, and saw no text all graphical and gdm was up EXTREMELY fast! I never seen my PC boot this quick. All the partitions were mounted when the desktop came up. I want to reboot just to see that happen again.

Thank you guys so much!

m4lte (malte-koester) wrote :

I experience the same problem in the official 9.10 release. On boot I get the following message:
My /home folder is on a separate partition. All partitions are ext3 formatted. I don't know where the dazuko-fs comes from.

mount: unknown filesystem type 'dazukofs'
mountall: mount /home [962] terminated with status 32
mountall: Filesystem could not be mounted: /home
mountall: mount /home [965] terminated with status 32
mountall: Filesystem could not be mounted: /home
init: mountall main process (513) terminated with status 4
Mount of root filesystem failed.
A maintenance shell will now be started.

Steve Langasek (vorlon) wrote :

Check your /etc/fstab. You probably have two entries for /home, one with this incorrect "dazukofs" and one with ext3. The old boot scripts would process both lines and ignore the failure of the second line; the new scripts consider the filesystem not mounted and wait indefinitely.

rwreed (randywreed) wrote :

I still get the problem with my xfs partition (lvm)
fstab says
UUID=133e4e3d-fa4d-4e70-afda-f2b460401e89 /home/randy/MyDocuments xfs defaults 1 2

on boot /home/randy/MyDocuments is empty

if I do sudo mount /dev/system/MyDocument /home/randy/MyDocuments

partition mounts with no problems

fstab is unchanged from 9.04, not problems then

rwreed (randywreed) wrote :

A bit more information. Using Kubuntu. Mountall is v1.0

m4lte (malte-koester) wrote :

@Steve Langasek #29
Thanks! I removed that line and now everything works fine again.

Jan Petersen (jpeterse77) wrote :

Hi All,

I have the same error 32 problem on my root file system (well all mount points really, since most are dependent on the root file system for mount points).
using final 9.10 release with MountAll v. 1.0, system is an upgrade from Jaunty.
I've tried all tips listed on various blog posts and forums, including changing UUID's to /dev/sd* entries, and disabling fsck in fstab, but nothing helps.
The root file system is loaded read-only at boot, and then the process halts with the error 32 messages.
I can force the boot to disregard the errors that it spews out by remounting the root system rw from recovery shell and then type exit. It's ugly but it allow me to enter the system. Albeit with only the root system mounted.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers