upstart/mountall hangs with LVM on RAID setup

[Christian Schürer-Waldheim]

Binary package hint: mdadm

After the recent update of upstart, my software RAID (LVM on RAID 1) is not started at boot anymore, thus the system hangs (at "Starting sad crypto disks").

A RAID setup is no unusual thing IMHO, so this problem is really severe.

[Christian Schürer-Waldheim (quincunx)]

I've added upstart as this problem is caused by the recent changes of upstart

[Scott James Remnant (Canonical) (canonical-scott)]

Could you provide:

  1) your /etc/fstab

  2) output of "blkid"

  3) output of "mdadm -D" for each RAID device (md0, md1) and partition (md0p1, etc.)

I suspect you're using a partitioned RAID?

[Christian Schürer-Waldheim (quincunx)]

N.B.: I booted with Kubuntu on a USB device, thus there are some extra partitions shown.

1) fstab:

# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# /dev/mapper/apus-root
UUID=b0a1fad6-f5bd-4d6d-acfd-90e1e1f84f20 / ext3 relatime,user_xattr,errors=remount-ro 0 1
# /dev/mapper/apus-boot
UUID=17528add-be7e-4916-bde2-4f4d2dc0b527 /boot ext3 relatime 0 2
# /dev/mapper/apus-swap
UUID=d463ce6f-c911-453a-b52e-f567bf1ed8e5 none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0
/dev/apus/home /home ext3 rw,acl 1 2

#usbfs
none /proc/bus/usb usbfs devgid=46,devmode=664 0 0

2) output of "blkid"

/dev/loop0: TYPE="squashfs"
/dev/loop1: UUID="849c5300-02d8-46ec-93ec-f8895c39cffc" TYPE="ext3"
/dev/sda1: UUID="0512da77-48b7-84d1-9e27-894509bcea2b" TYPE="linux_raid_member"
/dev/sdb1: UUID="0512da77-48b7-84d1-9e27-894509bcea2b" TYPE="linux_raid_member"
/dev/sdg1: LABEL="usbinst" UUID="48FF-856C" TYPE="vfat"
/dev/md0: UUID="rQFRKf-TrVb-NVgL-Rnj3-07eA-dtWi-i8Qs3d" TYPE="LVM2_member"
/dev/mapper/apus-boot: LABEL="boot" UUID="17528add-be7e-4916-bde2-4f4d2dc0b527" SEC_TYPE="ext2" TYPE="ext3"
/dev/mapper/apus-root: LABEL="apus-root" UUID="b0a1fad6-f5bd-4d6d-acfd-90e1e1f84f20" SEC_TYPE="ext2" TYPE="ext3"
/dev/mapper/apus-swap: UUID="d463ce6f-c911-453a-b52e-f567bf1ed8e5" TYPE="swap"
/dev/mapper/apus-home: LABEL="home" UUID="725e5266-ef6a-40a7-8a30-95735b7eac2c" SEC_TYPE="ext2" TYPE="ext3"
/dev/mapper/apus-datastore: UUID="0894b646-cf5b-45f1-bed3-7d5960631af0" TYPE="crypto_LUKS"

3) output of "mdadm -D /dev/md0"

/dev/md0:
        Version : 00.90
  Creation Time : Wed Jan 7 19:53:09 2009
     Raid Level : raid1
     Array Size : 488383936 (465.76 GiB 500.11 GB)
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Sep 17 22:01:33 2009
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 0512da77:48b784d1:9e278945:09bcea2b
         Events : 0.234135

    Number Major Minor RaidDevice State
       0 8 1 0 active sync /dev/sda1
       1 8 17 1 active sync /dev/sdb1

There are no partitions on the RAID but there is LVM setup on top of RAID. LVM provides several partitions: /dev/mapper/apus-* (as shown above)

[Christian Schürer-Waldheim (quincunx)]

Currently, when I start my computer, the boot process stops printing:

apus-root: clean, .../... files, .../... blocks
boot: clean, .../... files, .../... blocks

As you can see in the output of blkid (see my previous comment), apus-root and boot are the labels of two partitions on this computer. Shortly before the swap partition is being checked ("kinit: trying to resume from /dev/mapper/apus-swap"). This is evidence that the RAID and LVM are setup already. They partitions are just not mounted - so I've added mountall to this bug.

As far as I could see, mdadm just provides udev rules so that a raid is configured as soon as md-devices are found. And LVM is set up obviously too.

Why does mountall not mount the volumes on the LVM?

[Christian Schürer-Waldheim (quincunx)]

I've captured a log file of mountall (by adding "bash" to mountall.conf). There are no error messages, only a lot of debug messages.

When I get to the shell, the root (/) file system is already mounted. I can mount any other partition just by typing "mount boot" and so on. So everything is ready (MD + LVM). Don't know how to continue the boot process when I've mounted everything manually.

After I've issued the mountall command with debug arguments (taken from comment #7 of bug #430684), I don't come back to the shell, I even cannot kill the process with ctlr+c. Only a reboot with ctrl+alt+del is possible.

[Christian Schürer-Waldheim (quincunx)]

Shouldn't be caused by mdadm itself.

[Christian Schürer-Waldheim (quincunx)]

I've added "--debug" in mountall.conf (and removed bash). The log output shows that the partitions are mounted. The last lines are:

udev_monitor_watcher: block /dev/sdf (null) (null) (this is a card reader)
udev_monitor_watcher: block /dev/sdg1 48FF-856C usbinst (this is my usb stick with recovery system)

Then the boot stops and nothing happens.

[Scott James Remnant (Canonical) (canonical-scott)]

Are you still seeing the "sad" message?

[Christian Schürer-Waldheim (quincunx)]

No, because I've purged the cryptsetup package.

But still (all available updates applied today) the system just stops booting when mountall is issued. Please see my last posts with further details (e.g. mountall debug log).

Is there anything I can test/debug?

[Scott James Remnant (Canonical) (canonical-scott)]

On Mon, 2009-09-21 at 22:13 +0000, Christian Schürer-Waldheim wrote:

> No, because I've purged the cryptsetup package.
>
Then we'll be unable to continue debugging this :-(

 status invalid

If you're having other problems, please open a new bug.

Scott
--
Scott James Remnant
<email address hidden>

[Christian Schürer-Waldheim (quincunx)]

Sorry Scott, but I have to disagree. That I've purged cryptsetup is not the point here. But I fully understand that you can't overview all these bug reports opened after the recent change in the boot system. Cryptsetup was only installed to manually mount a luks partition, which is not one of the important system partitions.

The good news is that I could finally fix this problem this morning :-)

The cause for the problem and the solution are trivial. All my partitions are on an LVM volume. The device names are for instance /dev/mapper/apus-root. I configured this scheme with the alternative installation CD. Later, using system-config-lvm, I set up a new partition on the LVM for the /home area. This program adds a line to fstab too - and it reference the device /dev/apus/home (as you can see above in my fstab). It worked this way until mountall was introduced. I'm not sure if it is a valid name, though. So please have a look into this case, if either my fstab was not consistent with the standard of if mountall is missing a possible configuration.

Changing /dev/apus/home to the UUID solved my problem. I'll set this report to "status confirmed".

[Scott James Remnant (Canonical) (canonical-scott)]

On Tue, 2009-09-22 at 05:04 +0000, Christian Schürer-Waldheim wrote:

> The cause for the problem and the solution are trivial. All my
> partitions are on an LVM volume. The device names are for instance
> /dev/mapper/apus-root. I configured this scheme with the alternative
> installation CD. Later, using system-config-lvm, I set up a new
> partition on the LVM for the /home area. This program adds a line to
> fstab too - and it reference the device /dev/apus/home (as you can see
> above in my fstab).
>
Ah, great - thanks for the debugging

 duplicate 430542

Scott
--
Scott James Remnant
<email address hidden>