multipath-tools-boot, root mount failed - Device or resource busy

[Ron]

SRU Justification:
1. Impact: servers with multipath-tools-boot installed cannot mount root on boot
2. How bug addressed: The initramfs local-top script for multipath-tools waits for udev to handle all device renames resulting from multipath starting up before continuing on to mountroot.
3. Patch: is attached as a debdiff
4. To reproduce: install a server with multipath storage controller, install multipath-tools-boot, and reboot.
5. Regression potential: the fix is localized to an initramfs script shipped with multipath-tools-boot, so if there are regressions they should only affect multipath users, who cannot boot without some fix anyway.

Binary package hint: multipath-tools-boot

After installing "multipath-tools-boot", initramfs fails to mount /root by uuid with error: "failed: Device or resource busy". And boot process falls into (initramfs) prompt.

**Boot/Screen Output:**
Begin: Running /scripts/local-premount ... done.
mount: mounting /dev/disk/by-uuid/082f71b9-47dc-4d4b-bdfa-d5ca61f1e045 on /root
failed: Device or resource busy
Begin: Running /scripts/local-bottom ... done.
done.
<....> other failure messages mounting (dev, sys, proc, etc.) <....>
(initramfs)

**Further Debugging & Info:**
Procedures & commands, run shortly after (initramfs) prompt appears.
(initramfs) ls -l /dev/disk/by-uuid/
lrwxrwxrwx 1 10 082f71b9-47dc-4d4b-bdfa-d5ca61f1e045 -> ../../dm1
[ This appears correct!?! ]

(initramfs) mount /dev/disk/by-uuid/082f71b9-47dc-4d4b-bdfa-d5ca61f1e045 /root
[ SUCCESS ! ]

**WORKAROUND:**
I have created an initramfs script to slow-down the root mounting attempt:

File: /usr/share/initramfs-tools/scripts/local-top/waitfor-multipath
Content:
#!/bin/sh
VERBOSITY=0
PREREQ="multipath"
prereqs() { echo "$PREREQ"; }
case $1 in
prereqs)
 prereqs
 exit 0
 ;;
esac

echo "Waiting 5 seconds for multipaths & udev?? to settle down..."
sleep 5
exit 0
<----END---->

My guess for a general/global bug-fix is multipath-tools-boot script to include a slight delay/hold-down timer; to allow outside device-mapping procedures to complete. ..or a confirmation that dm devices have finished mapping.

**Observations:**
It appears multipath-tools-boot initramfs script executes successfully VERY quickly, and initramfs scripts continue with attempting to mount the root file system. Although, dm-# /dev/disk/by-uuid mappings are not instantly ready. This is why manually typing the same mount command at prompt is immediately successful.

_To duplicate error in a non-boot scenario:_
This exact same behavior can be observed & duplicated on running servers; when mounting a multipath (SAN) volume. As example; mounting multipath-SAN targets will create: /dev/sda, /dev/sdb, /dev/sdc, etc.... And multipath-tools+devmapper will then create the shared/linked /dev/dm-# mappings. You are then only supposed to use the /dev/dm-# virtual devices to access multipathed SAN devices (multipaths.)

.... If you then attempt to mount a direct SAN volume /dev/sd?# an error is raised: "mount: /dev/sd?# already mounted or /<mount_point> busy"

**Software & Versions:**
 OS: Ubuntu 10.10 (Maverick - server)
 Related Packages & versions:
  multipath-tools 0.4.8-14ubuntu4
  multipath-tools-boot 0.4.8-14ubuntu4
  udev 162-2.1
  initramfs-tools 0.98.1ubuntu6
  initramfs-tools-bin 0.98.1ubuntu6
  open-iscsi 2.0.871-0ubuntu5
  open-iscsi-utils 2.0.871-0ubuntu5
  libdevmapper1.02.1 2:1.02.39-1ubuntu6

**Hardware Details:**
 Servers affected: HP Proliant DL385 (G5, G5p, G6) (and others I'm sure.)
 Storage/SAN Type: iSCSI
 Storage/SAN Hardware: HP StorageWorks MSA2312i (and others I'm sure.)

 This problem affects servers with iSCSI-boot NICs, AND servers which have a USB-Key with /boot partition.

Please let me know if more information is necessary!

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug and helping to make Ubuntu better.

There is a package in ppa:serge-hallyn/initramfs-multipath2 which we've
been using to work around that problem. I'd been pushing a patch to
the Debian package, but it sounds like requesting a merge temporarily
into the Ubuntu packages would be worthwhile as well.

Can you try the package in the ppa above and let me know if you still
have an issue? Meanwhile I'll request a merge into the natty sources
for starters. (Source tree is at
https://code.launchpad.net/~serge-hallyn/ubuntu/lucid/initramfs-tools/udevsettle )

[Serge Hallyn (serge-hallyn)]

I'm hitting a bit of a roadblock in how to handle this process-wise. Natty needs a whole new package sync. That sync will have these problems fixed. Lucid and maverick however need these fixes. But the natty package sync is not yet done.

[Serge Hallyn (serge-hallyn)]

The fix for this is now in natty. I will request SRUs next week for lucid and maverick.

[Serge Hallyn (serge-hallyn)]

This debdiff is for the SRU of the fix which is in natty, to lucid and maverick.

[Craig Magina (craig.magina)]

I can verify that this fixes the issue.

ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 32 Jan 18 14:21 06eacbef-a3c2-4888-b9e8-faef800781cc -> ../../mapper/222dc0001550218ffp1
lrwxrwxrwx 1 root root 36 Jan 18 14:21 54c316e3-d875-42ae-88d1-451a9876f91e -> ../../mapper/2229a000155fe296e-part1
lrwxrwxrwx 1 root root 32 Jan 18 14:21 9acd1309-5946-48c5-8c9e-e56944494f0b -> ../../mapper/222710001553137dep1
lrwxrwxrwx 1 root root 36 Jan 18 14:21 BC2C-8741 -> ../../mapper/222eb0001559a8c3f-part1
lrwxrwxrwx 1 root root 36 Jan 18 14:21 ca8a7948-d069-4227-8e1d-31d22c5703fa -> ../../mapper/222ca000155636d68-part1

No "failed: Device or resource busy" found in the logs after booting with the fixed package.

[Clint Byrum (clint-fewbar)]

Accepted multipath-tools into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Clint Byrum (clint-fewbar)]

The upload to lucid-proposed was deleted, as the version was higher than that of the dev release, oneiric. If you happened to download that version while it was briefly available, you should manually downgrade back to 0.4.8-14ubuntu4, otherwise you run the risk of not updating this package on an upgrade to maverick.

[Serge Hallyn (serge-hallyn)]

Let's wait until the packages currently in *-proposed are moved to -updates, then I'll create a new set of packages for this.

WIthout this, we have systems which will simply not boot, so it definately remains high priority. Maybe ought to be moved to critical.

[JC Hulce (soaringsky)]

This bug affects Ubuntu 10.10, Maverick Meerkat. Maverick has reached end-of-life and is no longer supported, so I am closing the bugtask for Maverick. Please upgrade to a newer version of Ubuntu.
More information here: https://lists.ubuntu.com/archives/ubuntu-announce/2012-April/000158.html

[Rolf Leggewie (r0lf)]

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".