Boot stalls on mountall "disk drive not ready" question in multipath-tools >= 0.4.9-3ubuntu7.5
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
multipath-tools (Ubuntu) |
Triaged
|
High
|
Mathieu Trudel-Lapierre |
Bug Description
System information: Cisco UCS B200M2 blade, fnic.ko HBA. The system boots from local storage, but mounts the following file system on an EMC VNX during bootup:
opt_vnx (3600601603a713
size=50G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| |- 1:0:1:0 sdd 8:48 active ready running
| `- 0:0:0:0 sda 8:0 active ready running
`-+- policy='round-robin 0' prio=0 status=enabled
|- 0:0:1:0 sdb 8:16 active ready running
`- 1:0:0:0 sdc 8:32 active ready running
/etc/fstab contains:
/dev/mapper/opt_vnx /opt/vnx ext4 noatime 0 2
After upgrading the "multipath-tools" package to version 0.4.9-3ubuntu7.5 or higher, the system can no longer boot without manual intervention. Instead, the following question is asked (by mountall(8)) on the console:
The disk drive for /opt/vnx is not ready yet or not present.
keys:Continue to wait, or Press S to skip mounting or M for manual recovery
Waiting does nothing useful. Pressing "S" allows the boot to run to completion, and the "opt_vnx" *is* present when logging in to a completely booted system. However, it seems that this is discovered *after* the mountall(8) question appears and is skipped, as this log line appears later on in the boot process:
* Discovering and coalescing multipaths... [ OK ]
Downgrading multipath-tools to version 0.4.9-3ubuntu7.4 does resolve the problem, and allows the system to boot normally without manual intervention. Note that the dependent "kpartx" package does *not* need to be downgraded also, if this is left on version 0.4.9-3ubuntu7.9 it will still boot fine.
We experience this problem on serveral other systems apart from the one described in this bug report.
Changed in multipath-tools (Ubuntu): | |
assignee: | nobody → Mathieu Trudel-Lapierre (mathieu-tl) |
After reviewing the changes between -3ubuntu7.4 and -3ubuntu7.5, I have found that the problem is caused by the removal of the file /lib/udev/ rules.d/ 95-multipath. rules. In -3ubuntu7.4, it contained the following:
#
# udev rules for multipathing.
# The persistent symlinks are created with the kpartx rules
#
# socket for uevents /org/kernel/ dm/multipath_ event"
RUN+="socket:
# Coalesce multipath devices before multipathd is running (initramfs, early ="add|change" , SUBSYSTEM=="block", RUN+="/ sbin/multipath -v0 /dev/$name"
# boot)
ACTION=
If this file is manually reinstated (either to /lib/udev/rules.d or /etc/udev/rules.d), boot is again successful (even with the latest -3ubuntu7.9 multipath-tools package).
I see from the changelog that the removal was intentional:
* debian/rules: don't ship 95-multipath.rules udev rules anymore; they are multipath. udev: removed.
not necessary with multipath-tools listening for udev events directly.
* debian/
However, in order for this to actually work with multipath-backed "auto" filesystems in /etc/fstab, it seems necessary to ensure that multipathd is started before mountall runs. I am not entirely certain how to accomplish this, though, as mountall is started from Upstart while multipath-tools is stuck using a legacy SysV-init, and I do not know if it is possible to enforce service ordering between the two init systems. For what it's worth, renaming /etc/rcS. d/S21-multipath -tools- boot as etc/rcS. d/S00-multipath -tools- boot does not work around the issue.