[EMC Clarion] - Root filesystem goes "Read only" after installing multipath-tools on Lucid

[Joseph Salisbury]

Binary package hint: multipath-tools

Installing either multipath-tools-boot or multipath-tools causes the root partition to go 'Read only'. System is then unable to reboot. The only way to recover system is to boot in rescue mode and un-install multipath-tools.

The install of multipath-tools caused a segmentation fault. The following was done with the PPA at: http://ppa.launchpad.net/serge-hallyn/multipath-backport/ubuntu/. However, this same issue happens with the multipath-tools package that ships with Lucid.

pravin@ubuntu:~$ sudo apt-get install multipath-tools-boot
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  kpartx libaio1 multipath-tools
The following NEW packages will be installed:
  kpartx libaio1 multipath-tools multipath-tools-boot
0 upgraded, 4 newly installed, 0 to remove and 80 not upgraded.
Need to get 251kB of archives.
After this operation, 1,065kB of additional disk space will be used.
Do you want to continue [Y/n]? Y
Get:1 http://us.archive.ubuntu.com/ubuntu/ lucid/main libaio1
0.3.107-3ubuntu2 [9,608B]
Get:2 http://ppa.launchpad.net/serge-hallyn/multipath-backport/ubuntu/
lucid/main kpartx 0.4.8-14ubuntu7 [29.1kB]
Get:3 http://ppa.launchpad.net/serge-hallyn/multipath-backport/ubuntu/
lucid/main multipath-tools 0.4.8-14ubuntu7 [198kB]
Get:4 http://ppa.launchpad.net/serge-hallyn/multipath-backport/ubuntu/
lucid/main multipath-tools-boot 0.4.8-14ubuntu7 [13.8kB]
Fetched 251kB in 1s (135kB/s)
Preconfiguring packages ...
Selecting previously deselected package libaio1.
(Reading database ... 41797 files and directories currently installed.)
Unpacking libaio1 (from .../libaio1_0.3.107-3ubuntu2_amd64.deb) ...
Selecting previously deselected package kpartx.
Unpacking kpartx (from .../kpartx_0.4.8-14ubuntu7_amd64.deb) ...
Selecting previously deselected package multipath-tools.
Unpacking multipath-tools (from
.../multipath-tools_0.4.8-14ubuntu7_amd64.deb) ...
Selecting previously deselected package multipath-tools-boot.
Unpacking multipath-tools-boot (from
.../multipath-tools-boot_0.4.8-14ubuntu7_all.deb) ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
Setting up libaio1 (0.3.107-3ubuntu2) ...

Setting up kpartx (0.4.8-14ubuntu7) ...

Setting up multipath-tools (0.4.8-14ubuntu7) ...
 * Starting multipath daemon multipathd

                 [ OK ]

Setting up multipath-tools-boot (0.4.8-14ubuntu7) ...
debconf: DbDriver "templatedb": could not write
/var/cache/debconf/templates.dat-new: Read-only file system
dpkg: error processing multipath-tools-boot (--configure):
 subprocess installed post-installation script returned error exit status 1
touch: cannot touch `/var/lib/update-notifier/dpkg-run-stamp': Read-only
file system
sh: cannot create /var/lib/update-notifier/updates-available: Read-only file system
Segmentation fault

Hardware details:

OS: Lucid boots from boot LUN on SAN.
SAN:EMC Clariion
Hardware : UCS blade server from Cisco.

Attached is the multipath.conf file we are using.

I will attach the output from apport-bug shortly.
I will also attach the following:
$ uname -a > uname-a.log
$ cat /proc/version_signature > version.log
$ dmesg > dmesg.log
$ sudo lspci -vvnn > lspci-vvnn.log

Just let me know if any other data is required.

[Joseph Salisbury (jsalisbury)]

Logs from system that had this failure.

[Peter Petrakis (peter-petrakis)]

A few things, ask the customer to run the command 'multipath -v4'
as root after the package install failure so we can see what's going on.

There's at least one bug in the multipath.conf

1) scsi_id path is wrong
2) the blacklist regex is suspect

1)
Replace all calls to scsi id with:

"/lib/udev/scsi_id -g -u -d /dev/%n"

2) This regex assumes the sd names are ordered, this is never true.

Why are these sd devices being blacklisted to begin with?

blacklist {
       devnode "^sda[1-7]"
       devnode "^sd[f-z]"
       devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
       devnode "^hd[a-z]"
}

SD driver pre-allocates up to 15 partitions per whole block device,
the first regex only filters out the 1/2 of the possible partitions of
the first probed device.

If I'm to believe this blacklist then we're looking to serve
sda8-15, sdb, c,d,e and that's it.

I don't think this blacklist is viable, try commenting out the first two
devnodes.

Please try each change independently and as a whole, and report.

[Joseph Salisbury (jsalisbury)]

@Peter

Thanks for the suggestion. I will report back.

[Joseph Salisbury (jsalisbury)]

@Peter, we made all the suggested changes to the multipath.conf file, but still have the same issue. The root filesystem goes read-only as soon as multipathd starts. For the sake if time, the OS will now be installed on an internal drive instead of boot from SAN. I'll post the results from this test.

I also obtained the output of multipath -v4, which is attached.

[Joseph Salisbury (jsalisbury)]

Output from multipath -v4

[Peter Petrakis (peter-petrakis)]

From examining the output and can see why the blacklist was
defined in this manner, and how to correct it.

Also, since the luns are reporting themselves as DGC, specifying
a configuration for SYMMETRIX just won't work. Clarion and
Symmetrix are completely different animals from what I gather.

Attached is a new multipath.conf (rename it correctly before
moving to etc) that contains what I believe to be the configuration
they're trying to achieve. Unfortunately, this is array is so configurable,
all this effort could be a waste if it's not setup correctly.

Should this config not work, try deleting the 'devices' section
entirely and reboot. Multipath carries a configuration for DGC
already that might work.

# echo 'show config' | multipathd -k
...
        device {
                vendor DGC
                product .*
                product_blacklist LUNZ
                path_grouping_policy group_by_prio
                path_checker emc_clariion
                features 1 queue_if_no_path
                hardware_handler 1 emc
                prio_callout /sbin/mpath_prio_emc /dev/%n
                failback immediate
                no_path_retry 60
        }

Failing that I would suggest that the customer make the array work with
powerpath first, so we know the HW is configured correctly, and then
we can try again with multipath.

[Peter Petrakis (peter-petrakis)]

@Joe

What's the status of this? Is the issue resolved? Thanks.

[Joseph Salisbury (jsalisbury)]

@Peter

The decision was made to install Ubuntu on an internal disk and not the SAN. The issue was still happening at that time with 10.04 and 10.10. However, the system is not in productions, so it cannot be reconfigured.

[Peter Petrakis (peter-petrakis)]

Moving visibility to private since this is an resolved customer support issue.
We can't fix it if they're not willing to work with us. That probably means
either remote access or on site troubleshooting.

Peter