race condition leaves raid raw devices exposed

[Tormod Volden]

Binary package hint: dmraid

The -Z option in dmraid-activate is supposed the hide the raw devices when the raid is activated. This works sometimes, sometimes not. My mirror raid is on /dev/sda and /dev/sdb, yet today I see the sdbX devices. Yesterday I saw the sdaX devices as well, after one reboot none of them.

$ ls /dev/sd*
/dev/sda /dev/sdb1 /dev/sdb3 /dev/sdb5 /dev/sdb7 /dev/sdd /dev/sdf
/dev/sdb /dev/sdb2 /dev/sdb4 /dev/sdb6 /dev/sdc /dev/sde

If I run dmraid-activate manually, I see this message logged:
dmraid-activate: ERROR: Cannot retrieve RAID set information for isw_ececbiichd_osmoraid
which AFAICS means the dmraid -Z command is not run.

"dmraid -i -si isw_ececbiichd_osmoraid" returns nothing.

I suspect the occasional successful hiding happens in the initrd as devices are initially discovered, and will not work later.

ProblemType: Bug
Architecture: i386
DistroRelease: Ubuntu 9.04
Package: dmraid 1.0.0.rc15-6ubuntu1
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: dmraid
Uname: Linux 2.6.28-10-generic i686

[Tormod Volden (tormodvolden)]

Also with 1.0.0.rc15-6ubuntu2:

$ ls /dev/sd*
/dev/sda /dev/sda3 /dev/sda6 /dev/sdb1 /dev/sdb4 /dev/sdb7 /dev/sde
/dev/sda1 /dev/sda4 /dev/sda7 /dev/sdb2 /dev/sdb5 /dev/sdc /dev/sdf
/dev/sda2 /dev/sda5 /dev/sdb /dev/sdb3 /dev/sdb6 /dev/sdd

[Tormod Volden (tormodvolden)]

By adding some logging to dmraid-activate, I catched this:

ERROR: removing part 1 from /dev/sdb: Device or resource busy

In this case the /dev/sdbX devices were exposed.

[Tormod Volden (tormodvolden)]

I guess what happens is: sda is discovered, and "dmraid-activate sda" is called, which looks up the raid and finds that sda and sdb are part of it. It then starts the array with dmraid -Z which will try to hide both sdaX and sdbX. At the same time sdb is discovered, and dmraid actions here is blocking the -Z hiding in the first.

[Giuseppe Iuculano (giuseppe-iuculano)]

Confirmed bugs require confirmation from someone other than the original reporter :)

Can you please paste the output of cat /etc/fstab please?

[Tormod Volden (tormodvolden)]

I know, but I thought I had dug up enough information and proof by now :)

I am not at the machine right now, but I know that /etc/fstab only refers to /dev/mapper/isw_ devices.

[Tormod Volden (tormodvolden)]

Still happens on Karmic Alpha 6 live CD (on USB stick with patched initrd for bug #392510 and bug #385305. Some boots are fine, some not.

[Phillip Susi (psusi)]

Good catch. A possible workaround would be to patch dmraid to wait a second and retry a few times before giving up. I think a more proper long term solution is that dmraid should not be used to remove the partitions from the disk when activated, but rather the partitions should not be created in the first place. This means removing the partition table parsing from the kernel and letting udev handle it with partx, which should be smart enough to notice the disk is a raid member and skip processing it. This isn't likely to happen any time soon though.

[Tormod Volden (tormodvolden)]

For now, I have added this to /etc/rc.local:

if [ -e /dev/sda1 -o -e /dev/sdb1 ]; then
 logger "udev/dmraid did not hide raw devices"
 dmraid -ay -Z
fi