Comment 0 for bug 585027

Binary package hint: multipath-tools

Environment:

njones@fantail:~$ lsb_release -rd
Description: Ubuntu 10.04 LTS
Release: 10.04
njones@fantail:~$ apt-cache policy multipath-tools
multipath-tools:
  Installed: 0.4.8-14ubuntu4
  Candidate: 0.4.8-14ubuntu4
  Version table:
 *** 0.4.8-14ubuntu4 0
        500 http://nz.archive.ubuntu.com/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status

Issue:

While reproducing https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/571093 both myself and Dustin Kirkland encountered an apparent race condition in multipath and dmsetup.

We were attempting variants of 'while true; do sudo multipath -F; sudo multipath -v4 ; done' to create a set of udev add/remove events and noticed that shortly after starting, no output appeared within the 'udevadm monitor' command.

We observed that the output from a: multipath -v4 was similar to:

root@fantail:~# multipath -v4
Discover device /sys/block/ram0
ram0: device node name blacklisted
Discover device /sys/block/ram1
ram1: device node name blacklisted
Discover device /sys/block/ram2
ram2: device node name blacklisted
Discover device /sys/block/ram3
ram3: device node name blacklisted
Discover device /sys/block/ram4
ram4: device node name blacklisted
Discover device /sys/block/ram5
ram5: device node name blacklisted
Discover device /sys/block/ram6
ram6: device node name blacklisted
Discover device /sys/block/ram7
ram7: device node name blacklisted
Discover device /sys/block/ram8
ram8: device node name blacklisted
Discover device /sys/block/ram9
ram9: device node name blacklisted
Discover device /sys/block/ram10
ram10: device node name blacklisted
Discover device /sys/block/ram11
ram11: device node name blacklisted
Discover device /sys/block/ram12
ram12: device node name blacklisted
Discover device /sys/block/ram13
ram13: device node name blacklisted
Discover device /sys/block/ram14
ram14: device node name blacklisted
Discover device /sys/block/ram15
ram15: device node name blacklisted
Discover device /sys/block/loop0
loop0: device node name blacklisted
Discover device /sys/block/loop1
loop1: device node name blacklisted
Discover device /sys/block/loop2
loop2: device node name blacklisted
Discover device /sys/block/loop3
loop3: device node name blacklisted
Discover device /sys/block/loop4
loop4: device node name blacklisted
Discover device /sys/block/loop5
loop5: device node name blacklisted
Discover device /sys/block/loop6
loop6: device node name blacklisted
Discover device /sys/block/loop7
loop7: device node name blacklisted
Discover device /sys/block/sda
sda: not found in pathvec
sda: mask = 0x1f
sda: dev_t = 8:0
sda: size = 625142448
sda: subsystem = scsi
sda: vendor = ATA
sda: product = WDC WD3200AAKS-0
sda: rev = 12.0
sda: h:b:t:l = 0:0:0:0
sda: serial = WD-WCAS70000070
sda: getprio = NULL (internal default)
sda: prio = 1
sda: getuid = /lib/udev/scsi_id -g -u -d /dev/%n (config file default)
sda: uid = 1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070 (callout)
Discover device /sys/block/sr0
sr0: device node name blacklisted
Discover device /sys/block/sdb
sdb: not found in pathvec
sdb: mask = 0x1f
sdb: dev_t = 8:16
sdb: size = 15679488
sdb: subsystem = scsi
sdb: vendor = Kingston
sdb: product = DataTraveler G2
sdb: rev = PMAP
sdb: h:b:t:l = 6:0:0:0
sdb: serial =
sdb: getprio = NULL (internal default)
sdb: prio = 1
sdb: getuid = /lib/udev/scsi_id -g -u -d /dev/%n (config file default)
error calling out /lib/udev/scsi_id -g -u -d /dev/sdb
Discover device /sys/block/sdc
sdc: not found in pathvec
sdc: mask = 0x1f
sdc: dev_t = 8:32
sdc: size = 41943040
sdc: subsystem = scsi
sdc: vendor = IET
sdc: product = VIRTUAL-DISK
sdc: rev = 0
sdc: h:b:t:l = 7:0:0:0
sdc: serial =
sdc: getprio = NULL (internal default)
sdc: prio = 1
sdc: getuid = /lib/udev/scsi_id -g -u -d /dev/%n (config file default)
error calling out /lib/udev/scsi_id -g -u -d /dev/sdc
Discover device /sys/block/dm-0
dm-0: device node name blacklisted
Discovery status 0
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070 0:0:0:0 sda 8:0 1 [undef][undef
                                           6:0:0:0 sdb 8:16 1 [undef][undef
                                           7:0:0:0 sdc 8:32 1 [undef][undef
sda: ownership set to 1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070
sda: not found in pathvec
sda: mask = 0xc
sda: path checker = directio (config file default)
directio: called for 800
directio: called in synchronous mode
directio: starting new request
directio: async io getevents returns 1 (errno=No such file or directory)
directio: io finished 4096/0
sda: state = 2
sda: prio = 1
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: pgfailover = -1 (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: pgpolicy = failover (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: selector = round-robin 0 (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: features = 0 (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: hwhandler = 0 (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: rr_weight = 1 (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: minio = 1000 (config file default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: no_path_retry = NONE (internal default)
pg_timeout = NONE (internal default)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: set ACT_CREATE (map does not exist)
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: map already present
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: domap (0) failure for create/reload map
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070: remove multipath map
sda: orphaned

And a device was left behind in /dev/mapper:

njones@fantail:~$ ll /dev/mapper/
total 0
drwxr-xr-x 2 root root 80 2010-05-25 04:25 ./
drwxr-xr-x 18 root root 4080 2010-05-25 04:25 ../
brw-rw---- 1 root disk 252, 0 2010-05-25 04:25 1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070
crw-rw---- 1 root root 10, 59 2010-05-22 10:35 control

$ sudo dmsetup ls
1ATA_WDC_WD3200AAKS-00TMA0_WD-WCAS70000070 (252, 0)

We had to perform 'dmsetup remove <id>' to clear the 'map already present' message.

While I admit that this is very a very specific case, it is still a race condition and does warrant a bug report.