multipath ignores blacklist in multipath.conf

[gheeke]

Multipath in Ubuntu 12.04 ignores blacklist settings in multipath.conf

In multipath.conf the following blacklist is configured:

blacklist {
        wwid 36782bcb070070100166a7a1640a66db2
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z][[0-9]*]"
        devnode "^vd[a-z]"
        devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
        device {
                vendor "DELL*"
                product "PERC*"
        }
        device {
                vendor "*"
                product "Universal Xport"
        }
        device {
                vendor "*"
                product "MD3000"
        }
        device {
                vendor "*"
                product "MD3000i"
        }
        device {
                vendor "*"
                product "Virtual Disk"
        }
}

Still multipath -ll lists the internal SAS HDD (via Dell Perc 6/i Controller):

...
36782bcb070070100166a7a1640a66db2 dm-0 DELL,PERC 6/i
size=136G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 2:2:0:0 sda 8:0 active ready running
...

[Steve (steve-nonnel)]

Same problem as above

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[gdahlman (gdahlman)]

I think blacklist device maching is expanded as a regex and not as a glob, thus the * will not provide a match. Can you try the following change.

Replace:

 device {
                vendor "DELL*"
                product "PERC*"
        }

With:

       device {
               vendor DELL.*
               product PERC.*
       }

Also if you could paste the output from

$ sudo udisks --dump

It would help debug this issue, I am not on the Ubuntu team but I am trying to track down an issue that may be related to this.

[gheeke (gerhard-heeke)]

At the moment i've no Ubuntu 12.04 Testmachine at hand. ASAP I'll try your suggestions. But I don't think, that this is the problem. My Dell Controller has the following wwid:

36782bcb070070100166a7a1640a66db2 dm-0 DELL,PERC 6/i
size=136G features='0' hwhandler='0' wp=rw

In the Blacklist that wwid is explicitly listed:

blacklist {
        wwid 36782bcb070070100166a7a1640a66db2
...

Although the Controller is still listed.

In the Red Hat Documenation (http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/device_type_blacklist.html) they use also * as wildcard for multiple matches:

blacklist {
       device {
               vendor "IBM"
               product "3S42" #DS4200 Product 10
       }
       device {
               vendor "HP"
               product "*"
       }
}

[Peter Petrakis (peter-petrakis)]

multipath supports POSIX regexp: libmultipath/regex.c so you do need
the "." to inform the wildcard what it's matching against e.g. (1+ more characters).

There are also examples in the multipath.conf.defaults file that provide examples
of regexp like:

# device {
# vendor "HP"
# product "LOGICAL VOLUME.*"
# getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"

and

# device {
# vendor "DGC"
# product ".*"
# product_blacklist "LUNZ"

If you want to wildcard an entire field, leaving out the dot works, but when you
start matching within strings, you need to specify what it is you're matching.

#blacklist {
# devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
# devnode "^hd[a-z]"
# devnode "^dcssblk[0-9]*"
#}

You could try a dramatically simplified multipath.conf like this:

blacklist {
  device {
        vendor "*"
        product "*"
  }
}

blacklist_exceptions {
  device {
    vendor "NETAPP"
    product "LUN"
  }
}

The continue configuring your SAN under a new devices{ device {} } section.
In my case I just take the defaults. to prove to yourself that this works,
stop multipath server, flush all paths (multipath -F), and then change
the config file to include only the blacklist (exclude everything), run
multipath -v4 and observe.

===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod
3600508b1001cb2fce7d75a0d5bee1ea1 3:0:0:1 sda 8:0 1 undef faulty HP,LOGICA
360a98000534b504d6834654d53793373 2:0:0:0 sdb 8:16 4 undef ready NETAPP,LU
360a98000534b504d6834654d53793373 5:0:0:0 sdc 8:32 4 undef ready NETAPP,LU
360a98000534b504d6834654d53793373 2:0:1:0 sdd 8:48 1 undef ready NETAPP,LU
360a98000534b504d6834654d53793373 5:0:1:0 sde 8:64 1 undef ready NETAPP,LU
Jul 09 15:16:07 | sda: (HP:LOGICAL VOLUME) vendor/product blacklisted
Jul 09 15:16:07 | sdb: (NETAPP:LUN) vendor/product blacklisted
Jul 09 15:16:07 | sdc: (NETAPP:LUN) vendor/product blacklisted
Jul 09 15:16:07 | sdd: (NETAPP:LUN) vendor/product blacklisted
Jul 09 15:16:07 | sde: (NETAPP:LUN) vendor/product blacklisted

root@nashira:~# lsscsi
[0:0:0:0] cd/dvd hp DVD A DS8A5LH 1HE3 /dev/sr0
[2:0:0:0] disk NETAPP LUN 8020 /dev/sdb
[2:0:1:0] disk NETAPP LUN 8020 /dev/sdd
[3:0:0:0] storage HP P410i 3.66 -
[3:0:0:1] disk HP LOGICAL VOLUME 3.66 /dev/sda
[5:0:0:0] disk NETAPP LUN 8020 /dev/sdc
[5:0:1:0] disk NETAPP LUN 8020 /dev/sde

So it indeed blacklisted everything. and multipath -ll produces nothing.
Once I add my SAN to the blacklist exception block everything functions
as expected.

create: 360a98000534b504d6834654d53793373 undef NETAPP,LUN
size=16G features='1 queue_if_no_path' hwhandler='0' wp=undef
|-+- policy='round-robin 0' prio=4 status=undef
| |- 2:0:0:0 sdb 8:16 undef ready running
| `- 5:0:0:0 sdc 8:32 undef ready running
`-+- policy='round-robin 0' prio=1 status=...

[Peter Petrakis (peter-petrakis)]

@gheeke

Did you try the alternate config file style I proposed in comment #5?
Thanks.

[Dmitry Galenko (dmitry-galenko)]

Only first line with devnode work.

Working config:
devnode_blacklist {
## Ubuntu BUG #927540
devnode "^((ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*|(hd|xvd|vd|sd)[a-z]*)"
wwid "*"
}

## Use user friendly names, instead of using WWIDs as names.
defaults {
user_friendly_names yes
path_grouping_policy multibus
failback immediate
}

blacklist_exceptions {
wwid "1"
wwid "2"
wwid "3"
wwid "4"
wwid "..."
wwid "n"
}

[Dmitry Galenko (dmitry-galenko)]

Sorry, this config not working too.

[Peter Petrakis (peter-petrakis)]

That devnode line is actually part of the default in memory config,
see libmultipath/blacklist.c#setup_default_blist. So try dropping
that.

Just try and keep you multpath.conf as simple as possible,
to even *one* directive, test and record, then iterate until you start to see it
break. It's best to stop multipath-tools altogether and just "multipath -F &&
multipath -v4" until you find the breakage. Record the logs the entire time
using redirection or 'script' and attach your results to this issue. Thank you.

[Lukasz Szulik (bliter)]

This is not a bug of multipathd.
I gues that you are using DELL MD3xxx multipath installer. This install script provide new
initrd that have one significant change - add multipath.conf (the same as they add in standard /etc) in initrd environment.

So - at the initrd stage when lvm starts (with default lvm.conf @initrd) they links local filesystem lvm to multipath
device. When chroot phase is in time your config defined in root /etc/multipath.conf can't remove PERC
multipath dev because they are already in use by root fs.

Solution - please swap initramfs (@ /boot/..) to the original (it's now rename) and at the initrd stage no multipath
devices of PERC will be created. You multipath.conf in rootfs /etc/multipath.conf are OK).

regards LS

[Peter Petrakis (peter-petrakis)]

That's actually a good point Lukasz, one that I assume folks get by reading the
multipath guide first.

https://help.ubuntu.com/12.04/serverguide/multipath-devices.html

Whenever you use blacklists, in either lvm or multipath, you must update
the initramfs so those config files are copied to the ramdisk, otherwise you
can get a race that makes it look like your blacklist is having no effect.

# update-initramfs -u -k all

should do it.

[Peter Petrakis (peter-petrakis)]

Considering the time past and that I'm convinced that this is a user configuration
issue (Support) that should be addressed by comment #11, I'm closing this issue as Invalid.