disk device naming is unpredictable on multipath systems

[Ryan Beisner]

When deploying ceph via charm to a system with multipath storage, the block device names and order may not be predictable. This essentially makes automated deployment of ceph via the charm onto these systems impossible.

Take:

/dev/sda
/dev/sdb
/dev/sdc
/dev/sdd
/dev/sde
/dev/sdf

Those devices are not usable by ceph (errors out, indicating they are in use).

The corresponding /dev/dm-N devices would be usable, I believe, if they were predictable. But they are not. ie /dev/dm-3 might represent /dev/sdc on one deployment, but after a tear down and redeploy, /dev/dm-3 might represent the disk that contains the root linux filesystem (doh!).

ex.

### First Deploy:
root@gregory-ppc64:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 264.3G 0 disk
├─sda1 8:1 0 8M 0 part
├─sda2 8:2 0 264.3G 0 part
└─mpath0 (dm-0) 252:0 0 264.3G 0 mpath
  ├─mpath0-part1 (dm-2) 252:2 0 8M 0 part
  └─mpath0-part2 (dm-3) 252:3 0 264.3G 0 part /
sdb 8:16 0 264.3G 0 disk
└─mpath2 (dm-1) 252:1 0 264.3G 0 mpath
sdc 8:32 0 264.3G 0 disk
└─mpath3 (dm-5) 252:5 0 264.3G 0 mpath
sdd 8:48 0 264.3G 0 disk
└─mpath4 (dm-4) 252:4 0 264.3G 0 mpath
sde 8:64 0 264.3G 0 disk
└─mpath5 (dm-6) 252:6 0 264.3G 0 mpath
sdf 8:80 0 264.3G 0 disk
└─mpath1 (dm-7) 252:7 0 264.3G 0 mpath
sdg 8:96 0 264.3G 0 disk
├─sdg1 8:97 0 8M 0 part
├─sdg2 8:98 0 264.3G 0 part
└─mpath0 (dm-0) 252:0 0 264.3G 0 mpath
  ├─mpath0-part1 (dm-2) 252:2 0 8M 0 part
  └─mpath0-part2 (dm-3) 252:3 0 264.3G 0 part /
sdh 8:112 0 264.3G 0 disk
└─mpath2 (dm-1) 252:1 0 264.3G 0 mpath
sdi 8:128 0 264.3G 0 disk
└─mpath3 (dm-5) 252:5 0 264.3G 0 mpath
sdj 8:144 0 264.3G 0 disk
└─mpath4 (dm-4) 252:4 0 264.3G 0 mpath
sdk 8:160 0 264.3G 0 disk
└─mpath5 (dm-6) 252:6 0 264.3G 0 mpath
sdl 8:176 0 264.3G 0 disk
└─mpath1 (dm-7) 252:7 0 264.3G 0 mpath
sr0 11:0 1 521.9M 0 rom

...and...

### Second Deploy (same metal, no config changes):
ubuntu@gregory-ppc64:~$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 14.5G 0 disk
└─sda1 8:1 1 1.3G 0 part
sdb 8:16 0 264.3G 0 disk
├─sdb1 8:17 0 8M 0 part
├─sdb2 8:18 0 264.3G 0 part
└─mpath0 (dm-2) 252:2 0 264.3G 0 mpath
  ├─mpath0-part1 (dm-4) 252:4 0 8M 0 part
  └─mpath0-part2 (dm-5) 252:5 0 264.3G 0 part /
sdc 8:32 0 264.3G 0 disk
└─mpath1 (dm-0) 252:0 0 264.3G 0 mpath
sdd 8:48 0 264.3G 0 disk
└─mpath2 (dm-1) 252:1 0 264.3G 0 mpath
sde 8:64 0 264.3G 0 disk
└─mpath3 (dm-3) 252:3 0 264.3G 0 mpath
sdf 8:80 0 264.3G 0 disk
└─mpath4 (dm-6) 252:6 0 264.3G 0 mpath
sdg 8:96 0 264.3G 0 disk
└─mpath5 (dm-7) 252:7 0 264.3G 0 mpath
sdh 8:112 0 264.3G 0 disk
├─sdh1 8:113 0 8M 0 part
├─sdh2 8:114 0 264.3G 0 part
└─mpath0 (dm-2) 252:2 0 264.3G 0 mpath
  ├─mpath0-part1 (dm-4) 252:4 0 8M 0 part
  └─mpath0-part2 (dm-5) 252:5 0 264.3G 0 part /
sdi 8:128 0 264.3G 0 disk
└─mpath1 (dm-0) 252:0 0 264.3G 0 mpath
sdj 8:144 0 264.3G 0 disk
└─mpath2 (dm-1) 252:1 0 264.3G 0 mpath
sdk 8:160 0 264.3G 0 disk
└─mpath3 (dm-3) 252:3 0 264.3G 0 mpath
sdl 8:176 0 264.3G 0 disk
└─mpath4 (dm-6) 252:6 0 264.3G 0 mpath
sdm 8:192 0 264.3G 0 disk
└─mpath5 (dm-7) 252:7 0 264.3G 0 mpath
sr0 11:0 1 521.9M 0 rom

[Chris Holcombe (xfactor973)]

It should be possible to pin these down with an /etc/multipath.conf file.
Something like:

defaults {
        user_friendly_names yes
        path_grouping_policy multibus
}

It's been quite a while since I've written one of these so it might not be right. I do seem to recall friendly_names being important though :)

[Billy Olsen (billy-olsen)]

Ceph charm has been deprecated so not worth fixing there. Moving over to the charm-ceph-osd project as this has everything to do with what the ceph-osd charm does and nothing to do with what the ceph-mon charm does.

[nikhil kshirsagar (nkshirsagar)]

From what I understand after going through online documentation, if user_friendly_names is set to yes,instead of the wwid, multipath uses names like mpatha, mpathb, etc. Otherwise it would use the wwid, like /dev/mapper/3600508b250136ef71100f002100a0000

These device names would be consistent after reboot, so if we detect a multipath setup, the charm can definitely use the /dev/mapper/XXX names and they would be consistent across reboots, even though the underlying dm-X and sdX devices may not be named the same after the reboot. What storage is referred to by mpath1, for eg, would still be referred to by that device named mpath1 after reboots. This is because multipath uses the wwids and bindings files for this purpose to provide persistent naming.

[Luciano Lo Giudice (lmlogiudice)]

As an alternative solution, upon setting the device path (i.e: /dev/sda), the charm could map it to its ID by looking into /dev/disk/by-id and seeing what it corresponds to. If there's a match, then the charm would use that path thereon instead. There's the issue, however, of what would happen if the device is _not_ found in /dev/disk/by-id (This can happen with some exotic device types, or when uuids aren't refreshed properly).