disk device naming is unpredictable on multipath systems
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ceph Monitor Charm |
Invalid
|
Low
|
Unassigned | ||
Ceph OSD Charm |
Triaged
|
Low
|
nikhil kshirsagar | ||
OpenStack Ceph Charm (Retired) |
Won't Fix
|
Low
|
Unassigned | ||
ceph (Juju Charms Collection) |
Invalid
|
Low
|
Unassigned | ||
ceph-mon (Juju Charms Collection) |
Invalid
|
Low
|
Unassigned |
Bug Description
When deploying ceph via charm to a system with multipath storage, the block device names and order may not be predictable. This essentially makes automated deployment of ceph via the charm onto these systems impossible.
Take:
/dev/sda
/dev/sdb
/dev/sdc
/dev/sdd
/dev/sde
/dev/sdf
Those devices are not usable by ceph (errors out, indicating they are in use).
The corresponding /dev/dm-N devices would be usable, I believe, if they were predictable. But they are not. ie /dev/dm-3 might represent /dev/sdc on one deployment, but after a tear down and redeploy, /dev/dm-3 might represent the disk that contains the root linux filesystem (doh!).
ex.
### First Deploy:
root@gregory-
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 264.3G 0 disk
├─sda1 8:1 0 8M 0 part
├─sda2 8:2 0 264.3G 0 part
└─mpath0 (dm-0) 252:0 0 264.3G 0 mpath
├─mpath0-part1 (dm-2) 252:2 0 8M 0 part
└─mpath0-part2 (dm-3) 252:3 0 264.3G 0 part /
sdb 8:16 0 264.3G 0 disk
└─mpath2 (dm-1) 252:1 0 264.3G 0 mpath
sdc 8:32 0 264.3G 0 disk
└─mpath3 (dm-5) 252:5 0 264.3G 0 mpath
sdd 8:48 0 264.3G 0 disk
└─mpath4 (dm-4) 252:4 0 264.3G 0 mpath
sde 8:64 0 264.3G 0 disk
└─mpath5 (dm-6) 252:6 0 264.3G 0 mpath
sdf 8:80 0 264.3G 0 disk
└─mpath1 (dm-7) 252:7 0 264.3G 0 mpath
sdg 8:96 0 264.3G 0 disk
├─sdg1 8:97 0 8M 0 part
├─sdg2 8:98 0 264.3G 0 part
└─mpath0 (dm-0) 252:0 0 264.3G 0 mpath
├─mpath0-part1 (dm-2) 252:2 0 8M 0 part
└─mpath0-part2 (dm-3) 252:3 0 264.3G 0 part /
sdh 8:112 0 264.3G 0 disk
└─mpath2 (dm-1) 252:1 0 264.3G 0 mpath
sdi 8:128 0 264.3G 0 disk
└─mpath3 (dm-5) 252:5 0 264.3G 0 mpath
sdj 8:144 0 264.3G 0 disk
└─mpath4 (dm-4) 252:4 0 264.3G 0 mpath
sdk 8:160 0 264.3G 0 disk
└─mpath5 (dm-6) 252:6 0 264.3G 0 mpath
sdl 8:176 0 264.3G 0 disk
└─mpath1 (dm-7) 252:7 0 264.3G 0 mpath
sr0 11:0 1 521.9M 0 rom
...and...
### Second Deploy (same metal, no config changes):
ubuntu@
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 14.5G 0 disk
└─sda1 8:1 1 1.3G 0 part
sdb 8:16 0 264.3G 0 disk
├─sdb1 8:17 0 8M 0 part
├─sdb2 8:18 0 264.3G 0 part
└─mpath0 (dm-2) 252:2 0 264.3G 0 mpath
├─mpath0-part1 (dm-4) 252:4 0 8M 0 part
└─mpath0-part2 (dm-5) 252:5 0 264.3G 0 part /
sdc 8:32 0 264.3G 0 disk
└─mpath1 (dm-0) 252:0 0 264.3G 0 mpath
sdd 8:48 0 264.3G 0 disk
└─mpath2 (dm-1) 252:1 0 264.3G 0 mpath
sde 8:64 0 264.3G 0 disk
└─mpath3 (dm-3) 252:3 0 264.3G 0 mpath
sdf 8:80 0 264.3G 0 disk
└─mpath4 (dm-6) 252:6 0 264.3G 0 mpath
sdg 8:96 0 264.3G 0 disk
└─mpath5 (dm-7) 252:7 0 264.3G 0 mpath
sdh 8:112 0 264.3G 0 disk
├─sdh1 8:113 0 8M 0 part
├─sdh2 8:114 0 264.3G 0 part
└─mpath0 (dm-2) 252:2 0 264.3G 0 mpath
├─mpath0-part1 (dm-4) 252:4 0 8M 0 part
└─mpath0-part2 (dm-5) 252:5 0 264.3G 0 part /
sdi 8:128 0 264.3G 0 disk
└─mpath1 (dm-0) 252:0 0 264.3G 0 mpath
sdj 8:144 0 264.3G 0 disk
└─mpath2 (dm-1) 252:1 0 264.3G 0 mpath
sdk 8:160 0 264.3G 0 disk
└─mpath3 (dm-3) 252:3 0 264.3G 0 mpath
sdl 8:176 0 264.3G 0 disk
└─mpath4 (dm-6) 252:6 0 264.3G 0 mpath
sdm 8:192 0 264.3G 0 disk
└─mpath5 (dm-7) 252:7 0 264.3G 0 mpath
sr0 11:0 1 521.9M 0 rom
description: | updated |
Changed in ceph (Juju Charms Collection): | |
status: | New → Triaged |
Changed in ceph-mon (Juju Charms Collection): | |
status: | New → Triaged |
Changed in ceph (Juju Charms Collection): | |
importance: | Undecided → Low |
Changed in ceph-mon (Juju Charms Collection): | |
importance: | Undecided → Low |
Changed in charm-ceph: | |
importance: | Undecided → Low |
status: | New → Triaged |
Changed in ceph (Juju Charms Collection): | |
status: | Triaged → Invalid |
Changed in charm-ceph-mon: | |
importance: | Undecided → Low |
status: | New → Triaged |
Changed in ceph-mon (Juju Charms Collection): | |
status: | Triaged → Invalid |
It should be possible to pin these down with an /etc/multipath.conf file.
Something like:
defaults {
user_friendly_ names yes
path_grouping_ policy multibus
}
It's been quite a while since I've written one of these so it might not be right. I do seem to recall friendly_names being important though :)