Juju storage-get is reporting the incorrect device name for a volume mount with the OpenStack provider.
This has been seen on stable, candidate and edge. It is intermittent but consistent enough that it effects OpenStack charm engeineering CI, requiring retries unnecessarily.
Steps to reproduce:
Model snippet:
ceph-osd:
charm: ceph-osd
num_units: 3
storage:
osd-devices: cinder,10G
$ openstack volume list
+--------------------------------------+------------------------------+--------+------+------------------------------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+------------------------------+--------+------+------------------------------------------------+
| 657de65d-22b4-44d3-9347-b1a562525ac0 | juju-58b38a-ceph-fs-volume-1 | in-use | 10 | Attached to juju-58b38a-ceph-fs-5 on /dev/vdd |
| 6004573f-bbd3-4a7a-83b1-934ba782eef2 | juju-58b38a-ceph-fs-volume-2 | in-use | 10 | Attached to juju-58b38a-ceph-fs-6 on /dev/vdd |
| 1d083c8d-1125-46b0-be5c-1fae2b10e3d0 | juju-58b38a-ceph-fs-volume-0 | in-use | 10 | Attached to juju-58b38a-ceph-fs-4 on /dev/vdd |
On a unit where the mapping is disordered. Note in the bellow case vdc is the recently attached volume and vdd is the swap device. But storage-get reports the volume is at /dev/vdd.
root@juju-58b38a-ceph-fs-5:/var/lib/juju/agents/unit-ceph-osd-1/charm# storage-get -s osd-devices/1
kind: block
location: /dev/vdd
root@juju-58b38a-ceph-fs-5:/var/lib/juju/agents/unit-ceph-osd-1/charm# parted -l
Model: Virtio Block Device (virtblk)
Disk /dev/vdd: 4295MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 4295MB 4295MB linux-swap(v1)
Model: Virtio Block Device (virtblk)
Disk /dev/vdb: 16.1GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:
Number Start End Size File system Flags
1 0.00B 16.1GB 16.1GB ext4
Error: /dev/vdc: unrecognised disk label
Model: Virtio Block Device (virtblk)
Disk /dev/vdc: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 16.1GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
14 1049kB 5243kB 4194kB bios_grub
15 5243kB 116MB 111MB fat32 boot, esp
1 116MB 16.1GB 16.0GB ext4
This seems suspicious:
Error: /dev/vdc: unrecognised disk label
We do want to make sure that we have the right "chain of custody" from what we requested to be provisioned from the underlying provider, and how that actually gets mounted on the machine. Openstack is clearly telling us that it put the storage we requested at /dev/vdd and we're passing that information on to the charm.
Offhand it feels like it would be an Openstack bug if it is telling us "on the instance I just created, you should expect this storage to be available at /dev/XXX" but the storage isn't actually there.
Now if Openstack was just telling us "your device ID is ABCDEF-123-456" and we're looking at mounted device labels and not lining them up correctly, that would be our bug.
But given your "openstack volume list" command above, it does seem relevant.
I wonder if the issue is that somehow the device isn't getting the right label, and thus whatever fstab entry that tells it where it should be mounted isn't matching.