Swift volume fstab entries should use UUID for charm-created filesystems

[James Hebden]

Currently, the block device's /dev path is written to the `/etc/fstab` entry for any volumes initialised by the charm.

The charm should read the UUID of the created filesystem and use the `UUID=<calculated UUID>` format when writing out the device name to `/etc/fstab`. This prevents incorrect filesystems from being mounted under `/srv/node` when and if device names change, for example, due to kernel upgrades (most likely) or storage controller changes (less likely).

[James Page (james-page)]

+1

[Paul Collins (pjdc)]

Just a little background colour on how difficult a situation this can yield. I ran into this recently with a machine that simply had an extra disk (array B below) added, which changed the enumeration order when it was rebooted:

Smart Array P420i in Slot 0 (Embedded)

   array A

      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 1 TB, OK)

   array B

      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SATA, 3 TB, OK) <-- the new disk

   array C

      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 3 TB, OK)

[...]

   array L

      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SATA, 3 TB, OK)

Although in hindsight I probably could have deduced the new enumeration order -- I didn't look at this output until just now -- it is very difficult to take a batch of disks and work out which swift devices they are. XFS doesn't store the FS creation date anywhere that I can find, there is no file or directory on the filesystem that has never been touched since creation, and mkfs.xfs doesn't log the filesystem UUID, so the Juju charm log is no help.

James Hebden has written a script to look at the contents of the disks and match them up to the swift rings somehow (perhaps he can expand here) but I couldn't get it running on our older Swift deployment. I ended up parsing the object-server logs, extracting the device names and Swift partition numbers, and probing each disk to see which had the most partitions for each device, which worked in the end. But it was a bit harrowing. :)

[Junien F (axino)]

Please reconsider the criticality of this bug. When using LVM devices, the charm adds "/dev/dm-X" as device name in fstab. However, /dev/dm-X is not a stable name and can change between reboots if you have multiple VGs. With systemd, this resulted in my server dropping in a rescue shell (apparently that's what systemd does when it can't mount a filesystem in fstab).

Thanks

[Junien F (axino)]

Also, using a LV results in the mountpoint being /srv/node/dm-X, which is not ideal.

[Tytus Kurek (tkurek)]

Fix proposed to branch: master
Review: https://review.openstack.org/#/c/643162/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/643162
Committed: https://git.openstack.org/cgit/openstack/charm-swift-storage/commit/?id=edd00bee63c71ff34eaa728f7ed7fdb0efd13984
Submitter: Zuul
Branch: master

commit edd00bee63c71ff34eaa728f7ed7fdb0efd13984
Author: Tytus Kurek <email address hidden>
Date: Wed Mar 13 21:15:41 2019 +0000

    Use UUIDs of storage devices in fstab

    This patchset implements a logic to put UUIDS of storage devices in the
    fstab instead of device names.

    Change-Id: I2170c54b25ac9b3faacb3fd0b2889d86c531395f
    Closes-Bug: 1729478