Some kind of race condition causes DB volumes to appear on clean disks

[Stephen Taylor]

The OSD init script using ceph-volume to deploy Bluestore OSDs is seeing unexplained Bluestore DB volumes on disks that should be clean. These volumes are "orphans" as they have no corresponding OSD data volume and must be removed by the OSD init script prior to OSDs getting initialized. If they aren't, then there is insufficient disk space to create the necessary DB and WAL volumes for the OSD(s) being deployed.

Where these DB volumes are coming from is a mystery, as is the reason why cleaning them up and proceeding with the OSD deployment doesn't cause them to reappear and instead allows the OSD deployment to complete successfully. These volumes are only created by the ceph-volume Bluestore OSD init script, but the script appears to be encountering them before it executes any of its logic that creates them.

There must be some hidden issue here that will eventually cause a more serious issue and we need to understand how this is happening. Thus far this issue has only been observed in the gate, but it seems likely to occur elsewhere eventually if not resolved.

[Stephen Taylor (st053q)]

This is addressed with https://review.opendev.org/c/openstack/openstack-helm-infra/+/767638