OSCI/ServerStack: non-pristine devices cause random testing failures

[Alex Kavanagh]

When running on ServerStack/OSCI, ceph-osd can fail with a 'non-pristine device' failure. This is due to a complex interplay of technologies that deliver a block device to the ceph-osd unit as part of Juju storage. Essentially, the issue is:

1. The bundle specs a block device to be provided to ceph-osd
2. Juju (storage) requests that the openstack provider (essentially cinder) attach a block device to the ceph-osd unit as part of the provisioning.
3. cinder responds with a block name which is provided to the ceph-osd device.
4. Randomly, the actual ceph-osd unit boots, gets its block-device BUT it has a different name to the one provided to the charm (via config).
5. Ceph then fails as the block device name it has for the OSD doesn't actually match the block device on the unit due to (most likely) the random order of assignment of block devices when the unit is booting.

This bug is a clearing house for solutions/tests to solve this problem, which may include:

1. disabling swap on the ceph-osd unit.
2. Trying to use a loop device on the unit instead for consistent naming, via cloud-init.

[Ryan Beisner (1chb1n)]

Example:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-ceilometer/668331/3/3639/test_charm_func_full_6913/juju-status.txt

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-ceilometer/668331/3/3639/index.html

[Chris MacNaughton (chris.macnaughton)]

I suggest that something like https://review.opendev.org/#/c/668489/ would likely resolve this issue.

[Ryan Beisner (1chb1n)]

FYI, another example:

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-ceph-osd/677734/3/11732/test_charm_func_smoke_14207/juju-status-zaza-279c9b2ed1f7.txt

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-ceph-osd/677734/3/11732/index.html

[Ryan Beisner (1chb1n)]

For clarity, flavor change in https://review.opendev.org/#/c/668489/ caused the instance type selected to be one without a swap device, which was theorized to be contributing the issue with device naming.

Rather than push constraints everywhere, I adjusted the flavor in the CI so that the default instance type no longer has a swap device, but has 2GB (up from 1536M).

Let's watch and see if these disappear on the whole.

[Ryan Beisner (1chb1n)]

This recent, unrelated change, was failing on the non-pristine issue in this bug:
 https://review.opendev.org/#/c/677874/

I've retriggered testing after the flavor modifications, need to confirm when that is done.

[Ryan Beisner (1chb1n)]

After some time of not seeing this, which seemed to have coincided with removal of the swap disk from the flavor in use, it seems to have resurfaced:

https://review.opendev.org/#/c/678651 --> https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-ceph-proxy/678651/2/3800/index.html --> https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-ceph-proxy/678651/2/3800/test_charm_func_full_7059/juju-status-zaza-5f5f09382a6a.txt

[Ryan Beisner (1chb1n)]

In further inspection, I had only updated 1 flavor to remove the swap disk. I've now updated all of the flavors, and confirmed that none have swap devices defined. Also, digging to confirm which flavors actually get used in these tests.

[Ryan Beisner (1chb1n)]

On a ceph-proxy func test run, confirmed that all instances are m1.small (no swap).

[Camille Rodriguez (camille.rodriguez)]

Seen again here: https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-ceph-radosgw/678671/2/3830/index.html

[Camille Rodriguez (camille.rodriguez)]

Seen again in charm-cinder-backup tests https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-cinder-backup/678677/1/3829/index.html

[Ryan Beisner (1chb1n)]

I believe that the occurrences in comments #9 and #10 are actually a separate bug, limited to Disco and Eoan: https://bugs.launchpad.net/charm-nova-compute/+bug/1842751

Further, I have yet to see the SAME type of occurrence as this original bug after we adjusted the flavors.