CephRBDMirrorControlledFailoverTest fails in charm test gate

[Ryan Beisner]

The charm test fails periodically while trying to validate failover and fallback. This particular case was observed on Bionic Queens.

2019-08-22 23:32:44 [INFO] ERROR: test_fail_over_fall_back (zaza.openstack.charm_tests.ceph.rbd_mirror.tests.CephRBDMirrorControlledFailoverTest)
2019-08-22 23:32:44 [INFO] Validate controlled fail over and fall back.

ceph-rbd-mirror/0* blocked idle 12 172.17.115.16 Unit is ready (Pools WARNING (2) OK (1) Images unknown (2))

Artifact index:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-ceph-rbd-mirror/677902/1/3671/index.html

Juju status:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-ceph-rbd-mirror/677902/1/3671/test_charm_func_full_6947/juju-status-zaza-e70eb723d022.txt

Juju crashdump:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-ceph-rbd-mirror/677902/1/3671/test_charm_func_full_6947/juju-crashdump-4f78b72a-095c-46d7-808a-21c98080a09f.tar.xz

Test console output:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-ceph-rbd-mirror/677902/1/3671/consoleText.test_charm_func_full_6947.txt

[Ryan Beisner (1chb1n)]

Also FYI - I was examining the test output, which is not very clear. I want to add "I ran this action on this machine" type of logging, but libjuju already has that, so instead of adding that to zaza, I've proposed that here: https://github.com/juju/python-libjuju/issues/346