[CI] "test_cinder_volume_backup_create_delete" test invoked whilst cinder service was restarting (probably)

[Alex Kavanagh]

This is an unstable test:

2021-02-18 01:16:44 [INFO] ERROR: test_cinder_volume_backup_create_delete (zaza.openstack.charm_tests.cinder_backup_swift_proxy.tests.CinderBackupSwiftTest)
2021-02-18 01:16:44 [INFO] Create volume backup and then delete it.
2021-02-18 01:16:44 [INFO] ----------------------------------------------------------------------
2021-02-18 01:16:44 [INFO] Traceback (most recent call last):
2021-02-18 01:16:44 [INFO] File "/tmp/tmp.lrjSA8iVKi/func-smoke/lib/python3.5/site-packages/zaza/openstack/charm_tests/cinder_backup_swift_proxy/tests.py", line 47, in test_cinder_volume_backup_create_delete
2021-02-18 01:16:44 [INFO] volume.id)
2021-02-18 01:16:44 [INFO] File "/tmp/tmp.lrjSA8iVKi/func-smoke/lib/python3.5/site-packages/zaza/openstack/utilities/openstack.py", line 2508, in create_volume_backup
2021-02-18 01:16:44 [INFO] name=name)
2021-02-18 01:16:44 [INFO] File "/tmp/tmp.lrjSA8iVKi/func-smoke/lib/python3.5/site-packages/cinderclient/v2/volume_backups.py", line 70, in create
2021-02-18 01:16:44 [INFO] return self._create('/backups', body, 'backup')
2021-02-18 01:16:44 [INFO] File "/tmp/tmp.lrjSA8iVKi/func-smoke/lib/python3.5/site-packages/cinderclient/base.py", line 302, in _create
2021-02-18 01:16:44 [INFO] resp, body = self.api.client.post(url, body=body)
2021-02-18 01:16:44 [INFO] File "/tmp/tmp.lrjSA8iVKi/func-smoke/lib/python3.5/site-packages/cinderclient/client.py", line 217, in post
2021-02-18 01:16:44 [INFO] return self._cs_request(url, 'POST', **kwargs)
2021-02-18 01:16:44 [INFO] File "/tmp/tmp.lrjSA8iVKi/func-smoke/lib/python3.5/site-packages/cinderclient/client.py", line 205, in _cs_request
2021-02-18 01:16:44 [INFO] return self.request(url, method, **kwargs)
2021-02-18 01:16:44 [INFO] File "/tmp/tmp.lrjSA8iVKi/func-smoke/lib/python3.5/site-packages/cinderclient/client.py", line 191, in request
2021-02-18 01:16:44 [INFO] raise exceptions.from_response(resp, body)
2021-02-18 01:16:44 [INFO] cinderclient.exceptions.ClientException: Service cinder-backup could not be found. (HTTP 503) (Request-ID: req-f05ba8fc-cd47-49e5-99b1-2b0af84f837e)

Whilst in status:

cinder/0* active executing 6 172.17.109.27 8776/tcp Unit is ready

(although with async log collection this is hard to know).

Would probably pass if the call used a retry to try again when the service is available.

[Aurelien Lourot (aurelien-lourot)]

This is now consistently happening in the gate and makes us "blind":
https://review.opendev.org/c/openstack/charm-cinder-backup-swift-proxy/+/791185
https://review.opendev.org/c/openstack/charm-cinder-backup-swift-proxy/+/794493
https://openstack-ci-reports.ubuntu.com/artifacts/558/794493/1/check/bionic-queens-swift-v3/5587e03/job-output.txt