CI fault due to racy test

[Peter Sabaini]

Seeing a fault in CI when exercising ceph-radosgw where TLS config fails in a vault-enabled model because Apache is missing a certificate

This seems due to a race between the test and setting up a cert via vault

Details:

The test_005_virtual_hosted_bucket test fails in ceph/tests.py:1267 to create a bucket.

We don't have certificates yet. On the ceph-radosgw unit in syslog we can see:

Apr 9 21:48:47 juju-afc113-zaza-b381a1e6720b-3 radosgw[71139]: 2024-04-09T21:48:47.534+0000 7fad2748afc0 -1 shutting down
Apr 9 21:48:47 juju-afc113-zaza-b381a1e6720b-3 systemd[1]: Stopping Ceph rados gateway...
Apr 9 21:48:47 juju-afc113-zaza-b381a1e6720b-3 systemd[1]: <email address hidden>: Succeeded.
Apr 9 21:48:47 juju-afc113-zaza-b381a1e6720b-3 systemd[1]: Stopped Ceph rados gateway.
Apr 9 21:48:47 juju-afc113-zaza-b381a1e6720b-3 systemd[1]: Started Ceph rados gateway.
Apr 9 21:48:47 juju-afc113-zaza-b381a1e6720b-3 systemd[1]: Stopping The Apache HTTP Server...
Apr 9 21:48:48 juju-afc113-zaza-b381a1e6720b-3 apachectl[75233]: AH00526: Syntax error on line 40 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
Apr 9 21:48:48 juju-afc113-zaza-b381a1e6720b-3 apachectl[75233]: SSLCertificateFile: file '/etc/apache2/ssl/ceph-radosgw/cert_rgw.example.com' does not exist or is empty
Apr 9 21:48:48 juju-afc113-zaza-b381a1e6720b-3 apachectl[75231]: Action 'stop' failed.

First the test method configures virtual hosting buckets and public hostname:

2024-04-09 21:49:47 INFO juju.worker.uniter.operation runhook.go:159 ran "config-changed" hook (via explicit, bespoke hook script)

The test fails in ceph/tests.py:1267 with an SSL error:

2024-04-09 21:49:51.854202 | focal-medium | 2024-04-09 21:49:51 [INFO] urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:1131)

Only then the vault relation kicks in:

2024-04-09 21:49:52 DEBUG juju.worker.uniter.operation executor.go:85 running operation run relation-changed (14; unit: vault/0) hook for ceph-radosgw/0
...
2024-04-09 21:49:55 DEBUG unit.ceph-radosgw/0.juju-log server.go:316 certificates:14: Writing file /etc/apache2/ssl/ceph-radosgw/cert_rgw.example.com root:root 640

And apache succeeds:

Apr 9 21:50:06 juju-afc113-zaza-b381a1e6720b-3 systemd[1]: Started The Apache HTTP Server.

CI artefacts:
https://openstack-ci-reports.ubuntu.com/artifacts/1e3/915339/3/check/vault-focal-yoga-namespaced/1e3db73/index.html

[Peter Sabaini (peter-sabaini)]

Fixed in https://github.com/openstack-charmers/zaza-openstack-tests/pull/1197