HA Ceph RGW deployment fails to serve object IO with "ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol"

[utkarsh bhatt]

Ceph RGW deployed on an SQA environment fails to serve Object IO requests with SSL error.

This behaviour is seen on charms as old as quincy/stable.

First found during a run of zaza-tests:

2024-04-23 07:27:24 [INFO] ## Running Test zaza.openstack.charm_tests.ceph.tests.CephRGWTest ##
...
2024-04-23 07:33:01 [INFO] ======================================================================
2024-04-23 07:33:01 [INFO] ERROR: test_003_object_storage_and_secondary_block (zaza.openstack.charm_tests.ceph.tests.CephRGWTest)
2024-04-23 07:33:01 [INFO] Verify Object Storage API and Secondary Migration block.
2024-04-23 07:33:01 [INFO] ----------------------------------------------------------------------
...
2024-04-23 07:33:01 [INFO] ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
...
2024-04-23 07:33:01 [INFO] botocore.exceptions.SSLError: SSL validation failed for https://10.X.Y.170:443/zaza-container [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
2024-04-23 07:33:01 [INFO] ======================================================================
2024-04-23 07:33:01 [INFO] ERROR: test_005_virtual_hosted_bucket (zaza.openstack.charm_tests.ceph.tests.CephRGWTest)
...
2024-04-23 07:33:01 [INFO] raise SSLError(endpoint_url=request.url, error=e)
2024-04-23 07:33:01 [INFO] botocore.exceptions.SSLError: SSL validation failed for https://10.X.Y.170:443/zaza-bucket [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

Running a trivial curl against Ceph RGW endpoint fails with:
$ curl -kv https://10.X.Y.170:443/zaza-container
* Trying 10.X.Y.170:443...
* Connected to 10.X.Y.170 (10.X.Y.170) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* error:0A000126:SSL routines::unexpected eof while reading
* Closing connection 0
curl: (35) error:0A000126:SSL routines::unexpected eof while reading

JUJU Status
===========
$ juju status ceph-radosgw
Model Controller Cloud/Region Version SLA Timestamp
ceph-standalone foundations-maas maas_cloud/default 2.9.49 unsupported 08:20:33Z

App Version Status Scale Charm Channel Rev Exposed Message
ceph-radosgw 17.2.7 active 3 ceph-radosgw reef/candidate 573 no Unit is ready
hacluster-ceph-radosgw 2.1.2 active 3 hacluster 2.4/stable 131 no Unit is ready and clustered

Unit Workload Agent Machine Public address Ports Message
ceph-radosgw/0* active idle 3/lxd/1 10.X.Y.170 443/tcp Unit is ready
  hacluster-ceph-radosgw/0* active idle 10.X.Y.170 Unit is ready and clustered
ceph-radosgw/1 active idle 4/lxd/1 10.X.Y.171 443/tcp Unit is ready
  hacluster-ceph-radosgw/1 active idle 10.X.Y.171 Unit is ready and clustered
ceph-radosgw/2 active idle 5/lxd/1 10.X.Y.173 443/tcp Unit is ready
  hacluster-ceph-radosgw/2 active idle 10.X.Y.173 Unit is ready and clustered

Detailed HAproxy and Apache conf: https://pastebin.canonical.com/p/rVBbsWMbhh/
Behaviour with http/https curl calls: https://pastebin.canonical.com/p/Jm6GWQP3zm/