ceph-radosgw can't start apache2 because it can't find certs

[Bas de Bruijne]

In testrun:
https://solutions.qa.canonical.com/testruns/testRun/e3926795-0553-4c71-84d9-a5d3c9693dce

A single ceph-radosgw unit stays blocked:

ceph-radosgw/1 blocked (...) Services not running that should be: apache2

In the syslogs we can see that it can't start apache2 because the ssl certs don't exist:

-----------------------------
AH00526: Syntax error on line 13 of /etc/apache2/sites-enabled/openstack_https_frontend.conf:
SSLCertificateFile: file '/etc/apache2/ssl/ceph-radosgw/cert_rados-internal.silo4.solutionsqa' does not exist or is empty
juju-01e84b-1-lxd-2 apachectl[97662]: Action 'start' failed.
juju-01e84b-1-lxd-2 apachectl[97662]: The Apache error log may have more information.
-----------------------------

NRPE also reports the SSL socket being down:

-----------------------------
nrpe[219005]: INFO: SSL Socket Shutdown.
nrpe[219038]: Error: (use_ssl == true): Request packet version was invalid!
nrpe[219038]: Could not read request from client 10.246.166.192, bailing out...
-----------------------------

I can't find the cause of the SSL issues in the logs.

Crashdumps & other artefacts:
https://oil-jenkins.canonical.com/artifacts/e3926795-0553-4c71-84d9-a5d3c9693dce/index.html

[Alexander Balderson (asbalderson)]

I saw this again on nova-cloud-controller on wallaby.

Apache couldn't start because the cert didn't exist, but none of the nova services could start because the cert was invalid for connecting to keystone.

Adding nova-cloud-controller

Testrun at:
https://solutions.qa.canonical.com/testruns/testRun/8470733c-86b7-4623-9590-19d7a05d53bf

[Alex Kavanagh (ajkavanagh)]

As this is happening across two different charms, I'm beginning to think that this is a Vault charm HA issue. It might be related to this bug

* https://bugs.launchpad.net/vault-charm/+bug/1946361 <-- raised by Marian

e.g. any API service that is related to Vault when in HA mode (during deployment), possibly on a resource constrained system, may go into error with this bug. However, let's leave this open here to see if it happens with any other services.

[Bas de Bruijne (basdbruijne)]

Marked as invalid due to inactivity