When deploying OpenStack Train with Ceilometer and Gnocchi,
and also configuring all endpoints as TLS terminated with Vault,
ceilometer-agent-notification service will fail to connect to gnocchi.
Because of this, it will fail to collect metrics.
This is reproducible with the following bundle and network overlay.
Bundle:(the bundle is based on openstack-base bundle, adding ceilometer, gnocchi, and vault)
https://pastebin.ubuntu.com/p/2d87W7QvqS/
Overlays:
https://pastebin.ubuntu.com/p/vBQPqWkQNr/
After everything is deployed(Vault is unsealed, and ceilometer-upgrade is done),
and create any resources, for example, upload a glance image,
you will see the following error in /var/log/ceilometer/ceilometer-agent-notification.log in ceilometer/0
2020-03-18 13:05:53.565 28968 ERROR ceilometer.pipeline.sample [-] Pipeline meter_sink: Continue after error from publisher <ceilometer.publisher.gnocchi.GnocchiPublisher object at 0x7fc2de6ab4a8>: gnocchiclient.exceptions.BadRequest: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
Instead use the HTTPS scheme to access this URL, please.<br />
</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 172.16.10.204 Port 443</address>
</body></html>
(HTTP 400)
So it looks like, ceilometer-agent-notification still tries to connect to gnocchi with http, not https.
To mitigate this issue, restarting the ceilometer-agent-notification service will work for now.
I have attached the juju crashdump