Missing /var/snap/grafana/common/ssl causes breakage when adding certificates relation

[Paul Goins]

I recently tried to enable HTTPS on Grafana via vault-managed certificates.

This in theory should have been done via:

  juju add-relation vault:certificates grafana:certificates

However, the service did not switch over to HTTPS.

Juju logs showed the following snippet:

2022-03-21 19:32:27 INFO juju-log certificates:107: Invoking reactive handler: reactive/tls_client.py:18:store_ca
2022-03-21 19:32:27 DEBUG jujuc server.go:211 running hook tool "juju-log"
2022-03-21 19:32:27 INFO juju-log certificates:107: Writing CA certificate to /usr/local/share/ca-certificates/grafana.crt
2022-03-21 19:32:27 DEBUG certificates-relation-changed Updating certificates in /etc/ssl/certs...
2022-03-21 19:32:29 DEBUG certificates-relation-changed 1 added, 0 removed; done.
2022-03-21 19:32:29 DEBUG certificates-relation-changed Running hooks in /etc/ca-certificates/update.d...
2022-03-21 19:32:29 WARNING certificates-relation-changed rsync: change_dir#3 "/var/snap/grafana/common/ssl" failed: No such file or directory (2)
2022-03-21 19:32:29 WARNING certificates-relation-changed rsync error: errors selecting input/output files, dirs (code 3) at main.c(713) [Receiver=3.1.2]
2022-03-21 19:32:29 DEBUG certificates-relation-changed E: /etc/ca-certificates/update.d/sync-grafana-snap exited with code 3.
2022-03-21 19:32:29 DEBUG certificates-relation-changed done.
2022-03-21 19:32:29 DEBUG jujuc server.go:211 running hook tool "juju-log"
2022-03-21 19:32:29 INFO juju-log certificates:107: Generated ca-certificates.crt for grafana
2022-03-21 19:32:29 DEBUG jujuc server.go:211 running hook tool "juju-log"
2022-03-21 19:32:29 DEBUG juju-log certificates:107: tracer: set flag tls_client.ca_installed

The /etc/ca-certificates/update.d/sync-grafana-snap script performs an rsync of /etc/ssl/certs/ca-certificates.crt to /var/snap/grafana/common/ssl/ca-certificates.crt; however, the parent directory /var/snap/grafana/common/ssl does not exist, and so this does not complete. And unfortunately, this is also not caught; the charm appears to be in a non-broken state, yet services are running on plain HTTP rather than HTTPS.

[Martin Kalcok (martin-kalcok)]

I was only partially able to reproduce this issue. I can see the warnings in the logs when I relate grafana with vault/easyrsa. However it appears to be only temporary issue because when I log into the unit, the directory (and certificates) are in place and service is running over HTTPS as expected.

I think we should add `mkdir -p` into the sync script to avoid those warnings but this might not be the true cause of your issues.

Could you please share more details about the affected environment (ubuntu series/grafana charm version)?

[Nicholas Malacarne (nicholas-malacarne)]

Series: bionic
Grafana Version: 7.4.1
Grafana Charm Rev: 51

[Martin Kalcok (martin-kalcok)]

I'm still unable to reproduce this issue even with the specific setup described above. Since this issue is not widespread and was reported only by single customer it's likely that it;s an issue with that specific setup rather than the with the charm itself.