Add config for setting certs for https scraping

[Tim Penhey]

Juju controllers provide a basic auth endpoint for prometheus metrics, but needs to provide the tls cert config.

Right now, the following steps are needed to configure juju scraping:

juju controller-config ca-cert > controller-ca.cert
juju scp -m controller ./controller-ca.cert prometheus/0:~/controller-ca.cert
juju ssh -m controller prometheus/0
sudo mv controller-ca.cert /var/snap/prometheus/current

Then set the following scrape config:

- job_name: machine-0
  metrics_path: /introspection/metrics
  scheme: https
  static_configs:
    - targets: ['10.144.10.6:17070']
  basic_auth:
    username: user-prometheus
    password: --redacted--
  tls_config:
    ca_file: /var/snap/prometheus/current/controller-ca.cert

This has a few issues. Firstly it means we have to know where the certs can be read from, and also having to copy them across. Would be good if there was some charm config we could set, so we could set the following config:

-- begin scrape-certs.yaml
controller-ca.cert: |
  -----BEGIN CERTIFICATE-----
  ... details
  -----END CERTIFICATE-----

-- end scrape-certs.yaml

And be able to refer to those certs in the scrape config without having to know where the charm actually puts them:

-- begin scrape-jobs.yaml
- job_name: machine-0
  ... other details
  tls_config:
    ca_file: {cert-dir}/controller-ca.cert

-- end scrape-jobs.yaml

Then set config like this:

juju config prometheus <email address hidden> <email address hidden>

[Jacek Nykis (jacekn)]

Self-signed SSL certs are, generally speaking, a bad idea and they pose problems to operators.
The solution you propose does not solve the problem fully - you still have to ship certs around, the difference is that you use "juju config" to do that rather than "juju scp"

I think much better solutions to this problem could be:
1. Non-SSL metrics endpoint support in juju. See LP1671764
2. Small proxy charm, similar to https://jujucharms.com/u/axwalk/juju-introspection/
3. Add option to juju to use valid SSL cert
4. Relation support in juju controllers so that we can relate prometheus to it and get SSL cert using "the juju way"

IMO option 1 is the simplest one from prometheus perspective but unfortunately it depends on the juju team.
Another one worth looking into is option 2. The charm is nearly there, it probably needs some testing and improvements.

Adding new charm option to the prometheus charm to work around juju limitations is IMO wrong approach. If we get it fixed in juju then everybody benefits, including people who run non-juju deployed prometheus.

Having said that, if you disagree and are willing to add this new option to the charm I think we can merge it in.