Add mysqld-prometheus-exporter to charm

[JamesLin]

Transfer the metrics to prometheus and use a common library to integrate between nagios and prometheus. This may help the COS migration in the end of this year. If it works, it can be a good pattern for other modules.

If we go with standard Prometheus mysql exporter and then build the rules for nagios externally (ie in another charm perhaps), then the migration for MySQL is probably a bit more straightforward as the Prometheus relation already exists and the charm can add alert rules for alert manager and bootstack can build off the nagios stuff as a bridge

TODO:

- Snap install exporter https://github.com/prometheus/mysqld_exporter
  - [ ] Need implement a snap containing the mysqld prometheus exporter from the prometheus community.

  - [ ] Install snap in the charm.
  - [ ] Create user for exporter in mysql

[JamesLin (jneo8)]

question: Should we install exporter by default?

Probably not, we should control snap installation by `charms.layer.snap` python package base on config `enable_prometheus_exporter`.

Or create user && install after relation join prometheus

[JamesLin (jneo8)]

question: When should we create user for exporter?

We definitely want to create an user for exporter.

But where should we implement this logic?
This also relate to https://bugs.launchpad.net/charm-mysql-innodb-cluster/+bug/1982712/comments/1
We can create user by default or depend on config.

[JamesLin (jneo8)]

We should not default install snap or either create user on the charm.
The action will be relation-driven when prometheus_scrape interface join.

[James Page (james-page)]

Agreed - I think the snap install and configuration of the required user should happen when its required - i.e. when the charm is related to either Prometheus or Nagios.

Neither operation will be impacting on MySQL (i.e. no restart required) so I think this is fine.

It means if the charm is not monitored, we don't incur the load of the exporter or the increased security attack surface of another network endpoint and the user in MySQL.

[JamesLin (jneo8)]

Another issue:

I found there are some similar charms like:

https://launchpad.net/charm-prometheus-grok-exporter
https://launchpad.net/charm-prometheus-blackbox-exporter

They implement exporter as a charm and relation to workload && prometheus.
I think this way of design is more flexible than install in the mysql.

We should implement another charm for prometheus mysql exporter with juju operator framework.
Relation the charm as subordinate to each mysql node in mysql innodb cluster

But later we will need to discuss how to give user & password to the exporter charm.

[Billy Olsen (billy-olsen)]

@jneo8, no I think this should be part of the mysql charm itself. Moving it to another charm makes sense if it can be abstracted or provided for multiple charms - but that's not the case here.

It is very domain specific, i.e you aren't going to add a mysql-prometheus-exporter charm to something that *isn't* mysql so you should just embed it in the charm itself.

[JamesLin (jneo8)]

More information about why don't use subordinate charm.

1. Charms are supposed to be devops distilled, meaning encompassing best practice operations into the charms themselves. That way, when you deploy mysql you'll get not only mysql but you'll also get a solid way on how to monitor that mysql cluster, you'll get operational semantics on alerts that are generally important and actions that allow you to simplify the operations (like backup, etc, etc)
 1.1 once you start pulling the monitoring pieces out of the charm, you start to lose the operations bits

2. As Billy mention, the charm should include the monitor logic itself.
  2.1. The mysql prometheus exporter charm is not generic, we won't relate it to another charm
  2.2 The exporter will live with the principle charm, the objective is to be able to just juju deploy mysql and then relate mysql and the prometheus charm itself

3. There may be a version management issue when exporter lives outside of the principle charm
  2.1 If we shipped the principle charm without the exporter compatibility fixed, and therefore operator could't upgrade to the newer principle or they'd lose the monitoring. Putting them together in the same project, so we can cover it with testing.

4. It may bring more complexity when we have to deal with two different things at the same time(without subordinate) - but this also happen in subordinate pattern.