(latest/edge) AssertionError: Exporter permission not correct on 172.16.0.95

[Felipe Reyes]

The gate for mysql-innodb-cluster is failing with this error:

======================================================================
FAIL: test_01_exporter_http_check (zaza.openstack.charm_tests.mysql.test_prometheus_mysql_exporter.PrometheusMySQLExporterTest)
Check exporter endpoint is working.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-innodb-cluster/src/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/mysql/test_prometheus_mysql_exporter.py", line 95, in test_01_exporter_http_check
    self._check_exporter_http_check(unit)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-innodb-cluster/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-innodb-cluster/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-innodb-cluster/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-innodb-cluster/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-innodb-cluster/src/.tox/func-target/lib/python3.8/site-packages/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-mysql-innodb-cluster/src/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/mysql/test_prometheus_mysql_exporter.py", line 81, in _check_exporter_http_check
    self.fail(
AssertionError: Exporter permission not correct on 172.16.0.95

Ref: https://openstack-ci-reports.ubuntu.com/artifacts/549/888789/2/check/jammy/5499575/

[Felipe Reyes (freyes)]

Reproduced the issue in a lab environment:

$ curl -s http://10.5.0.123:9104/metrics | grep mysql_up
# HELP mysql_up Whether the MySQL server is up.
# TYPE mysql_up gauge
mysql_up 0
$ cat /var/snap/mysqld-exporter/common/etc/my.cnf
[client]
user=prom_exporter
password=9RZz3PyCpNZPGHcbP4PJM9jHqBXn6JSYXfrSsn3HzBk
port=3306
host=10.5.0.123
$ mysql -h 10.5.0.123 -u prom_exporter --password='9RZz3PyCpNZPGHcbP4PJM9jHqBXn6JSYXfrSsn3HzBk'
mysql: File '/etc/mysql/mysql.conf.d/mysqld.cnf' not found (OS errno 13 - Permission denied)
mysql: [ERROR] Stopped processing the 'includedir' directive in file /etc/mysql/my.cnf at line 21.
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'prom_exporter'@'10.5.0.123' (using password: YES)
$ sudo mysql -h 10.5.0.123 -u prom_exporter --password='9RZz3PyCpNZPGHcbP4PJM9jHqBXn6JSYXfrSsn3HzBk'
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'prom_exporter'@'10.5.0.123' (using password: YES)

https://pastebin.ubuntu.com/p/PPnWDVbxQd/

Setting the password to the value stored in the leader databag fixes the issue:

mysql> SET PASSWORD FOR prom_exporter@10.5.0.123 = '9RZz3PyCpNZPGHcbP4PJM9jHqBXn6JSYXfrSsn3HzBk';
Query OK, 0 rows affected (0.01 sec)

$ curl -s http://10.5.0.123:9104/metrics | grep mysql_up
# HELP mysql_up Whether the MySQL server is up.
# TYPE mysql_up gauge
mysql_up 1

so the issue seems to be in the creation of the prom_exporter user.

[Felipe Reyes (freyes)]

I added some instrumentation code[0] where I could catch the prom_exporter user being created with a password set to None:

2023-08-28 23:29:39 INFO unit.mysql-innodb-cluster/1.juju-log server.go:316 prometheus:10: Invoking reactive handler: reactive/prometheus_mysql_exporter_handlers.py:12:create_local_prometheus_exporter_user
2023-08-28 23:29:39 INFO unit.mysql-innodb-cluster/1.juju-log server.go:316 prometheus:10: prom_exporter@10.5.3.204 = None
2023-08-28 23:29:39 INFO unit.mysql-innodb-cluster/1.juju-log server.go:316 prometheus:10: prom_exporter@localhost = None
2023-08-28 23:29:39 ERROR unit.mysql-innodb-cluster/1.juju-log server.go:316 prometheus:10: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/reactive/prometheus_mysql_exporter_handlers.py", line 18, in create_local_prometheus_exporter_user
    if not instance.create_user(
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 339, in wrapped_f
    return self(f, *args, **kw)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 430, in __call__
    do = self.iter(retry_state=retry_state)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 367, in iter
    return fut.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 433, in __call__
    result = fn(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/lib/charm/openstack/mysql_innodb_cluster.py", line 628, in create_user
    assert password, msg
AssertionError: Password can't be empty: None (user: prom_exporter, addresses: ['10.5.3.204', 'localhost'])

[0]
+ if user == "prom_exporter":
+ for address in addresses:
+ ch_core.hookenv.log('%s@%s = %s' % (user, address, password),
+ "INFO")
+ msg = "Password can't be empty: %s (user: %s, addresses: %s)" % \
+ (password, user, addresses)
+ assert password, msg

[Felipe Reyes (freyes)]

https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/892989

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/892989
Committed: https://opendev.org/openstack/charm-mysql-innodb-cluster/commit/83bc7b36aeb038b7ce2d16ec023fa971bfc091f1
Submitter: "Zuul (22348)"
Branch: master

commit 83bc7b36aeb038b7ce2d16ec023fa971bfc091f1
Author: Felipe Reyes <email address hidden>
Date: Mon Aug 28 20:58:47 2023 -0400

    Delay prom_exporter user creation until password is available.

    There are situations where the create_local_prometheus_exporter_user()
    handler could be executed in a follower unit before the leader units
    gets the chance to generate a password for the prom_exporter user and
    store it in the leader databag, this has the consequence that the
    prom_exporter user is created with the password 'None' (string
    representation of the `None` value), this leads to the problem that the
    mysql_exporter daemon can't connect to mysqld.

    This patch checks that the password is available, if it isn't, then bail
    out and let subsequent executions to re-evaluate this condition.

    Closes-Bug: #2033306
    Change-Id: I58fc489fd4cc4e06b11c293f63e73bd0ab18897f