cluster unhealthy after adding certificates relation to vault

[Jason Hobbs]

After adding a certificates relation to vault, my mysql-innodb-cluster relations are in a bad state: MySQL InnoDB Cluster not healthy: None

status: https://paste.ubuntu.com/p/584Fqy7gXb/
bundle: https://paste.ubuntu.com/p/ThsGB82s2x/

There is an error in the log:

2020-12-15 21:06:00 ERROR juju-log Cluster is unavailable: Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
mysqlsh.DBError: MySQL Error (1045): Shell.connect: Access denied for user 'clusteruser'@'192.168.33.39' (using password: YES)

[Jason Hobbs (jason-hobbs)]

sub'd to field high

[Liam Young (gnuoy)]

It looks like the mysql units are missing a cert, from the /var/log/mysql/error.log on mysql-innodb-cluster/0 I see:

2020-12-15T20:12:18.808324Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error loading certification file /etc/mysql/tls/mysql-innodb-cluster/cert_192.168.33.39'

[Liam Young (gnuoy)]

I'll continue to dig in and see if I can see why that cert is missing.

[Liam Young (gnuoy)]

The symlinks for the certificate and key that match the db-router binding are missing.

In an environment in which the bug was reproduced:

# network-get --primary-address db-router
10.80.0.10

# ls -l /etc/mysql/tls/mysql-innodb-cluster/
total 16
lrwxrwxrwx 1 root root 65 Dec 17 13:31 cert_10.0.0.154 -> /etc/mysql/tls/mysql-innodb-cluster/cert_juju-0eac99-1-lxd-0.maas
-rw-r----- 1 root mysql 2574 Dec 17 13:31 cert_juju-0eac99-1-lxd-0.maas
lrwxrwxrwx 1 root root 64 Dec 17 13:31 key_10.0.0.154 -> /etc/mysq

I think there are two issues going on here. Firstly recent fixes to charmhelpers.contrib.openstack.cert_utils are not in the stable version of the mysql-innodb-cluster charm. Secondly it looks, to me, like the charm is requesting a cert for the wrong internal IPs. I am redeploying at the moment to confirm if a change still needs to be made the certificate request that the mysql-innodb-cluster charm generates.

Bundle to reproduce issue:
series: focal
machines:
  "0":
  "1":
  "2":
applications:
  mysql-innodb-cluster:
    #charm: cs:mysql-innodb-cluster-3
    charm: cs:~openstack-charmers-next/mysql-innodb-cluster
    num_units: 3
    to:
    - lxd:0
    - lxd:1
    - lxd:2
    options:
      enable-binlogs: true
    bindings:
      "": public
      certificates: public
      cluster: internal
      coordinator: public
      db-router: internal
      shared-db: public
    trust: true
  vault:
    charm: cs:vault-41
    num_units: 1
    to:
    - lxd:0
    options:
      hostname: vault-internal.production.solutionsqa
      vip: 10.80.0.200
    constraints: spaces=public
    bindings:
      "": internal
      access: internal
      certificates: internal
      cluster: internal
      db: internal
      etcd: internal
      external: internal
      ha: internal
      nrpe-external-master: internal
      secrets: internal
      shared-db: internal
    trust: true
  vault-mysql-router:
    charm: cs:mysql-router-4
    bindings:
      "": public
      certificates: public
      db-router: internal
      juju-info: public
      shared-db: public
    trust: true
  hacluster-vault:
    charm: cs:hacluster-72
    options:
      cluster_count: 1
    bindings:
      "": public
      ha: public
      hanode: public
      juju-info: public
      nrpe-external-master: public
      pacemaker-remote: public
      peer-availability: public
    trust: true
relations:
- - mysql-innodb-cluster:db-router
  - vault-mysql-router:db-router
- - vault-mysql-router:shared-db
  - vault:shared-db
- - hacluster-vault:ha
  - vault:ha
- - mysql-innodb-cluster:certificates
  - vault:certificates

Then configure vault:

functest-configure -m <model-name> -c zaza.openstack.charm_tests.vault.setup.auto_initialize_no_validation_no_wait

[David Ames (thedac)]

After much consensus building, we have a solution in play:

Landed
https://github.com/juju/charm-helpers/pull/556
https://review.opendev.org/c/openstack/charms.openstack/+/769811

Waiting on OSCI and reviews:
https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/769843

[David Ames (thedac)]

Has passed tests and is ready for review and landing:
https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/769843