Juju status stuck in 'Allowed_units list provided but this unit not present'

[Cedric Lemarchand]

In an process of moving (remove-unit then add-unit) a kestone unit, the new unit 'keystone-mysql-router/114:' is stuck in 'Allowed_units list provided but this unit not present'.

With some debug with is_db_ready() in charmhelpers/contrib/openstack/utils.py, it seams that the code "relation_get(rid=rid, unit=unit, attribute=key)" returns None.

This is confirmed by looking the relation in juju where keystone-mysql-router/114: miss the 'allowed_units' part :

'''
  - relation-id: 122
    endpoint: shared-db
    related-endpoint: shared-db
    application-data: {}
    related-units:
      keystone-mysql-router/0:
        in-scope: true
        data:
          allowed_units: keystone/2
          db_host: 127.0.0.1
          db_port: "3306"
          egress-subnets: 10.140.4.196/32
          ingress-address: 10.140.4.196
          password: ****
          private-address: 10.140.4.196
          wait_timeout: "3600"
      keystone-mysql-router/1:
        in-scope: true
        data:
          allowed_units: keystone/1
          db_host: 127.0.0.1
          db_port: "3306"
          egress-subnets: 10.140.4.181/32
          ingress-address: 10.140.4.181
          password: ****
          private-address: 10.140.4.181
          wait_timeout: "3600"
      keystone-mysql-router/114:
        in-scope: true
        data:
          db_host: 127.0.0.1
          db_port: "3306"
          egress-subnets: 10.140.4.9/32
          ingress-address: 10.140.4.9
          password: ****
          private-address: 10.140.4.9
          wait_timeout: "3600"

'''

Versions informations bellow :

'''
keystone 17.0.1 waiting 3 keystone stable 330 no Some units are not ready
keystone-mysql-router 8.0.30 active 3 mysql-router 8.0/stable 30 no Unit is ready
'''

I am not sure that there wasnt a "juju remove-unit --force" issued.
Also before the unit remove/add process, the keystone-mysql-router has been upgraded from 8.0.19 to 8.0.30, could be related.

Please advice if more information is needed.

[Mathew Clarke (matclarke)]

I'm also hitting this bug I've tried manually adding the missing allowed_unit data back into the relation data with "juju run -u keystone-mysql-router/<Unit no> -- "relation-set -r shared-db:579 allowed_units='keystone/<unit no>'"" and it fixes it for around 5 mins then juju deletes the allowed_unit and then complains its missing again. I'm running the following charm version on focal.

keystone 658
mysql-router 111

[Alex Kavanagh (ajkavanagh)]

@mattclarke Please could you add the debug logs for the mysql-router related to keystone? That may shed light on what's going on. also if you could do a "juju show-unit <keystone-mysql-router-unit>" and "juju show-unit <keystone-unit>" that may also help to work out what's going on. Obviously, please redact any passwords, etc., if this is a production system. Thanks.

[Launchpad Janitor (janitor)]

[Expired for MySQL Router Charm because there has been no activity for 60 days.]

[Alex Kavanagh (ajkavanagh)]

I'm fairly certain that this is not a mysql-router bug, but in fact related to (or caused by) the mysql-innodb-cluster bug: https://bugs.launchpad.net/charm-mysql-innodb-cluster/+bug/2064127

i.e. it's about inconsistent relation data being available by the mysql-innodb-cluster unit depending on leadership changes.