Comment 3 for bug 1881596

Revision history for this message
Aurelien Lourot (aurelien-lourot) wrote :

Hitting this on focal-ussuri while implementing a scaleback test for hacluster https://review.opendev.org/#/c/741592/ . In this test we're scaling back keystone. On focal-ussuri hacluster isn't the only subordinate of keystone: there is also mysql-router. When scaling back we end up with:

keystone/0* blocked executing 3 172.17.100.7 5000/tcp Missing relations: database, Allowed_units list provided but this unit not present
  keystone-mysql-router/2 error idle 172.17.100.7 hook failed: "shared-db-relation-departed"
keystone/1 blocked idle 4 172.17.100.15 5000/tcp Database not initialised
  hacluster/1 blocked idle 172.17.100.15 Insufficient peer units for ha cluster (require 3)
  keystone-mysql-router/1 active idle 172.17.100.15 Unit is ready
keystone/2 blocked idle 5 172.17.100.23 5000/tcp Database not initialised
  hacluster/0* blocked idle 172.17.100.23 Insufficient peer units for ha cluster (require 3)
  keystone-mysql-router/0* active idle 172.17.100.23 Unit is ready

2020-08-27 19:23:24 ERROR juju-log shared-db:4: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-keystone-mysql-router-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-keystone-mysql-router-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-keystone-mysql-router-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-keystone-mysql-router-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-keystone-mysql-router-2/charm/reactive/mysql_router_handlers.py", line 102, in proxy_shared_db_responses
    instance.proxy_db_and_user_responses(db_router, shared_db)
  File "lib/charm/openstack/mysql_router.py", line 481, in proxy_db_and_user_responses
    unit = sending_interface.all_joined_units[0]
  File "/var/lib/juju/agents/unit-keystone-mysql-router-2/.venv/lib/python3.8/site-packages/charms/reactive/endpoints.py", line 582, in __getitem__
    return super().__getitem__(self._translate_key(key))
IndexError: list index out of range

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-hacluster/741592/10/6706/index.html