[focal][ussuri] nova-cloud-controller shared-db-relation-changed fails due to sql permissions error

[Alex Kavanagh]

Essentially:

2020-04-16 15:45:36 DEBUG juju-log shared-db:44: Database is NOT initialised
2020-04-16 15:45:37 DEBUG juju-log shared-db:44: Generating template context for amqp
2020-04-16 15:45:37 INFO juju-log shared-db:44: Generating template context for cell v2 share-db
2020-04-16 15:45:37 INFO juju-log shared-db:44: Migrating the nova-api database.

...

(1045, "Access denied for user \'nova\'@\'172.17.110.10\' (using password: YES)")

2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/shared-db-relation-changed", line 1383, in <module>
2020-04-16 15:46:56 DEBUG shared-db-relation-changed main()
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/shared-db-relation-changed", line 1375, in main
2020-04-16 15:46:56 DEBUG shared-db-relation-changed hooks.execute(sys.argv)
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/core/hookenv.py", line 934, in execute
2020-04-16 15:46:56 DEBUG shared-db-relation-changed self._hooks[hook_name]()
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/nova_cc_utils.py", line 1632, in wrapped_f
2020-04-16 15:46:56 DEBUG shared-db-relation-changed return f(*args, **kwargs)
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/contrib/openstack/utils.py", line 1682, in wrapped_f
2020-04-16 15:46:56 DEBUG shared-db-relation-changed return restart_on_change_helper(
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/core/host.py", line 741, in restart_on_change_helper
2020-04-16 15:46:56 DEBUG shared-db-relation-changed r = lambda_f()
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/contrib/openstack/utils.py", line 1683, in <lambda>
2020-04-16 15:46:56 DEBUG shared-db-relation-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/shared-db-relation-changed", line 413, in db_changed
2020-04-16 15:46:56 DEBUG shared-db-relation-changed leader_init_db_if_ready()
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/shared-db-relation-changed", line 118, in leader_init_db_if_ready
2020-04-16 15:46:56 DEBUG shared-db-relation-changed ncc_utils.migrate_nova_databases()
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2020-04-16 15:46:56 DEBUG shared-db-relation-changed return f(*args, **kwargs)
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/nova_cc_utils.py", line 991, in migrate_nova_databases
2020-04-16 15:46:56 DEBUG shared-db-relation-changed initialize_cell_databases()
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-0/charm/hooks/nova_cc_utils.py", line 802, in initialize_cell_databases
2020-04-16 15:46:56 DEBUG shared-db-relation-changed subprocess.check_output(cmd)
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
2020-04-16 15:46:56 DEBUG shared-db-relation-changed return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2020-04-16 15:46:56 DEBUG shared-db-relation-changed File "/usr/lib/python3.8/subprocess.py", line 512, in run
2020-04-16 15:46:56 DEBUG shared-db-relation-changed raise CalledProcessError(retcode, process.args,
2020-04-16 15:46:56 DEBUG shared-db-relation-changed subprocess.CalledProcessError: Command '['nova-manage', 'cell_v2', 'map_cell0']' returned non-zero exit status 255.

This was from an OSCI run, so I've not been able to examine the database. I'll do a manual run and see if it's a timing issue (i.e. not all mysql8 machines are up), a "writing to the wrong mysql8 server -- i.e. one that is R/O, or just a new permissions issue around the user being used.

[Dmitrii Shcherbakov (dmitriis)]

Marking as incomplete for now. The issue looks important but we need to find out if it is reproducible.

[Alex Kavanagh (ajkavanagh)]

This is in fact due to a mysql-router missing feature; the ability for the parent charm to connect to more than one database at once. Marking it is a duplicate of: https://bugs.launchpad.net/charm-mysql-router/+bug/1876188 which is tracking the addition of that feature.