During a series upgrade of percona-cluster, nova-cloud-controller threw a hook error because the mysql server wasn't available during an ampq relation hook.
unit-nova-cloud-controller-2: 17:36:59 ERROR unit.nova-cloud-controller/2.juju-log amqp:39: list_cells failed
b'An error has occurred:\nTraceback (most recent call last):\n File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1657, in main\n ret = fn(*
fn_args, **fn_kwargs)\n File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1377, in list_cells\n context.get_admin_context())\n File "/usr/
lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper\n result = fn(cls, context, *args, **kwargs)\n File "/usr/lib/python2.7/d
ist-packages/nova/objects/cell_mapping.py", line 137, in get_all\n db_mappings = cls._get_all_from_db(context)\n File "/usr/lib/python2.7/dist-packages/os
lo_db/sqlalchemy/enginefacade.py", line 985, in wrapper\n with self._transaction_scope(context):\n File "/usr/lib/python2.7/contextlib.py", line 17, in __
enter__\n return self.gen.next()\n File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1035, in _transaction_scope\n conte
xt=context) as resource:\n File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__\n return self.gen.next()\n File "/usr/lib/python2.7/dist-packag
es/oslo_db/sqlalchemy/enginefacade.py", line 638, in _session\n bind=self.connection, mode=self.mode)\n File "/usr/lib/python2.7/dist-packages/oslo_db/sql
alchemy/enginefacade.py", line 403, in _create_session\n self._start()\n File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line
489, in _start\n engine_args, maker_args)\n File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 513, in _setup_for_connection
\n sql_connection=sql_connection, **engine_kwargs)\n File "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 43, in decorator\n return w
rapped(*args, **kwargs)\n File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 184, in create_engine\n test_conn = _test_connection
(engine, max_retries, retry_interval)\n File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 370, in _test_connection\n six.reraise
(type(de_ref), de_ref)\n File "<string>", line 3, in reraise\nDBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'1
0.5.110.241\' ([Errno 113] EHOSTUNREACH)")\n\n'
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed Traceback (most recent call last):
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/hooks/amqp-relation-changed", line 1377, in <module>
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed main()
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/hooks/amqp-relation-changed", line 1369, in main
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed hooks.execute(sys.argv)
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/charmhelpers/core/hookenv.py", line 934, in execute
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed self._hooks[hook_name]()
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/hooks/nova_cc_utils.py", line 1631, in wrapped_f
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed return f(*args, **kwargs)
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/charmhelpers/contrib/openstack/utils.py", line 1684, in wrapped_f
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed stopstart, restart_functions)
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/charmhelpers/core/host.py", line 741, in restart_on_change_helper
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed r = lambda_f()
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/charmhelpers/contrib/openstack/utils.py", line 1683, in <lambda>
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache']
,
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/hooks/amqp-relation-changed", line 324, in amqp_changed
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed update_cell_db_if_ready_allowed_units()
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/hooks/amqp-relation-changed", line 174, in update_cell_db_if_ready_allowed_units
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed update_cell_db_if_ready(db_rid=rid, unit=unit)
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/hooks/amqp-relation-changed", line 157, in update_cell_db_if_ready
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed ncc_utils.update_cell_database()
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/hooks/nova_cc_utils.py", line 876, in update_cell_database
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed cell1_uuid = get_cell_uuid('cell1')
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/hooks/nova_cc_utils.py", line 830, in get_cell_uuid
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed cells = get_cell_details()
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-2/char
m/hooks/nova_cc_utils.py", line 848, in get_cell_details
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed out = subprocess.check_output(cmd).decode('utf-8')
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/usr/lib/python3.6/subprocess.py", line 356, in check_
output
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed **kwargs).stdout
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed File "/usr/lib/python3.6/subprocess.py", line 438, in run
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed output=stdout, stderr=stderr)
unit-nova-cloud-controller-2: 17:36:59 DEBUG unit.nova-cloud-controller/2.amqp-relation-changed subprocess.CalledProcessError: Command '['sudo', 'nova-manage'
, 'cell_v2', 'list_cells', '--verbose']' returned non-zero exit status 1.
unit-nova-cloud-controller-2: 17:36:59 ERROR juju.worker.uniter.operation hook "amqp-relation-changed" failed: exit status 1
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit b53c597b15b4a0d
Author: Liam Young <email address hidden>
Date: Wed Mar 11 07:16:01 2020 +0000
Do not run db updates if db is in maintenance mode
As witih the shared-db hook do not run db updates if the database
is in maintenace mode
Closes-Bug: #1866864
Change-Id: I65619271d8a421