| 2017-08-23 16:58:10 |
Danny Hammo |
description |
While testing openstack upgrade from M to N and O release, the ncc after the upgrade gets stuck at "db migration" and eventually (it takes about 20mins) errors to:
nova-cloud-controller/1* error idle 1/lxd/9 10.230.40.26 8774/tcp hook failed: "shared-db-relation-changed" for mysql:shared-db
Also noticed that the none leader ncc fails to start haproxy and memcached, which blocks the service:
nova-cloud-controller/0 blocked idle 0/lxd/9 10.230.40.39 8774/tcp Services not running that should be: haproxy, memcached
I have hit this issue on every try. All charms were upgraded to 17.02 release prior to openstack upgrade.
Charm release version:
Stdout: |
commit-sha-1: 8c0168c9c37b97f7e85684b740c9014a12a5e3a0
commit-short: 8c0168c
branch: HEAD
remote: https://github.com/openstack/charm-nova-cloud-controller
info-generated: Thu Jul 6 19:28:20 UTC 2017
note: This file should exist only in a built or released charm artifact (not in the charm source code tree).
UnitId: nova-cloud-controller/1
sosreport from the affected unit is attached |
While testing openstack upgrade from M to N and O release, the ncc after the upgrade gets stuck at "db migration" and eventually (it takes about 20mins) errors to:
nova-cloud-controller/1* error idle 1/lxd/9 10.230.40.26 8774/tcp hook failed: "shared-db-relation-changed" for mysql:shared-db
Also noticed that the none leader ncc fails to start haproxy and memcached, which blocks the service:
nova-cloud-controller/0 blocked idle 0/lxd/9 10.230.40.39 8774/tcp Services not running that should be: haproxy, memcached
I have hit this issue on every try. All charms were upgraded to 17.02 release prior to openstack upgrade.
The juju-unit logs throws the following traceback:
2017-08-11 00:25:41 INFO juju-log shared-db:14: Migrating the nova database.
2017-08-11 00:25:45 DEBUG shared-db-relation-changed Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
2017-08-11 00:25:45 DEBUG shared-db-relation-changed Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
2017-08-11 00:25:46 DEBUG shared-db-relation-changed Traceback (most recent call last):
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/shared-db-relation-changed", line 1190, in <module>
2017-08-11 00:25:46 DEBUG shared-db-relation-changed main()
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/shared-db-relation-changed", line 1184, in main
2017-08-11 00:25:46 DEBUG shared-db-relation-changed hooks.execute(sys.argv)
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
2017-08-11 00:25:46 DEBUG shared-db-relation-changed self._hooks[hook_name]()
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/nova_cc_utils.py", line 1102, in wrapped_f
2017-08-11 00:25:46 DEBUG shared-db-relation-changed f(*args)
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1864, in wrapped_f
2017-08-11 00:25:46 DEBUG shared-db-relation-changed restart_functions)
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/charmhelpers/core/host.py", line 655, in restart_on_change_helper
2017-08-11 00:25:46 DEBUG shared-db-relation-changed r = lambda_f()
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1863, in <lambda>
2017-08-11 00:25:46 DEBUG shared-db-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/shared-db-relation-changed", line 475, in db_changed
2017-08-11 00:25:46 DEBUG shared-db-relation-changed leader_init_db_if_ready()
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/shared-db-relation-changed", line 189, in leader_init_db_if_ready
2017-08-11 00:25:46 DEBUG shared-db-relation-changed migrate_nova_databases()
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2017-08-11 00:25:46 DEBUG shared-db-relation-changed return f(*args, **kwargs)
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/nova_cc_utils.py", line 790, in migrate_nova_databases
2017-08-11 00:25:46 DEBUG shared-db-relation-changed migrate_nova_database()
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/var/lib/juju/agents/unit-nova-cloud-controller-1/charm/hooks/nova_cc_utils.py", line 700, in migrate_nova_database
2017-08-11 00:25:46 DEBUG shared-db-relation-changed subprocess.check_output(cmd)
2017-08-11 00:25:46 DEBUG shared-db-relation-changed File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
2017-08-11 00:25:46 DEBUG shared-db-relation-changed raise CalledProcessError(retcode, cmd, output=output)
2017-08-11 00:25:46 DEBUG shared-db-relation-changed subprocess.CalledProcessError: Command '['nova-manage', 'db', 'sync']' returned non-zero exit status 1
Charm release version:
Stdout: |
commit-sha-1: 8c0168c9c37b97f7e85684b740c9014a12a5e3a0
commit-short: 8c0168c
branch: HEAD
remote: https://github.com/openstack/charm-nova-cloud-controller
info-generated: Thu Jul 6 19:28:20 UTC 2017
note: This file should exist only in a built or released charm artifact (not in the charm source code tree).
UnitId: nova-cloud-controller/1
sosreport from the affected unit is attached |
|
