| 2018-07-26 10:32:57 |
Frode Nordahl |
bug |
|
|
added bug |
| 2018-07-26 10:36:08 |
Frode Nordahl |
charm-keystone: importance |
Undecided |
High |
|
| 2018-07-26 10:36:11 |
Frode Nordahl |
charm-keystone: status |
New |
Triaged |
|
| 2018-07-26 10:36:19 |
Frode Nordahl |
charm-keystone: milestone |
|
18.08 |
|
| 2018-07-26 10:59:58 |
Frode Nordahl |
charm-keystone: importance |
High |
Medium |
|
| 2018-07-26 11:01:04 |
Frode Nordahl |
description |
2018-07-26 10:28:05 INFO juju-log Migrating the keystone database.
2018-07-26 10:28:16 DEBUG config-changed Traceback (most recent call last):
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 818, in <module>
2018-07-26 10:28:16 DEBUG config-changed main()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 811, in main
2018-07-26 10:28:16 DEBUG config-changed hooks.execute(sys.argv)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 823, in execute
2018-07-26 10:28:16 DEBUG config-changed self._hooks[hook_name]()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1453, in wrapped_f
2018-07-26 10:28:16 DEBUG config-changed restart_functions)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-07-26 10:28:16 DEBUG config-changed r = lambda_f()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1452, in <lambda>
2018-07-26 10:28:16 DEBUG config-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-07-26 10:28:16 DEBUG config-changed return f(*args, **kwargs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 196, in config_changed
2018-07-26 10:28:16 DEBUG config-changed do_openstack_upgrade_reexec(configs=CONFIGS)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 605, in do_openstack_upgrade_reexec
2018-07-26 10:28:16 DEBUG config-changed do_openstack_upgrade(configs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 647, in do_openstack_upgrade
2018-07-26 10:28:16 DEBUG config-changed migrate_database()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2018-07-26 10:28:16 DEBUG config-changed return f(*args, **kwargs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 695, in migrate_database
2018-07-26 10:28:16 DEBUG config-changed peer_store('db-initialised', 'True')
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/peerstorage/__init__.py", line 201, in peer_store
2018-07-26 10:28:16 DEBUG config-changed 'peer relation {}'.format(relation_name))
2018-07-26 10:28:16 DEBUG config-changed ValueError: Unable to detect peer relation cluster
2018-07-26 10:28:16 ERROR juju.worker.uniter.operation runhook.go:114 hook "config-changed" failed: exit status 1 |
There seems to be a race condition between the availability of the cluster relation and the usage of peer storage in the charm:
2018-07-26 10:28:05 INFO juju-log Migrating the keystone database.
2018-07-26 10:28:16 DEBUG config-changed Traceback (most recent call last):
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 818, in <module>
2018-07-26 10:28:16 DEBUG config-changed main()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 811, in main
2018-07-26 10:28:16 DEBUG config-changed hooks.execute(sys.argv)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 823, in execute
2018-07-26 10:28:16 DEBUG config-changed self._hooks[hook_name]()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1453, in wrapped_f
2018-07-26 10:28:16 DEBUG config-changed restart_functions)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-07-26 10:28:16 DEBUG config-changed r = lambda_f()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1452, in <lambda>
2018-07-26 10:28:16 DEBUG config-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-07-26 10:28:16 DEBUG config-changed return f(*args, **kwargs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 196, in config_changed
2018-07-26 10:28:16 DEBUG config-changed do_openstack_upgrade_reexec(configs=CONFIGS)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 605, in do_openstack_upgrade_reexec
2018-07-26 10:28:16 DEBUG config-changed do_openstack_upgrade(configs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 647, in do_openstack_upgrade
2018-07-26 10:28:16 DEBUG config-changed migrate_database()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2018-07-26 10:28:16 DEBUG config-changed return f(*args, **kwargs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 695, in migrate_database
2018-07-26 10:28:16 DEBUG config-changed peer_store('db-initialised', 'True')
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/peerstorage/__init__.py", line 201, in peer_store
2018-07-26 10:28:16 DEBUG config-changed 'peer relation {}'.format(relation_name))
2018-07-26 10:28:16 DEBUG config-changed ValueError: Unable to detect peer relation cluster
2018-07-26 10:28:16 ERROR juju.worker.uniter.operation runhook.go:114 hook "config-changed" failed: exit status 1 |
|
| 2018-07-26 11:01:35 |
Frode Nordahl |
description |
There seems to be a race condition between the availability of the cluster relation and the usage of peer storage in the charm:
2018-07-26 10:28:05 INFO juju-log Migrating the keystone database.
2018-07-26 10:28:16 DEBUG config-changed Traceback (most recent call last):
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 818, in <module>
2018-07-26 10:28:16 DEBUG config-changed main()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 811, in main
2018-07-26 10:28:16 DEBUG config-changed hooks.execute(sys.argv)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 823, in execute
2018-07-26 10:28:16 DEBUG config-changed self._hooks[hook_name]()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1453, in wrapped_f
2018-07-26 10:28:16 DEBUG config-changed restart_functions)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-07-26 10:28:16 DEBUG config-changed r = lambda_f()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1452, in <lambda>
2018-07-26 10:28:16 DEBUG config-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-07-26 10:28:16 DEBUG config-changed return f(*args, **kwargs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 196, in config_changed
2018-07-26 10:28:16 DEBUG config-changed do_openstack_upgrade_reexec(configs=CONFIGS)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 605, in do_openstack_upgrade_reexec
2018-07-26 10:28:16 DEBUG config-changed do_openstack_upgrade(configs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 647, in do_openstack_upgrade
2018-07-26 10:28:16 DEBUG config-changed migrate_database()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2018-07-26 10:28:16 DEBUG config-changed return f(*args, **kwargs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 695, in migrate_database
2018-07-26 10:28:16 DEBUG config-changed peer_store('db-initialised', 'True')
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/peerstorage/__init__.py", line 201, in peer_store
2018-07-26 10:28:16 DEBUG config-changed 'peer relation {}'.format(relation_name))
2018-07-26 10:28:16 DEBUG config-changed ValueError: Unable to detect peer relation cluster
2018-07-26 10:28:16 ERROR juju.worker.uniter.operation runhook.go:114 hook "config-changed" failed: exit status 1 |
There seems to be a race condition between the availability of the cluster relation and the usage of peer storage in the charm.
From time to time I am seeing this:
2018-07-26 10:28:05 INFO juju-log Migrating the keystone database.
2018-07-26 10:28:16 DEBUG config-changed Traceback (most recent call last):
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 818, in <module>
2018-07-26 10:28:16 DEBUG config-changed main()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 811, in main
2018-07-26 10:28:16 DEBUG config-changed hooks.execute(sys.argv)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 823, in execute
2018-07-26 10:28:16 DEBUG config-changed self._hooks[hook_name]()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1453, in wrapped_f
2018-07-26 10:28:16 DEBUG config-changed restart_functions)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-07-26 10:28:16 DEBUG config-changed r = lambda_f()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1452, in <lambda>
2018-07-26 10:28:16 DEBUG config-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-07-26 10:28:16 DEBUG config-changed return f(*args, **kwargs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 196, in config_changed
2018-07-26 10:28:16 DEBUG config-changed do_openstack_upgrade_reexec(configs=CONFIGS)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 605, in do_openstack_upgrade_reexec
2018-07-26 10:28:16 DEBUG config-changed do_openstack_upgrade(configs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 647, in do_openstack_upgrade
2018-07-26 10:28:16 DEBUG config-changed migrate_database()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2018-07-26 10:28:16 DEBUG config-changed return f(*args, **kwargs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 695, in migrate_database
2018-07-26 10:28:16 DEBUG config-changed peer_store('db-initialised', 'True')
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/peerstorage/__init__.py", line 201, in peer_store
2018-07-26 10:28:16 DEBUG config-changed 'peer relation {}'.format(relation_name))
2018-07-26 10:28:16 DEBUG config-changed ValueError: Unable to detect peer relation cluster
2018-07-26 10:28:16 ERROR juju.worker.uniter.operation runhook.go:114 hook "config-changed" failed: exit status 1 |
|
| 2018-07-26 12:58:16 |
Frode Nordahl |
charm-keystone: assignee |
|
Frode Nordahl (fnordahl) |
|
| 2018-07-26 12:58:19 |
Frode Nordahl |
charm-keystone: status |
Triaged |
In Progress |
|
| 2018-07-27 12:14:49 |
Frode Nordahl |
charm-keystone: importance |
Medium |
High |
|
| 2018-07-30 08:33:24 |
OpenStack Infra |
charm-keystone: status |
In Progress |
Fix Committed |
|
| 2018-09-06 14:40:55 |
David Ames |
charm-keystone: status |
Fix Committed |
Fix Released |
|
