Upgrade from kilo to liberty fails

[Liam Young]

When upgrading keystone from kilo to liberty the charm fails with:

2015-10-15 09:53:46 INFO juju-log Wrote template /etc/haproxy/haproxy.cfg.
2015-10-15 09:53:46 DEBUG juju-log Database is initialised
2015-10-15 09:53:47 INFO config-changed Traceback (most recent call last):
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 637, in <module>
2015-10-15 09:53:47 INFO config-changed main()
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 629, in main
2015-10-15 09:53:47 INFO config-changed hooks.execute(sys.argv)
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 704, in execute
2015-10-15 09:53:47 INFO config-changed self._hooks[hook_name]()
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 393, in wrapped_f
2015-10-15 09:53:47 INFO config-changed f(*args, **kwargs)
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1219, in inner_synchronize_ca_if_changed2
2015-10-15 09:53:47 INFO config-changed return f(*args, **kwargs)
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 173, in config_changed
2015-10-15 09:53:47 INFO config-changed update_all_identity_relation_units()
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 251, in update_all_identity_relation_units
2015-10-15 09:53:47 INFO config-changed ensure_initial_admin(config)
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 713, in ensure_initial_admin
2015-10-15 09:53:47 INFO config-changed return _ensure_initial_admin(config)
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 42, in _retry_on_exception_inner_2
2015-10-15 09:53:47 INFO config-changed return f(*args, **kwargs)
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 692, in _ensure_initial_admin
2015-10-15 09:53:47 INFO config-changed create_tenant("admin")
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 537, in create_tenant
2015-10-15 09:53:47 INFO config-changed import manager
2015-10-15 09:53:47 INFO config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 2, in <module>
2015-10-15 09:53:47 INFO config-changed from keystoneclient.v2_0 import client
2015-10-15 09:53:47 INFO config-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/__init__.py", line 1, in <module>
2015-10-15 09:53:47 INFO config-changed from keystoneclient.v2_0.client import Client # noqa
2015-10-15 09:53:47 INFO config-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 18, in <module>
2015-10-15 09:53:47 INFO config-changed from keystoneclient.auth.identity import v2 as v2_auth
2015-10-15 09:53:47 INFO config-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/__init__.py", line 14, in <module>
2015-10-15 09:53:47 INFO config-changed from keystoneclient.auth.identity import generic
2015-10-15 09:53:47 INFO config-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/generic/__init__.py", line 14, in <module>
2015-10-15 09:53:47 INFO config-changed from keystoneclient.auth.identity.generic.password import Password # noqa
2015-10-15 09:53:47 INFO config-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/generic/password.py", line 19, in <module>
2015-10-15 09:53:47 INFO config-changed from keystoneclient.auth.identity import v2
2015-10-15 09:53:47 INFO config-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v2.py", line 19, in <module>
2015-10-15 09:53:47 INFO config-changed from keystoneclient import access
2015-10-15 09:53:47 INFO config-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/access.py", line 20, in <module>
2015-10-15 09:53:47 INFO config-changed from oslo_utils import timeutils
2015-10-15 09:53:47 INFO config-changed File "/usr/lib/python2.7/dist-packages/oslo_utils/timeutils.py", line 24, in <module>
2015-10-15 09:53:47 INFO config-changed from debtcollector import removals
2015-10-15 09:53:47 INFO config-changed File "/usr/lib/python2.7/dist-packages/debtcollector/__init__.py", line 17, in <module>
2015-10-15 09:53:47 INFO config-changed from debtcollector import _utils
2015-10-15 09:53:47 INFO config-changed ImportError: cannot import name _utils
2015-10-15 09:53:47 ERROR juju.worker.uniter.operation runhook.go:103 hook "config-changed" failed: exit status 1

Reproduce with:

$ cat config.yaml
keystone:
  openstack-origin: cloud:trusty-kilo
$ juju deploy --config config.yaml local:trusty/keystone
Added charm "local:trusty/keystone-1" to the environment.
$ juju set keystone openstack-origin='cloud:trusty-liberty'

[Corey Bryant (corey.bryant)]

Looking at python-debtcollector, /usr/lib/python2.7/dist-packages/debtcollector/_utils.py should exist. Can you check if it's there?

[Liam Young (gnuoy)]

All the files referenced in the stack trace are there.

[Liam Young (gnuoy)]

The do_openstack_upgrade appears to run fine and it's update_all_identity_relation_units that is failing later on in the config-changed hook. Rerunning the hook runs update_all_identity_relation_units but not do_openstack_upgrade and that runs fine. The problem looks to me like a change in the packages underneath the running hook.