The legacy-tempest-dsvm-full-devstack-plugin-ceph is failing 100% of the time with the error:
ImportError: No module named urlmap
when keystone tries to start. This is because the PasteDeploy package can't find the Paste package to import modules. This goes back to a workaround [0] we did in response to switching to the Pike UCA where we are forcing a pip install update of PasteDeploy only.
Now that we recently updated to the Queens UCA [1], we're running into a similar problem, this time *because* of the old workaround.
I think we probably need to just force a pip install update of Paste (to go along with the existing force pip install update of PasteDeploy) in order to make this work for any situation.
[0] https://review.openstack.org/510660
[1] https://review.openstack.org/554314
Apr 20 02:48:43.240138 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: CRITICAL keystone [-] Unhandled error: ImportError: No module named urlmap
Apr 20 02:48:43.240372 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone Traceback (most recent call last):
Apr 20 02:48:43.240730 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/bin/keystone-wsgi-public", line 54, in <module>
Apr 20 02:48:43.240949 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone application = initialize_public_application()
Apr 20 02:48:43.241204 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/opt/stack/new/keystone/keystone/server/wsgi.py", line 134, in initialize_public_application
Apr 20 02:48:43.241420 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone config_files=_get_config_files())
Apr 20 02:48:43.241712 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/opt/stack/new/keystone/keystone/server/wsgi.py", line 66, in initialize_application
Apr 20 02:48:43.241931 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone startup_application_fn=loadapp)
Apr 20 02:48:43.242229 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/opt/stack/new/keystone/keystone/server/common.py", line 48, in setup_backends
Apr 20 02:48:43.242444 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone res = startup_application_fn()
Apr 20 02:48:43.242682 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/opt/stack/new/keystone/keystone/server/wsgi.py", line 63, in loadapp
Apr 20 02:48:43.242905 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone 'config:%s' % find_paste_config(), name)
Apr 20 02:48:43.243140 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/opt/stack/new/keystone/keystone/version/service.py", line 51, in loadapp
Apr 20 02:48:43.243355 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone controllers.latest_app = deploy.loadapp(conf, name=name)
Apr 20 02:48:43.243567 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
Apr 20 02:48:43.243790 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone return loadobj(APP, uri, name=name, **kw)
Apr 20 02:48:43.244006 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 271, in loadobj
Apr 20 02:48:43.244219 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone global_conf=global_conf)
Apr 20 02:48:43.244438 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
Apr 20 02:48:43.244650 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone global_conf=global_conf)
Apr 20 02:48:43.244861 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
Apr 20 02:48:43.245115 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone return loader.get_context(object_type, name, global_conf)
Apr 20 02:48:43.245338 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 454, in get_context
Apr 20 02:48:43.245552 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone section)
Apr 20 02:48:43.245762 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
Apr 20 02:48:43.245973 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone object_type, name=use, global_conf=global_conf)
Apr 20 02:48:43.246195 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 406, in get_context
Apr 20 02:48:43.246408 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone global_conf=global_conf)
Apr 20 02:48:43.246618 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
Apr 20 02:48:43.246830 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone global_conf=global_conf)
Apr 20 02:48:43.247063 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
Apr 20 02:48:43.247277 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone return loader.get_context(object_type, name, global_conf)
Apr 20 02:48:43.247488 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 620, in get_context
Apr 20 02:48:43.247700 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone object_type, name=name)
Apr 20 02:48:43.247914 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 646, in find_egg_entry_point
Apr 20 02:48:43.248132 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone possible.append((entry.load(), protocol, entry.name))
Apr 20 02:48:43.248356 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2324, in load
Apr 20 02:48:43.248569 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone return self.resolve()
Apr 20 02:48:43.248782 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
Apr 20 02:48:43.249017 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone module = __import__(self.module_name, fromlist=['__name__'], level=0)
Apr 20 02:48:43.249252 ubuntu-xenial-inap-mtl01-0003633647 <email address hidden>[6449]: ERROR keystone ImportError: No module named urlmap
Patch is proposed here: https:/