grenade job fail due to missing memcache module

[Slawek Kaplonski]

We saw it quite often recently that grenade jobs are failing due to missing memcache module while keystone is starting.

Error message:

2019-03-29 00:22:39.361 | + lib/keystone:bootstrap_keystone:564 : /usr/local/bin/keystone-manage bootstrap --bootstrap-username admin --bootstrap-password secretadmin --bootstrap-project-name admin --bootstrap-role-name admin --bootstrap-service-name keystone --bootstrap-region-id RegionOne --bootstrap-admin-url http://10.209.34.128/identity --bootstrap-public-url http://10.209.34.128/identity
2019-03-29 00:22:41.201 | CRITICAL keystone [-] Unhandled error: ModuleNotFoundError: No module named 'memcache'
2019-03-29 00:22:41.201 | ERROR keystone Traceback (most recent call last):
2019-03-29 00:22:41.201 | ERROR keystone  File "/usr/local/bin/keystone-manage", line 10, in <module>
2019-03-29 00:22:41.201 | ERROR keystone  sys.exit(main())
2019-03-29 00:22:41.201 | ERROR keystone  File "/opt/stack/old/keystone/keystone/cmd/manage.py", line 41, in main
2019-03-29 00:22:41.201 | ERROR keystone  cli.main(argv=sys.argv, developer_config_file=developer_config)
2019-03-29 00:22:41.201 | ERROR keystone  File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 1216, in main
2019-03-29 00:22:41.201 | ERROR keystone  CONF.command.cmd_class.main()
2019-03-29 00:22:41.201 | ERROR keystone  File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 177, in main
2019-03-29 00:22:41.201 | ERROR keystone  klass = cls()
2019-03-29 00:22:41.201 | ERROR keystone  File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 66, in __init__
2019-03-29 00:22:41.201 | ERROR keystone  self.bootstrapper = bootstrap.Bootstrapper()
2019-03-29 00:22:41.201 | ERROR keystone  File "/opt/stack/old/keystone/keystone/cmd/bootstrap.py", line 31, in __init__
2019-03-29 00:22:41.201 | ERROR keystone  backends.load_backends()
2019-03-29 00:22:41.201 | ERROR keystone  File "/opt/stack/old/keystone/keystone/server/backends.py", line 41, in load_backends
2019-03-29 00:22:41.202 | ERROR keystone  cache.configure_cache()
2019-03-29 00:22:41.202 | ERROR keystone  File "/opt/stack/old/keystone/keystone/common/cache/core.py", line 124, in configure_cache
2019-03-29 00:22:41.202 | ERROR keystone  cache.configure_cache_region(CONF, region)
2019-03-29 00:22:41.202 | ERROR keystone  File "/usr/local/lib/python3.6/dist-packages/oslo_cache/core.py", line 235, in configure_cache_region
2019-03-29 00:22:41.202 | ERROR keystone  '%s.' % conf.cache.config_prefix)
2019-03-29 00:22:41.202 | ERROR keystone  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 591, in configure_from_config
2019-03-29 00:22:41.202 | ERROR keystone  "%sreplace_existing_backend" % prefix, False),
2019-03-29 00:22:41.202 | ERROR keystone  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 423, in configure
2019-03-29 00:22:41.202 | ERROR keystone  _config_prefix
2019-03-29 00:22:41.202 | ERROR keystone  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/api.py", line 89, in from_config_dict
2019-03-29 00:22:41.202 | ERROR keystone  for key in config_dict
2019-03-29 00:22:41.202 | ERROR keystone  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 208, in __init__
2019-03-29 00:22:41.202 | ERROR keystone  super(MemcacheArgs, self).__init__(arguments)
2019-03-29 00:22:41.202 | ERROR keystone  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 108, in __init__
2019-03-29 00:22:41.202 | ERROR keystone  self._imports()
2019-03-29 00:22:41.202 | ERROR keystone  File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 287, in _imports
2019-03-29 00:22:41.202 | ERROR keystone  import memcache # noqa
2019-03-29 00:22:41.202 | ERROR keystone ModuleNotFoundError: No module named 'memcache'

Logstash query which I used to find similar examples: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ModuleNotFoundError%3A%20No%20module%20named%20'memcache'%5C%22

Looks that it happens at least for 10 runs in last 7 days.

[Colleen Murphy (krinkle)]

This is an issue in devstack. You can see python-memcached is installed using python2.7:

http://logs.openstack.org/47/641747/8/check/neutron-grenade-dvr-multinode/f6fe13c/logs/old/devstacklog.txt#_2019-03-29_06_15_08_086

While keystone is installed with python3:

http://logs.openstack.org/47/641747/8/check/neutron-grenade-dvr-multinode/f6fe13c/logs/old/devstacklog.txt#_2019-03-29_06_15_13_680

For some reason the check to see whether it is compatible with python3 is failing:

http://logs.openstack.org/47/641747/8/check/neutron-grenade-dvr-multinode/f6fe13c/logs/old/devstacklog.txt#_2019-03-29_06_15_07_676

Even though the metadata is correct on pypi:

https://pypi.org/project/python-memcached/

The same problem is occurring for other packages too, so if it wasn't keystone it probably would have been something else:

http://logs.openstack.org/47/641747/8/check/neutron-grenade-dvr-multinode/f6fe13c/logs/old/devstacklog.txt#_2019-03-29_06_12_59_809

[Matt Riedemann (mriedem)]

This is likely a duplicate of bug 1820892.