Unable to run stack.sh on fresh new Ubuntu Xenial 16.04 LTS, script fails with "No module named 'memcache' "

[Denis Makogon]

Unable to run stack.sh on fresh new Ubuntu Xenial 16.04 LTS, script fails with "No module named 'memcache' "

Traceback:

+lib/keystone:bootstrap_keystone:630 /usr/local/bin/keystone-manage bootstrap --bootstrap-username admin --bootstrap-password ubuntu --bootstrap-project-name admin --bootstrap-role-name admin --bootstrap-service-name keystone --bootstrap-region-id RegionOne --bootstrap-admin-url http://192.168.0.115/identity_admin --bootstrap-public-url http://192.168.0.115/identity --bootstrap-internal-url http://192.168.0.115/identity
2016-11-28 11:51:39.723 15663 CRITICAL keystone [-] ImportError: No module named 'memcache'
2016-11-28 11:51:39.723 15663 TRACE keystone Traceback (most recent call last):
2016-11-28 11:51:39.723 15663 TRACE keystone File "/usr/local/bin/keystone-manage", line 10, in <module>
2016-11-28 11:51:39.723 15663 TRACE keystone sys.exit(main())
2016-11-28 11:51:39.723 15663 TRACE keystone File "/opt/stack/keystone/keystone/cmd/manage.py", line 45, in main
2016-11-28 11:51:39.723 15663 TRACE keystone cli.main(argv=sys.argv, config_files=config_files)
2016-11-28 11:51:39.723 15663 TRACE keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 1269, in main
2016-11-28 11:51:39.723 15663 TRACE keystone CONF.command.cmd_class.main()
2016-11-28 11:51:39.723 15663 TRACE keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 365, in main
2016-11-28 11:51:39.723 15663 TRACE keystone klass = cls()
2016-11-28 11:51:39.723 15663 TRACE keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 66, in __init__
2016-11-28 11:51:39.723 15663 TRACE keystone self.load_backends()
2016-11-28 11:51:39.723 15663 TRACE keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 129, in load_backends
2016-11-28 11:51:39.723 15663 TRACE keystone drivers = backends.load_backends()
2016-11-28 11:51:39.723 15663 TRACE keystone File "/opt/stack/keystone/keystone/server/backends.py", line 32, in load_backends
2016-11-28 11:51:39.723 15663 TRACE keystone cache.configure_cache()
2016-11-28 11:51:39.723 15663 TRACE keystone File "/opt/stack/keystone/keystone/common/cache/core.py", line 124, in configure_cache
2016-11-28 11:51:39.723 15663 TRACE keystone cache.configure_cache_region(CONF, region)
2016-11-28 11:51:39.723 15663 TRACE keystone File "/usr/local/lib/python3.5/dist-packages/oslo_cache/core.py", line 201, in configure_cache_region
2016-11-28 11:51:39.723 15663 TRACE keystone '%s.' % conf.cache.config_prefix)
2016-11-28 11:51:39.723 15663 TRACE keystone File "/usr/local/lib/python3.5/dist-packages/dogpile/cache/region.py", line 552, in configure_from_config
2016-11-28 11:51:39.723 15663 TRACE keystone "%swrap" % prefix, None),
2016-11-28 11:51:39.723 15663 TRACE keystone File "/usr/local/lib/python3.5/dist-packages/dogpile/cache/region.py", line 417, in configure
2016-11-28 11:51:39.723 15663 TRACE keystone _config_prefix
2016-11-28 11:51:39.723 15663 TRACE keystone File "/usr/local/lib/python3.5/dist-packages/dogpile/cache/api.py", line 81, in from_config_dict
2016-11-28 11:51:39.723 15663 TRACE keystone for key in config_dict
2016-11-28 11:51:39.723 15663 TRACE keystone File "/usr/local/lib/python3.5/dist-packages/dogpile/cache/backends/memcached.py", line 208, in __init__
2016-11-28 11:51:39.723 15663 TRACE keystone super(MemcacheArgs, self).__init__(arguments)
2016-11-28 11:51:39.723 15663 TRACE keystone File "/usr/local/lib/python3.5/dist-packages/dogpile/cache/backends/memcached.py", line 108, in __init__
2016-11-28 11:51:39.723 15663 TRACE keystone self._imports()
2016-11-28 11:51:39.723 15663 TRACE keystone File "/usr/local/lib/python3.5/dist-packages/dogpile/cache/backends/memcached.py", line 287, in _imports
2016-11-28 11:51:39.723 15663 TRACE keystone import memcache # noqa
2016-11-28 11:51:39.723 15663 TRACE keystone ImportError: No module named 'memcache'
2016-11-28 11:51:39.723 15663 TRACE keystone

local.conf

[[local|localrc]]

USE_PYTHON3=True
PYTHON3_VERSION=3.5

Python: 3.5.2

Ubuntu version (lsb_release -a):
Distributor ID: Ubuntu
Description: Ubuntu 16.04 LTS
Release: 16.04
Codename: xenial

[Denis Makogon (lildee1991)]

Python3 package for memcache has different name from Python2 package:
https://pypi.python.org/pypi/python3-memcached

Python2 package:
https://pypi.python.org/pypi/python-memcached

This is also a problem for:
https://bitbucket.org/zzzeek/dogpile.cache

And subsequently for Keystone running under Python-3.5.2 with memcached

[Denis Makogon (lildee1991)]

It appears that this bug also affected by OpenStack requirements:
https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L308

because there's no constraint for python3-memcached

[Tony Breeds (o-tony)]

Actucally python3-memcached is defunct and hasn't made a release for 3+ years. python-memcache is the canonical python 2 and python3 library

[Tony Breeds (o-tony)]

Please include your entire local.con (with password removed) so we can get to the bottom of this

[Tony Breeds (o-tony)]

python-memcache is the canonical version. I don't believe that this is a devstack issue

[Lance Bragstad (lbragstad)]

In keystone, we consider python-memcache to be an extra package (at least according to our setup.cfg) [0]. It's not being installed as a dependency in keystone's requirements.txt file.

[0] https://github.com/openstack/keystone/blob/4fc95c8f9201eb09b8d1ff54b8e8d260264aabbc/setup.cfg#L31-L32

[Lance Bragstad (lbragstad)]

In order to make python-memcache a first-class requirement, we'd also need to propose it to OpenStack requirements [0]. I believe Morgan has more scope around the context of python-memcache being an extra dependency of keystone than I do, so I want to make sure he weighs in on the discussion as well.

[0] https://github.com/openstack/requirements

[Tony Breeds (o-tony)]

Actually it's already in global-requirements.txt otherwise it wouldn't be allowed in setup.cfg

http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt#n232

[Steve Martinelli (stevemar)]

So what's getting laid down? Is it python-memcache or python3-memcache? As Tony said, python-memcache is the correct version for both py2 and py3. python3-memcache has not been updated in 3 years.

From a keystone point of view, we specify python-memcache.

I'm not sure what devstack is laying down, but if it's python3-memcache, thats wrong.

[Tony Breeds (o-tony)]

So I gave this a try (it's still building) but I'm more convinced that it's something to do with your customised environment.

There are a number of system tools that don't work with Python3 so please provide a description on how you made python3 the default, what patches you applied to devstack to even get it to run.

[Tony Breeds (o-tony)]

Yup definitely something strange going on. My steps to reproduce:

1) Install Xenial
2) cd /usr/bin; sudo ln -sf python3.5 python)
3) cd /usr/bin; sudo sed -i 's,usr/bin/python,usr/bin/python2.7,g' pycompile pyclean
4) cd /path/to/devstack; sed -i 's,usr/bin/python,usr/bin/python2.7,g' tools/outfilter.py tools/worlddump/py
5) cd /path/to/devstack ; cat >> local.conf <<EOM
[[local|localrc]]
USE_PYTHON3=True
PYTHON3_VERSION=3.5
Q_PLUGIN=ml2
INSTALL_TESTONLY_PACKAGES=True
ADMIN_PASSWORD=LALALA
MYSQL_PASSWORD=LALALA
RABBIT_PASSWORD=LALALA
SERVICE_PASSWORD=LALALA
SERVICE_TOKEN=XXXXXX
API_RATE_LIMIT=False
LOGFILE=/opt/stack/logs/stack.sh.log

disable_service n-net
enable_service q-svc
enable_service q-dhcp
enable_service q-l3
enable_service q-meta
enable_service q-vpn
enable_service q-metering
enable_service q-lbaas
enable_service q-fwaas
enable_service neutron
enable_service q-agt
EOM

6) cd /path/to/devstack ; ./stack.sh

After waiting for a while *devstack* does fail with:
2016-12-02 05:43:40.749 | ++lib/keystone:create_keystone_accounts:369 openstack project show admin -f value -c id
2016-12-02 05:43:42.535 | Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.

Which doesn't really surprise me and it's the bug at hand. On my system the memcache module is actually provided by python-memcached

 stack@devstack-xenial:~/projects/openstack/openstack-dev/devstack$ python -c 'import memcache;print(memcache.__file__)'
 /usr/local/lib/python3.5/dist-packages/memcache.py

So I'm quite convinced that a we don't yet support a python3 *only* system and b) there isn't anything wrong here that we don't expect due to "a"

[Steve Martinelli (stevemar)]

Has this been solved by Dim's latest work to get keystone working on py35? We have a gate job for that now, see a successful run here: http://logs.openstack.org/73/421573/1/check/gate-keystone-dsvm-py35-functional-v3-only-ubuntu-xenial-nv/3c5863b/console.html

It seems to be able to lay down python-memcache, use py3 and execute a bunch of tests.

[Steve Martinelli (stevemar)]

Dims' patch: https://github.com/openstack/keystone/commit/b4012e808552cef2d51ed81d29c77acbe04bfb33

[Morgan Fainberg (mdrnstm)]

Marked as invalid. Not a lot we can do about python-memcached, but with the py3-first testing i think we are beyond this as a bug.