python-memcache (and therefore) token memcache persistence driver does not support ipv6

[Frode Nordahl]

(morganfainberg):
OpenStack Manuals (for both Master and Kilo) need to be updated to eliminate the recommendation to use the memcache token persistence backend. The memcache token persistence backend is a poor choice due to performance concerns of the code itself, the fact that it is assumed that the token backend is stable storage (memcached is not) and can expose security concerns if restarted in some scenarios (PKI tokens and revoked tokens becoming valid again), and finally that the python-memcache library is all around poor (it will not work with IPv6 and is not Python3 compatible, among other poor design choices).

========================================================================
The memcache backend driver that utilizes "python-memcache" does not support IPv6.

I have included three scenarios (A, B and C) that will reproduce the bug and a control test that succeeds with same configuration using IPv4-resolving hostname.

To reproduce scenario A: Bare IPv6 address in config
1) Configure keystone according to http://docs.openstack.org/kilo/install-guide/install/apt/content/keystone-install.html
2) In section [memcache] in /etc/keystone/keystone.conf change servers = line:
 servers = 2001:db8:1000:1:f816:3eff:fe2a:f9c7:11211,2001:db8:1000:1:f816:3eff:fee9:9ce3:11211,2001:db8:1000:1:f816:3eff:fead:8f7f:11211
3) Restart keystone/apache
4) Attempt to issue token:
 openstack --os-auth-url http://192.168.0.15:35357 --os-project-name admin --os-username admin --os-auth-type password token issue

ERROR: openstack An unexpected error prevented the server from fulfilling your request: Unable to parse connection string: "2001:db8:1000:1:f816:3eff:fe2a:f9c7:11211" (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-7c2bfd39-4b83-462b-92c6-f75f7677c8e5)

To reproduce scenario B: IPv6 address enclosed in brackets
1) Configure keystone according to http://docs.openstack.org/kilo/install-guide/install/apt/content/keystone-install.html
2) In section [memcache] in /etc/keystone/keystone.conf change servers = line:
 servers = [2001:db8:1000:1:f816:3eff:fe2a:f9c7]:11211,[2001:db8:1000:1:f816:3eff:fee9:9ce3]:11211,[2001:db8:1000:1:f816:3eff:fead:8f7f]:11211
3) Restart keystone/apache
4) Attempt to issue token:
 openstack --os-auth-url http://192.168.0.15:35357 --os-project-name admin --os-username admin --os-auth-type password token issue

ERROR: openstack An unexpected error prevented the server from fulfilling your request: Unable to parse connection string: "[2001:db8:1000:1:f816:3eff:fe2a:f9c7]:11211" (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-869eb953-74af-4336-b3e1-dc3a417180f9)

To reproduce scenario C: hostname that resolves to IPv6-only address
1) Configure keystone according to http://docs.openstack.org/kilo/install-guide/install/apt/content/keystone-install.html
2) In section [memcache] in /etc/keystone/keystone.conf change servers = line:
 servers = keystone-1:11211,keystone-2:11211,keystone-3:11211

3) Edit /etc/hosts:
2001:db8:1000:1:f816:3eff:fe2a:f9c7 keystone-1
2001:db8:1000:1:f816:3eff:fee9:9ce3 keystone-2
2001:db8:1000:1:f816:3eff:fead:8f7f keystone-3

3) Restart keystone/apache
4) Attempt to issue token:

openstack --os-auth-url http://192.168.0.15:35357 --os-project-name admin --os-username admin --os-auth-type password token issue
Password:
ERROR: openstack Maximum lock attempts on _lockusertokens-30dbbe8174b24174a3a24d1ae554ab17 occurred. (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-efd53eae-4bcf-4fd9-bab2-dd4c86fb9798)

Control test:
1) Configure keystone according to http://docs.openstack.org/kilo/install-guide/install/apt/content/keystone-install.html
2) In section [memcache] in /etc/keystone/keystone.conf change servers = line:
 servers = keystone-1:11211,keystone-2:11211,keystone-3:11211

3) Edit /etc/hosts:
192.168.0.15 keystone-1
192.168.0.14 keystone-2
192.168.0.16 keystone-3

3) Restart keystone/apache
4) Attempt to issue token:

openstack --os-auth-url http://192.168.0.15:35357 --os-project-name admin --os-username admin --os-auth-type password token issue
Password:
+------------+----------------------------------+
| Field | Value |
+------------+----------------------------------+
| expires | 2015-06-05T00:31:30Z |
| id | 2a188e9950f44decb78f196b5a3c3f78 |
| project_id | 91bb6f536fca40a68fb5d4cf72527388 |
| user_id | 30dbbe8174b24174a3a24d1ae554ab17 |
+------------+----------------------------------+

[Frode Nordahl (fnordahl)]

A workaround is to change to a more capable memcached library for the token persistence backend, for example pylibmc.

I will propose a patch that at least makes this configurable.

[Dolph Mathews (dolph)]

With Fernet, you don't need any token persistence, and for caching purposes, you can already use pylibmc. I'm leaning towards Won't Fix, unless the solution you're proposing is not particularly risky (I doubt we'll have enough users to safely vet it come Liberty).

[Morgan Fainberg (mdrnstm)]

1) The default token persistence backend is NOT memcached in Kilo. It is still SQL, just the same as before. If the packages you are using are defaulting to memcached and are not working for you, please take this up with the packager - as they are overriding the defaults.

2) As Dolph pointed out pylibmc is usable today in the backend, it is not the default however. Long term we should look at moving to pymemcache or similar.

3) In general, I encourage everyone to move to fernet tokens over persisting tokens in memcached. The memcached persistence driver should probably be removed as assuming that memcached is a good "stable" storage (for persistent data) is a bad choice in general. However, due to the use of the memcached driver, I expect that we will not be able to deprecate it.

[Morgan Fainberg (mdrnstm)]

4) python-memcache is known to be an awful library. It is on the long list of things for us to replace, but at the moment fairly far down the list due to the relatively low intersection of pure IPv6 hosts and those utilizing caching/memcache token persistence.

[Frode Nordahl (fnordahl)]

1) The Kilo Install Guides advices the user to set up memcache for token persistence:
http://docs.openstack.org/kilo/install-guide/install/zypper/content/keystone-install.html
http://docs.openstack.org/kilo/install-guide/install/yum/content/keystone-install.html
http://docs.openstack.org/kilo/install-guide/install/apt/content/keystone-install.html

From a user perspective, what this guide says is perceived as the default or the recommended configuration.

2) I am not setting up the cache, but the token persistence. The code involved has the choice of memcache backend hard coded:
https://github.com/openstack/keystone/blob/master/keystone/token/persistence/backends/memcache.py

Do you suggest using SQL for presistance and set up whatever memcached library I want as cache?

3) Fernet looks really interesting and we are allready looking into migrating to it. Is there a document available that describes best practices for migrating a live running system? We have made some quick attempts in our test enviroment and was partially unsuccessfull, we will look more into it.

4) I understand this, and I do not expect miracles or unicorns appearing in broad daylight. What I do offer though is to roll up my sleves, putch in and provide a patch that might help other people in the same situation. I totally agree that Fernet looks like the end game and the way to go, but our systems has to be up and running with what we have right now until we get there.

[Morgan Fainberg (mdrnstm)]

I highly advise against the memcache backend for tokens in all deployments. The memcache persistence driver is not a good option. SQL is superior in most cases provided a regular flush is performed.

We should get the install guide updated (the memcache driver should never ever be used).

[Morgan Fainberg (mdrnstm)]

I would like to point out that the memcache backend is absolutely not the default driver. SQL is the default driver.

[Morgan Fainberg (mdrnstm)]

https://github.com/openstack/keystone/blob/8125a8913d233f3da0eaacd09aa8e0b794ea98cb/keystone/common/config.py#L253-L255

[Morgan Fainberg (mdrnstm)]

Added to the top of the bug report for openstack-manuals:

OpenStack Manuals (for both Master and Kilo) need to be updated to eliminate the recommendation to use the memcache token persistence backend. The memcache token persistence backend is a poor choice due to performance concerns of the code itself, the fact that it is assumed that the token backend is stable storage (memcached is not) and can expose security concerns if restarted in some scenarios (PKI tokens and revoked tokens becoming valid again), and finally that the python-memcache library is all around poor (it will not work with IPv6 and is not Python3 compatible, among other poor design choices).

We should be recommending the SQL backend (or Fernet) with a cron to flush the tokens (if SQL is used).

[Morgan Fainberg (mdrnstm)]

I am unfortunately unfamilar with docbook formats and am unsure of the best way to update these cleanly.

If the guides were in RST I'd provide a fix at this time.

[Lana (loquacity)]

Hey Morgan, thanks for the info. Luckily for you, the Install Guide is being converted to RST for Liberty ;) In the meantime, I've triaged this, so someone from the Install Guide speciality team can pick it up.

[David Stanek (dstanek)]

Marking as WONTFIX since there isn't any work to be done on Keystone.

[KATO Tomoyuki (kato-tomoyuki)]

Kilo EOL'd

[Jeffrey Guan (double12gzh)]

The same issue met in Mitaka.

DEBUG (connectionpool:395) "POST /v2.0/tokens HTTP/1.1" 500 156
DEBUG (session:466) Request returned failure status: 500
DEBUG (shell:1082) Maximum lock attempts on _lockusertokens-9ba348dba0e84244a7d264b170ba25a2 occurred. (HTTP 500) (Request-ID: req-a3438f72-cbdc-4e1d-a8f0-48963900469b)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 1080, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 914, in main
    api_version = api_versions.discover_version(self.cs, api_version)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 267, in discover_version
    client)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 248, in _get_server_version_range
    version = client.versions.get_current()
  File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 84, in get_current
    return self._get_current()
  File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 56, in _get_current
    url = "%s" % self.api.client.get_endpoint()
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 132, in get_endpoint
    return self.session.get_endpoint(auth or self.auth, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 661, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 210, in get_endpoint
    service_catalog = self.get_access(session).service_catalog
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
    authenticated=False, log=False)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 572, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 94, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 467, in request
    raise exceptions.from_response(resp, method, url)
InternalServerError: Maximum lock attempts on _lockusertokens-9ba348dba0e84244a7d264b170ba25a2 occurred. (HTTP 500) (Request-ID: req-a3438f72-cbdc-4e1d-a8f0-48963900469b)

[Jeffrey Guan (double12gzh)]

The part of configuration in /etc/keystone/keystone.conf is:
...
[token]
...
driver=memcache_pool
...

[Jeffrey Guan (double12gzh)]

[cache]
enabled = True
...
backend = oslo_cache.memcache_pool