Keystonemiddleware crashes when memcache encryption is enabled with Swift

[Eamonn O'Toole]

We've come across the following issue when deploying standalone Swift servers using TripleO, where we've enabled auth token memcache with encryption. We get this error from the Swift proxy:

Nov 11 15:17:49 overcloud-swiftstorage1-ohdtremvbiw3 proxy-server: Error: An error occurred: #012Traceback (most recent call last):#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/swift/common/middlewar
e/catch_errors.py", line 41, in handle_request#012 resp = self._app_call(env)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/swift/common/wsgi.py", line 582, in _app_call#012 resp = self.app(env,
 self._start_response)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/swift/common/middleware/gatekeeper.py", line 90, in __call__#012 return self.app(env, gatekeeper_response)#012 File "/opt/stack
/venvs/openstack/local/lib/python2.7/site-packages/swift/common/middleware/healthcheck.py", line 57, in __call__#012 return self.app(env, start_response)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packag
es/swift/common/middleware/proxy_logging.py", line 289, in __call__#012 iterable = self.app(env, my_start_response)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/swift/common/middleware/memcache.py
", line 85, in __call__#012 return self.app(env, start_response)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/swift/common/middleware/crossdomain.py", line 82, in __call__#012 return self.app(e
nv, start_response)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/swift/common/middleware/tempurl.py", line 295, in __call__#012 return self.app(env, start_response)#012 File "/opt/stack/venvs/ope
nstack/local/lib/python2.7/site-packages/swift/common/middleware/formpost.py", line 231, in __call__#012 return self.app(env, start_response)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/keystonem
iddleware/auth_token.py", line 710, in __call__#012 token_info = self._validate_token(user_token, env)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/keystonemiddleware/auth_token.py", line 891, in
_validate_token#012 self._token_cache.store_invalid(token_id)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/keystonemiddleware/auth_token.py", line 1714, in store_invalid#012 self._cache_store(t
oken_id, self._INVALID_INDICATOR)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/keystonemiddleware/auth_token.py", line 1822, in _cache_store#012 data_to_store = memcache_crypt.protect_data(keys, s
erialized_data)#012 File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/keystonemiddleware/_memcache_crypt.py", line 166, in protect_data#012 data = encrypt_data(keys['ENCRYPTION'], data)#012 File "/opt/sta
ck/venvs/openstack/local/lib/python2.7/site-packages/keystonemiddleware/_memcache_crypt.py", line 80, in wrapper#012 raise CryptoUnavailableError()#012CryptoUnavailableError (txn: tx9bf0c765e603404e8a776-0054622899)

Looking in the _memcache_crypt.py code the problem is that pycrypto isn't installed in the Swift venv. pycrypto isn't listed in the Keystonemiddleware requirements.txt file. Since memcache encryption in Keystone middleware relies on pycrypto, and to avoid this issue where the Swift proxy errors out, we believe that pyrcypto should be added to Keystonemiddleware's requirements.txt file.

--- (morganfainberg): This is a documentation bug, see my comment #2

[Morgan Fainberg (mdrnstm)]

This does not affect Keystone but middleware. Changed the target.

[Morgan Fainberg (mdrnstm)]

I've looked into this and this is the same category as the python-memecache library is not in requirements. This is likely a documentation bug to clearly communicate that pycrypto is needed in this case. Unfortunately, this is because pypi and pip are limited in their packaging, there isn't really a concept of a "optional" or "recommended" package (such as APT would communicate).

At this point we won't be adding pycrypto or memcache lib to the requirements.txt as it is not needed in most deployments. If pycrypto was needed for all deployments or was used outside of the very specific memcached use-case adding it to the requirements.txt would be more appropriate.

An alternative is we could provide another requirements.txt (requirements-memcache.txt) that at least communicates that this is needed. This doesn't really "solve" anything that improved documentation wouldn't also solve.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/134993

[OpenStack Infra (hudson-openstack)]

Change abandoned by Eamonn O'Toole (<email address hidden>) on branch: master
Review: https://review.openstack.org/134161
Reason: Abandoning - will submit a TripleO fix instead.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/134993
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=c00c98209d1af763a62441e1048708ddc6f1b388
Submitter: Jenkins
Branch: master

commit c00c98209d1af763a62441e1048708ddc6f1b388
Author: Rodrigo Duarte Sousa <email address hidden>
Date: Mon Nov 17 13:13:18 2014 -0300

    Adds Memcached dependencies doc

    Since Memcached is not used by the majority of deployments, its
    dependencies are not included by default in the requirements.txt
    file. This patch adds the documentation about the need to
    manually install those dependencies.

    Change-Id: Ic6252b1e00168fa2236f8a892212084da6cfdd64
    Closes-Bug: 1392264