Keystone running out of memory

[David McNally]

After running keystone for a few days I see it running out of memory with the following error in the logs:

2014-08-06 07:45:37.647 4873 ERROR keystone.common.wsgi [-] [Errno 12] Cannot allocate memory
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi Traceback (most recent call last):
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi File "/opt/stack/venvs/keystone/local/lib/python2.7/site-packages/keystone/common/wsgi.py", line 212, in __call__
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi result = method(context, **params)
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi File "/opt/stack/venvs/keystone/local/lib/python2.7/site-packages/keystone/token/controllers.py", line 143, in authenticate
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi auth_token_data, roles_ref=roles_ref, catalog_ref=catalog_ref)
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi File "/opt/stack/venvs/keystone/local/lib/python2.7/site-packages/keystone/token/providers/common.py", line 399, in issue_v2_token
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi token_id = self._get_token_id(token_data)
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi File "/opt/stack/venvs/keystone/local/lib/python2.7/site-packages/keystone/token/providers/pkiz.py", line 41, in _get_token_id
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi CONF.signing.keyfile))
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi File "/opt/stack/venvs/keystone/local/lib/python2.7/site-packages/keystoneclient/common/cms.py", line 198, in pkiz_sign
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi PKIZ_CMS_FORM)
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi File "/opt/stack/venvs/keystone/local/lib/python2.7/site-packages/keystoneclient/common/cms.py", line 338, in cms_sign_data
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi stderr=subprocess.PIPE)
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi File "/opt/stack/venvs/keystone/local/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 45, in __init__
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi errread, errwrite)
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi File "/usr/lib/python2.7/subprocess.py", line 1223, in _execute_child
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi self.pid = os.fork()
2014-08-06 07:45:37.647 4873 TRACE keystone.common.wsgi OSError: [Errno 12] Cannot allocate memory

[Dolph Mathews (dolph)]

Are you using the KVS token backend (or any other KVS drivers using providing unmanaged in-process persistence)?

[David Stanek (dstanek)]

I'd also like to know how much memory you have on the node running Keystone and how much memory the Keystone process is consuming.

[Dawei Ding (dading)]

Same issue has been spotted in our production environment. also FD leaking were seen which stacked up to 15k FD used by keystone process.

[Dolph Mathews (dolph)]

Dawei: Regarding file descriptors, I suspect you're running into bug 1382906. I'd recommend switching from PKI to UUID tokens to avoid the problem as an immediate workaround.

[Dawei Ding (dading)]

Dolph, thanks for the reference. Yes I believe it was the cause.
I am actually thinking switching to UUID but still evaluating the performance impact on token verification.
Or patching the client may be another option.

[Launchpad Janitor (janitor)]

[Expired for Keystone because there has been no activity for 60 days.]