Error running keystone-manage token_flush

[Adam Huffman]

I have 1.3 million rows in my keystone.token table, so I'm trying to trim this with the command keystone-manage token_flush. However, this command fails with the following message:

keystone-manage token_flush
/usr/lib/python2.6/site-packages/keystone/openstack/common/db/sqlalchemy/session.py:424: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  m = re.match(operational_error.message)

The real error is the following:

2014-07-17 13:16:19.830 8982 TRACE keystone Traceback (most recent call last):
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/bin/keystone-manage", line 51, in <module>
2014-07-17 13:16:19.830 8982 TRACE keystone cli.main(argv=sys.argv, config_files=config_files)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib/python2.6/site-packages/keystone/cli.py", line 190, in main
2014-07-17 13:16:19.830 8982 TRACE keystone CONF.command.cmd_class.main()
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib/python2.6/site-packages/keystone/cli.py", line 154, in main
2014-07-17 13:16:19.830 8982 TRACE keystone token_manager.driver.flush_expired_tokens()
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib/python2.6/site-packages/keystone/token/backends/sql.py", line 229, in flush_expired_tokens
2014-07-17 13:16:19.830 8982 TRACE keystone query.delete(synchronize_session=False)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2603, in delete
2014-07-17 13:16:19.830 8982 TRACE keystone delete_op.exec_()
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 816, in exec_
2014-07-17 13:16:19.830 8982 TRACE keystone self._do_exec()
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 942, in _do_exec
2014-07-17 13:16:19.830 8982 TRACE keystone params=self.query._params)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib/python2.6/site-packages/keystone/openstack/common/db/sqlalchemy/session.py", line 439, in _wrap
2014-07-17 13:16:19.830 8982 TRACE keystone return f(self, *args, **kwargs)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib/python2.6/site-packages/keystone/openstack/common/db/sqlalchemy/session.py", line 709, in execute
2014-07-17 13:16:19.830 8982 TRACE keystone return super(Session, self).execute(*args, **kwargs)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 934, in execute
2014-07-17 13:16:19.830 8982 TRACE keystone clause, params or {})
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-07-17 13:16:19.830 8982 TRACE keystone params)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-07-17 13:16:19.830 8982 TRACE keystone compiled_sql, distilled_params
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-07-17 13:16:19.830 8982 TRACE keystone context)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
2014-07-17 13:16:19.830 8982 TRACE keystone exc_info
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/compat.py", line 196, in raise_from_cause
2014-07-17 13:16:19.830 8982 TRACE keystone reraise(type(exception), exception, tb=exc_tb)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
2014-07-17 13:16:19.830 8982 TRACE keystone context)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 324, in do_execute
2014-07-17 13:16:19.830 8982 TRACE keystone cursor.execute(statement, parameters)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute
2014-07-17 13:16:19.830 8982 TRACE keystone self.errorhandler(self, exc, value)
2014-07-17 13:16:19.830 8982 TRACE keystone File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-07-17 13:16:19.830 8982 TRACE keystone raise errorclass, errorvalue
2014-07-17 13:16:19.830 8982 TRACE keystone OperationalError: (OperationalError) (1206, 'The total number of locks exceeds the lock table size') 'DELETE FROM token WHERE token.expires < %s' (datetime.datetime(2014, 7, 17, 12, 10, 14, 894903),)
2014-07-17 13:16:19.830 8982 TRACE keystone

It would be helpful if the client reported this error, rather than the unhelpful current error message.

[Morgan Fainberg (mdrnstm)]

This looks like a duplicate (effectively) of bug 1188378 .

Which release of OpenStack are you using? Icehouse? Havana? Earlier?

If you are tracking close to master (Juno), you could try this batch-delete code.

I'm unsure if we can backport this code to earlier releases because it is a significant change in functionality. The best bet (short of the code for bug 1188378 ) is likely to do more frequent cleanups of the token table. Initially you might need to do a manual cleanup.

If you don't mind answering the above questions, it might be solved moving forward and we can evaluate backporting the fix.

[Morgan Fainberg (mdrnstm)]

I just checked, this batch code has existed since Icehouse.

[Launchpad Janitor (janitor)]

[Expired for Keystone because there has been no activity for 60 days.]