I found this while investigating bug 1162845, but discovered a different symptom. These two bugs may be dupes.
Rapidly generating PKI tokens on v3 eventually results in a stream of "OSError: Too many open files"
I've found two workarounds:
1. Revert to token_format = UUID in keystone.conf
1. Revert to using v2 auth instead of v3 auth
$ ulimit -n
64
$ cat auth-v3.json
{"auth": {"identity": {"methods": ["password"], "password": {"user": {"name": "admin", "password": "admin", "domain": {"id": "default"}}}}, "scope": {"project": {"name": "admin", "domain": {"id": "default"}}}}}
$ ab -n 1000 -c 10 -p auth-v3.json -T application/json http://localhost:5000/v3/auth/tokens
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
apr_socket_recv: Connection reset by peer (54)
Total of 201 requests complete
Backtrace from keystone's logs:
ERROR [keystone.auth.controllers] [Errno 24] Too many open files
Traceback (most recent call last):
File "/Users/dolph/Projects/keystone/keystone/auth/controllers.py", line 290, in authenticate_for_token
context, auth_context, auth_info)
File "/Users/dolph/Projects/keystone/keystone/auth/token_factory.py", line 304, in create_token
trust)
File "/Users/dolph/Projects/keystone/keystone/auth/token_factory.py", line 211, in get_token_data
self._populate_scope(token_data, domain_id, project_id)
File "/Users/dolph/Projects/keystone/keystone/auth/token_factory.py", line 63, in _populate_scope
self.context, project_id)
File "/Users/dolph/Projects/keystone/keystone/common/manager.py", line 47, in _wrapper
File "/Users/dolph/Projects/keystone/keystone/identity/backends/sql.py", line 208, in get_project
File "/Users/dolph/Projects/keystone/keystone/common/sql/core.py", line 219, in get_session
File "/Users/dolph/Projects/keystone/keystone/common/sql/core.py", line 248, in get_engine
File "/Users/dolph/Projects/keystone/keystone/common/sql/core.py", line 246, in new_engine
File "/Users/dolph/Environments/os/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 338, in create_engine
File "/Users/dolph/Environments/os/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 72, in create
File "/Users/dolph/Environments/os/lib/python2.7/site-packages/sqlalchemy/dialects/sqlite/pysqlite.py", line 309, in create_connect_args
File "/Users/dolph/Environments/os/bin/../lib/python2.7/posixpath.py", line 347, in abspath
OSError: [Errno 24] Too many open files
This issue appears to be unreproducible after switching to the sql connection to mysql.