Its quite easy to hit the nofiles limit with high levels of concurrency and a ceph backend:
2014-10-24 08:29:05.356 22327 CRITICAL glance [-] error: [Errno 24] Too many open files 2014-10-24 08:29:05.356 22327 TRACE glance Traceback (most recent call last): 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/bin/glance-api", line 10, in <module> 2014-10-24 08:29:05.356 22327 TRACE glance sys.exit(main()) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/cmd/api.py", line 83, in main 2014-10-24 08:29:05.356 22327 TRACE glance server.start(config.load_paste_app('glance-api'), default_port=9292) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 271, in start 2014-10-24 08:29:05.356 22327 TRACE glance self.run_child() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 321, in run_child 2014-10-24 08:29:05.356 22327 TRACE glance self.run_server() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 343, in run_server 2014-10-24 08:29:05.356 22327 TRACE glance debug=False) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 671, in server 2014-10-24 08:29:05.356 22327 TRACE glance client_socket = sock.accept() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 179, in accept 2014-10-24 08:29:05.356 22327 TRACE glance res = socket_accept(fd) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 56, in socket_accept 2014-10-24 08:29:05.356 22327 TRACE glance return descriptor.accept() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/socket.py", line 202, in accept 2014-10-24 08:29:05.356 22327 TRACE glance sock, addr = self._sock.accept() 2014-10-24 08:29:05.356 22327 TRACE glance error: [Errno 24] Too many open files 2014-10-24 08:29:05.356 22327 TRACE glance 2014-10-24 08:29:05.840 22316 ERROR glance.wsgi.server [-] Not respawning child 22327, cannot recover from termination
Especially as the api workers run as thread, not separate processes.
Increasing the default nofiles via upstart would be a good solution.
ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: glance-api 1:2014.2-0ubuntu1~cloud0 [origin: Canonical] ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6 Uname: Linux 3.13.0-35-generic x86_64 ApportVersion: 2.14.1-0ubuntu3.5 Architecture: amd64 CrashDB: { "impl": "launchpad", "project": "cloud-archive", "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml", } Date: Fri Oct 24 09:08:50 2014 PackageArchitecture: all SourcePackage: glance UpgradeStatus: No upgrade log present (probably fresh install) modified.conffile..etc.glance.glance.api.conf: [deleted] modified.conffile..etc.glance.glance.api.paste.ini: [deleted] modified.conffile..etc.glance.glance.cache.conf: [deleted] modified.conffile..etc.glance.glance.scrubber.conf: [deleted] modified.conffile..etc.glance.policy.json: [deleted] modified.conffile..etc.glance.schema.image.json: [deleted]
Its quite easy to hit the nofiles limit with high levels of concurrency and a ceph backend:
2014-10-24 08:29:05.356 22327 CRITICAL glance [-] error: [Errno 24] Too many open files glance- api", line 10, in <module> python2. 7/dist- packages/ glance/ cmd/api. py", line 83, in main start(config. load_paste_ app('glance- api'), default_port=9292) python2. 7/dist- packages/ glance/ common/ wsgi.py" , line 271, in start python2. 7/dist- packages/ glance/ common/ wsgi.py" , line 321, in run_child python2. 7/dist- packages/ glance/ common/ wsgi.py" , line 343, in run_server python2. 7/dist- packages/ eventlet/ wsgi.py" , line 671, in server python2. 7/dist- packages/ eventlet/ greenio. py", line 179, in accept python2. 7/dist- packages/ eventlet/ greenio. py", line 56, in socket_accept python2. 7/socket. py", line 202, in accept
2014-10-24 08:29:05.356 22327 TRACE glance Traceback (most recent call last):
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/bin/
2014-10-24 08:29:05.356 22327 TRACE glance sys.exit(main())
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/
2014-10-24 08:29:05.356 22327 TRACE glance server.
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/
2014-10-24 08:29:05.356 22327 TRACE glance self.run_child()
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/
2014-10-24 08:29:05.356 22327 TRACE glance self.run_server()
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/
2014-10-24 08:29:05.356 22327 TRACE glance debug=False)
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/
2014-10-24 08:29:05.356 22327 TRACE glance client_socket = sock.accept()
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/
2014-10-24 08:29:05.356 22327 TRACE glance res = socket_accept(fd)
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/
2014-10-24 08:29:05.356 22327 TRACE glance return descriptor.accept()
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/
2014-10-24 08:29:05.356 22327 TRACE glance sock, addr = self._sock.accept()
2014-10-24 08:29:05.356 22327 TRACE glance error: [Errno 24] Too many open files
2014-10-24 08:29:05.356 22327 TRACE glance
2014-10-24 08:29:05.840 22316 ERROR glance.wsgi.server [-] Not respawning child 22327, cannot recover from termination
Especially as the api workers run as thread, not separate processes.
Increasing the default nofiles via upstart would be a good solution.
ProblemType: Bug 2-0ubuntu1~ cloud0 [origin: Canonical] ature: User Name 3.13.0- 35.62-generic 3.13.11.6
"impl" : "launchpad",
"project" : "cloud-archive",
"bug_ pattern_ url": "http:// people. canonical. com/~ubuntu- archive/ bugpatterns/ bugpatterns. xml", ture: all conffile. .etc.glance. glance. api.conf: [deleted] conffile. .etc.glance. glance. api.paste. ini: [deleted] conffile. .etc.glance. glance. cache.conf: [deleted] conffile. .etc.glance. glance. scrubber. conf: [deleted] conffile. .etc.glance. policy. json: [deleted] conffile. .etc.glance. schema. image.json: [deleted]
DistroRelease: Ubuntu 14.04
Package: glance-api 1:2014.
ProcVersionSign
Uname: Linux 3.13.0-35-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
CrashDB:
{
}
Date: Fri Oct 24 09:08:50 2014
PackageArchitec
SourcePackage: glance
UpgradeStatus: No upgrade log present (probably fresh install)
modified.
modified.
modified.
modified.
modified.
modified.