CRITICAL glance [-] error: [Errno 24] Too many open files

[James Page]

Its quite easy to hit the nofiles limit with high levels of concurrency and a ceph backend:

2014-10-24 08:29:05.356 22327 CRITICAL glance [-] error: [Errno 24] Too many open files
2014-10-24 08:29:05.356 22327 TRACE glance Traceback (most recent call last):
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/bin/glance-api", line 10, in <module>
2014-10-24 08:29:05.356 22327 TRACE glance sys.exit(main())
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/cmd/api.py", line 83, in main
2014-10-24 08:29:05.356 22327 TRACE glance server.start(config.load_paste_app('glance-api'), default_port=9292)
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 271, in start
2014-10-24 08:29:05.356 22327 TRACE glance self.run_child()
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 321, in run_child
2014-10-24 08:29:05.356 22327 TRACE glance self.run_server()
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 343, in run_server
2014-10-24 08:29:05.356 22327 TRACE glance debug=False)
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 671, in server
2014-10-24 08:29:05.356 22327 TRACE glance client_socket = sock.accept()
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 179, in accept
2014-10-24 08:29:05.356 22327 TRACE glance res = socket_accept(fd)
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 56, in socket_accept
2014-10-24 08:29:05.356 22327 TRACE glance return descriptor.accept()
2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/socket.py", line 202, in accept
2014-10-24 08:29:05.356 22327 TRACE glance sock, addr = self._sock.accept()
2014-10-24 08:29:05.356 22327 TRACE glance error: [Errno 24] Too many open files
2014-10-24 08:29:05.356 22327 TRACE glance
2014-10-24 08:29:05.840 22316 ERROR glance.wsgi.server [-] Not respawning child 22327, cannot recover from termination

Increasing the default nofiles via upstart would be a good solution.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: glance-api 1:2014.2-0ubuntu1~cloud0 [origin: Canonical]
ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6
Uname: Linux 3.13.0-35-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.5
Architecture: amd64
CrashDB:
 {
                "impl": "launchpad",
                "project": "cloud-archive",
                "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
             }
Date: Fri Oct 24 09:08:50 2014
PackageArchitecture: all
SourcePackage: glance
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.glance.glance.api.conf: [deleted]
modified.conffile..etc.glance.glance.api.paste.ini: [deleted]
modified.conffile..etc.glance.glance.cache.conf: [deleted]
modified.conffile..etc.glance.glance.scrubber.conf: [deleted]
modified.conffile..etc.glance.policy.json: [deleted]
modified.conffile..etc.glance.schema.image.json: [deleted]

[James Page (james-page)]

Latest glance packages for Kilo set an appropriate NOFILES limit - marking Fix Released.

[Nobuto Murata (nobuto)]

ah, wonderful!

https://launchpad.net/ubuntu/+source/openstack-pkg-tools/21ubuntu2
> openstack-pkg-tools (21ubuntu2) vivid; urgency=medium
>
> * Add respawn directive to upstart template.
> * Increase default nofile limit to 65535/65535.