Ubuntu Cloud Archive

  1. Bug #1385134
  2. Activity log

Activity log for bug #1385134

Date Who What changed Old value New value Message
2014-10-24 09:10:34 James Page bug added bug
2014-10-24 09:10:52 James Page tags amd64 apport-bug third-party-packages trusty uec-images amd64 apport-bug scale-testing third-party-packages trusty uec-images
2014-10-24 09:11:01 James Page bug task added glance (Ubuntu)
2014-10-24 09:59:20 James Page description Its quite easy to hit the nofiles limit with high levels of concurrency and a ceph backend: 2014-10-24 08:29:05.356 22327 CRITICAL glance [-] error: [Errno 24] Too many open files 2014-10-24 08:29:05.356 22327 TRACE glance Traceback (most recent call last): 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/bin/glance-api", line 10, in <module> 2014-10-24 08:29:05.356 22327 TRACE glance sys.exit(main()) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/cmd/api.py", line 83, in main 2014-10-24 08:29:05.356 22327 TRACE glance server.start(config.load_paste_app('glance-api'), default_port=9292) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 271, in start 2014-10-24 08:29:05.356 22327 TRACE glance self.run_child() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 321, in run_child 2014-10-24 08:29:05.356 22327 TRACE glance self.run_server() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 343, in run_server 2014-10-24 08:29:05.356 22327 TRACE glance debug=False) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 671, in server 2014-10-24 08:29:05.356 22327 TRACE glance client_socket = sock.accept() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 179, in accept 2014-10-24 08:29:05.356 22327 TRACE glance res = socket_accept(fd) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 56, in socket_accept 2014-10-24 08:29:05.356 22327 TRACE glance return descriptor.accept() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/socket.py", line 202, in accept 2014-10-24 08:29:05.356 22327 TRACE glance sock, addr = self._sock.accept() 2014-10-24 08:29:05.356 22327 TRACE glance error: [Errno 24] Too many open files 2014-10-24 08:29:05.356 22327 TRACE glance 2014-10-24 08:29:05.840 22316 ERROR glance.wsgi.server [-] Not respawning child 22327, cannot recover from termination Especially as the api workers run as thread, not separate processes. Increasing the default nofiles via upstart would be a good solution. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: glance-api 1:2014.2-0ubuntu1~cloud0 [origin: Canonical] ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6 Uname: Linux 3.13.0-35-generic x86_64 ApportVersion: 2.14.1-0ubuntu3.5 Architecture: amd64 CrashDB: { "impl": "launchpad", "project": "cloud-archive", "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml", } Date: Fri Oct 24 09:08:50 2014 PackageArchitecture: all SourcePackage: glance UpgradeStatus: No upgrade log present (probably fresh install) modified.conffile..etc.glance.glance.api.conf: [deleted] modified.conffile..etc.glance.glance.api.paste.ini: [deleted] modified.conffile..etc.glance.glance.cache.conf: [deleted] modified.conffile..etc.glance.glance.scrubber.conf: [deleted] modified.conffile..etc.glance.policy.json: [deleted] modified.conffile..etc.glance.schema.image.json: [deleted] Its quite easy to hit the nofiles limit with high levels of concurrency and a ceph backend: 2014-10-24 08:29:05.356 22327 CRITICAL glance [-] error: [Errno 24] Too many open files 2014-10-24 08:29:05.356 22327 TRACE glance Traceback (most recent call last): 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/bin/glance-api", line 10, in <module> 2014-10-24 08:29:05.356 22327 TRACE glance sys.exit(main()) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/cmd/api.py", line 83, in main 2014-10-24 08:29:05.356 22327 TRACE glance server.start(config.load_paste_app('glance-api'), default_port=9292) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 271, in start 2014-10-24 08:29:05.356 22327 TRACE glance self.run_child() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 321, in run_child 2014-10-24 08:29:05.356 22327 TRACE glance self.run_server() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 343, in run_server 2014-10-24 08:29:05.356 22327 TRACE glance debug=False) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 671, in server 2014-10-24 08:29:05.356 22327 TRACE glance client_socket = sock.accept() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 179, in accept 2014-10-24 08:29:05.356 22327 TRACE glance res = socket_accept(fd) 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 56, in socket_accept 2014-10-24 08:29:05.356 22327 TRACE glance return descriptor.accept() 2014-10-24 08:29:05.356 22327 TRACE glance File "/usr/lib/python2.7/socket.py", line 202, in accept 2014-10-24 08:29:05.356 22327 TRACE glance sock, addr = self._sock.accept() 2014-10-24 08:29:05.356 22327 TRACE glance error: [Errno 24] Too many open files 2014-10-24 08:29:05.356 22327 TRACE glance 2014-10-24 08:29:05.840 22316 ERROR glance.wsgi.server [-] Not respawning child 22327, cannot recover from termination Increasing the default nofiles via upstart would be a good solution. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: glance-api 1:2014.2-0ubuntu1~cloud0 [origin: Canonical] ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6 Uname: Linux 3.13.0-35-generic x86_64 ApportVersion: 2.14.1-0ubuntu3.5 Architecture: amd64 CrashDB:  {                 "impl": "launchpad",                 "project": "cloud-archive",                 "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",              } Date: Fri Oct 24 09:08:50 2014 PackageArchitecture: all SourcePackage: glance UpgradeStatus: No upgrade log present (probably fresh install) modified.conffile..etc.glance.glance.api.conf: [deleted] modified.conffile..etc.glance.glance.api.paste.ini: [deleted] modified.conffile..etc.glance.glance.cache.conf: [deleted] modified.conffile..etc.glance.glance.scrubber.conf: [deleted] modified.conffile..etc.glance.policy.json: [deleted] modified.conffile..etc.glance.schema.image.json: [deleted]
2015-01-22 11:52:53 James Page glance (Ubuntu): importance Undecided Medium
2015-01-22 11:52:55 James Page cloud-archive: importance Undecided Medium
2015-01-22 11:53:28 James Page glance (Ubuntu): status New Fix Released
2015-01-22 11:53:31 James Page cloud-archive: status New Fix Released
2015-01-23 04:27:50 Nobuto Murata bug added subscriber Nobuto MURATA
2015-01-23 04:30:04 Nobuto Murata bug task added openstack-pkg-tools (Ubuntu)
2015-01-23 04:30:49 Nobuto Murata openstack-pkg-tools (Ubuntu): status New Fix Released
2015-03-24 07:52:12 Lei Wang bug added subscriber Ray Wang