Twisted Unhandled Error in Deferred - Too many open files: /var/log/
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
Critical
|
Gavin Panella | ||
1.7 |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
From pserv.log, there are 660227 such errors during a short period of time: ~ 1 hour and 34 minutes.
=======
ubuntu@
680227
=======
There are also 386375 oops files.
They start at
2014-12-26 21:35:17+0000 [-] Unhandled error in Deferred
and end at
2014-12-26 22:09:40+0000 [-] Unhandled error in Deferred:
This is one entry:
=======
2014-12-26 21:49:59+0000 [-] Unhandled error in Deferred:
2014-12-26 21:49:59+0000 [-] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
--- <exception caught here> ---
File "/usr/lib/
result = context.call(ctx, function, *args, **kwargs)
File "/usr/lib/
return self.currentCon
File "/usr/lib/
return func(*args,**kw)
File "/usr/lib/
return publisher(report)
File "/usr/lib/
=======
There were also 18 occurrences on 12/21 and 36 oops files.
Around time of these Unhandled errors, we saw deployment related failures such as this one:
=======
Launching instance
WARNING picked arbitrary tools &{1.20.
ERROR bootstrap failed: cannot start bootstrap instance: gomaasapi: got error back from server: 500 INTERNAL SERVER ERROR (u'ipmi')
Bootstrap failed, destroying environment
ERROR Bootstrap failed, and the environment could not be destroyed: gomaasapi: got error back from server: 503 SERVICE UNAVAILABLE (Unable to connect to cluster 'OIL Cluster' (037c960b-
ERROR cannot start bootstrap instance: gomaasapi: got error back from server: 500 INTERNAL SERVER ERROR (u'ipmi')
=======
This is for maas: 1.7.1~rc1+
Related branches
- Blake Rouse (community): Approve
-
Diff: 397 lines (+11/-201)12 files modifieddocs/development/cluster-bootstrap.rst (+1/-1)
etc/maas/pserv.yaml (+0/-11)
required-packages/base (+1/-5)
setup.py (+0/-3)
src/maas/settings.py (+0/-3)
src/maasserver/management/commands/runserver.py (+0/-45)
src/maasserver/tests/test_runserver.py (+0/-38)
src/provisioningserver/config.py (+6/-1)
src/provisioningserver/plugin.py (+1/-54)
src/provisioningserver/tests/test_config.py (+0/-2)
src/provisioningserver/tests/test_plugin.py (+1/-1)
src/provisioningserver/tests/test_plugin_services.py (+1/-37)
- Gavin Panella (community): Approve
-
Diff: 397 lines (+11/-201)12 files modifieddocs/development/cluster-bootstrap.rst (+1/-1)
etc/maas/pserv.yaml (+0/-11)
required-packages/base (+1/-5)
setup.py (+0/-3)
src/maas/settings.py (+0/-3)
src/maasserver/management/commands/runserver.py (+0/-45)
src/maasserver/tests/test_runserver.py (+0/-38)
src/provisioningserver/config.py (+6/-1)
src/provisioningserver/plugin.py (+1/-54)
src/provisioningserver/tests/test_config.py (+0/-2)
src/provisioningserver/tests/test_plugin.py (+1/-1)
src/provisioningserver/tests/test_plugin_services.py (+1/-37)
description: | updated |
tags: | added: oil |
Changed in maas: | |
milestone: | none → 1.7.2 |
Changed in maas: | |
milestone: | 1.7.2 → next |
Changed in maas: | |
milestone: | 1.7.2 → next |
Changed in maas: | |
milestone: | next → none |
> There are also 386375 oops files.
The OOPS system is an extra moving part that doesn't really add anything. We need to get rid of it. The logs should go in MAAS' normal logs.
Now, it seems your installation has a problem so bad it's creating a crazy amount of OOPS reports and this, in turn, becomes a problem ("Too many open files"). Could you please try seeing if you could get more information about the *original* problem? I think you will be able to find more information in the OOPS files (the ones that could be written before the "Too many open files" problem started happening). OOPS files are bson encoded files in /var/log/ maas/oops/ <day>/; usually, `strings <oops-file>` is all it takes to get a good idea about what the problem is.