MMM2 Angel potentially fills filesystem on infinit error condition
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
mysql-mmm |
Fix Released
|
Low
|
Pascal Hofmann |
Bug Description
System: RHEL 5.3 64bit, MMM 2.0.10
When attempting to run mmmd_agent I get the following error:
2009/11/03 16:01:10 FATAL Listener: Can't create socket!
2009/11/03 16:01:10 FATAL Child exited with exitcode 99, restarting
The problem is that Angel loops indefinitely, regardless of the error and it doesn't stop to take a breath... so in a situation like this that is infinite, Angel will eventually fill the filesystem with logs and emails to root.
Tue Nov 3 16:01:10 CST 2009
-------
36K /var/log/mysql-mmm/
1.9M /var/spool/
2009/11/03 16:01:10 FATAL Child exited with exitcode 99, restarting
2009/11/03 16:01:10 FATAL Listener: Can't create socket!
2009/11/03 16:01:10 FATAL Child exited with exitcode 99, restarting
2009/11/03 16:01:10 FATAL Listener: Can't create socket!
2009/11/03 16:01:10 FATAL Child exited with exitcode 99, restarting
2009/11/03 16:01:10 FATAL Listener: Can't create socket!
2009/11/03 16:01:10 FATAL Child exited with exitcode 99, restarting
2009/11/03 16:01:10 FATAL Listener: Can't create socket!
2009/11/03 16:01:10 FATAL Child exited with exitcode 99, restarting
2009/11/03 16:01:10 FATAL Listener: Can't create socket!
Tue Nov 3 16:04:11 CST 2009
-------
1.4M /var/log/mysql-mmm/
235M /var/spool/
2009/11/03 16:04:10 FATAL Child exited with exitcode 99, restarting
2009/11/03 16:04:10 FATAL Listener: Can't create socket!
2009/11/03 16:04:10 FATAL Child exited with exitcode 99, restarting
2009/11/03 16:04:10 FATAL Listener: Can't create socket!
2009/11/03 16:04:10 FATAL Child exited with exitcode 99, restarting
2009/11/03 16:04:10 FATAL Listener: Can't create socket!
2009/11/03 16:04:11 FATAL Child exited with exitcode 99, restarting
2009/11/03 16:04:11 FATAL Listener: Can't create socket!
2009/11/03 16:04:11 FATAL Child exited with exitcode 99, restarting
2009/11/03 16:04:11 FATAL Listener: Can't create socket!
A simple suggestion would be to have Angel sleep on each loop iteration, if even for just 1 second it will slow down the effect of an infinite error state. What I think would possibly be better is have a failure/
Related branches
- Pascal Hofmann: Approve
-
Diff: 60 lines (+21/-2)1 file modifiedlib/Common/Angel.pm (+21/-2)
Changed in mysql-mmm: | |
assignee: | nobody → Pascal Hofmann (pascalhofmann) |
importance: | Undecided → Low |
status: | New → Confirmed |
Changed in mysql-mmm: | |
status: | Fix Committed → Fix Released |
I am adding a suggested patch that will sleep 10 seconds on failure, and after 10 consecutive failed attempts shutsdown Angel.