object auditor crashed
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Fix Released
|
Undecided
|
David Goetz |
Bug Description
I have been running a swift cluster with 3 storage nodes for many days without incident. Then nagios informed me that the object-auditor process was not running on any of them. Right before the time of the report I saw the following two error messages in the log, each repeated twice before the process stopped. The same thing happened on all three storage nodes at about the same time referencing the same /src/node/... file in each case. I am running this version:
ii swift-object 1.4.2-0ubuntu0~
Log messages:
Aug 29 17:40:10 jc2 object-auditor Begin object audit "forever" mode (ZBF)
Aug 29 17:40:10 jc2 object-auditor ERROR Trying to audit /srv/node/
Aug 29 17:40:40 jc2 object-auditor UNCAUGHT EXCEPTION#
Related branches
- John Dickinson: Approve
- gholt (community): Approve
-
Diff: 93 lines (+47/-24)2 files modifiedswift/obj/auditor.py (+27/-24)
test/unit/obj/test_auditor.py (+20/-0)
Changed in swift: | |
assignee: | nobody → David Goetz (david-goetz) |
Changed in swift: | |
milestone: | none → 1.4.3 |
status: | New → Fix Committed |
status: | Fix Committed → Fix Released |
I should add that there were no other errors in the log and that there had not been any activity of adding/removing files, accounts or containers for several days before this happened. The system was "idling" when it happened.