rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default)
Bug #1153077 reported by
Mark Stillwell
This bug affects 4 people
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Expired
|
High
|
Unassigned | ||
maas (Ubuntu) |
Expired
|
High
|
Unassigned |
Bug Description
With the default MAAS install I was having a problem where celery would become unresponsive and stop running new tasks (including powering compute nodes on/off) after a while, and eventually the system would have out of memory errors. The problem seems to have been that the queue was filling up with dhcp_upload_leases tasks (I checked on this by getting tasks out with the amqp tools...). This problem goes away if the timedelta in /usr/share/
Related branches
lp:~allenap/maas/dhcp-leases-parsing
- Raphaël Badin (community): Approve
-
Diff: 773 lines (+407/-212)2 files modifiedsrc/provisioningserver/dhcp/leases_parser_fast.py (+87/-0)
src/provisioningserver/dhcp/tests/test_leases_parser.py (+320/-212)
lp:~allenap/maas/dhcp-leases-parsing-wire-up
- Gavin Panella (community): Approve
-
Diff: 12 lines (+1/-1)1 file modifiedsrc/provisioningserver/dhcp/leases.py (+1/-1)
lp:~allenap/maas/dhcp-leases-parsing--1.5
- Gavin Panella (community): Approve
-
Diff: 773 lines (+407/-212)2 files modifiedsrc/provisioningserver/dhcp/leases_parser_fast.py (+87/-0)
src/provisioningserver/dhcp/tests/test_leases_parser.py (+320/-212)
lp:~allenap/maas/dhcp-leases-parsing-wire-up--1.5
- Gavin Panella (community): Approve
-
Diff: 12 lines (+1/-1)1 file modifiedsrc/provisioningserver/dhcp/leases.py (+1/-1)
summary: |
- rabbitmq queue fills up and salary stops executing tasks when + rabbitmq queue fills up and celery stops executing tasks when upload_dhcp_leases is done every minute (by default) |
Changed in maas (Ubuntu): | |
importance: | Undecided → Critical |
Changed in maas: | |
importance: | Undecided → High |
Changed in maas (Ubuntu): | |
importance: | Critical → High |
To post a comment you must log in.
As an update, the problem does not go away actually, it just takes longer to develop. So, for some reason a task is getting hung or something, causing all tasks behind it to not be executed and filling the queue.