| 2014-04-09 14:29:03 |
Blake Rouse |
bug |
|
|
added bug |
| 2014-04-09 14:31:23 |
Blake Rouse |
description |
maas-cluster-celery service eats CPU time, as one or more celeryd processes consumes 100% of the cpu. This looks to be due to the dhcpd.lease file being very large, as the time it takes to complete the upload_dhcp_leases file is excessive.
[2014-04-09 14:05:28,405: INFO/MainProcess] Task provisioningserver.tasks.upload_dhcp_leases[f8470913-fff7-49a6-a651-0e2b45901cbb] succeeded in 147.969564829s: None
ubuntu@maas-trusty:~$ ls -lh /var/lib/maas/dhcp
total 18M
-rw-r--r-- 1 root root 17M Apr 9 14:06 dhcpd.leases
-rw-r--r-- 1 root root 366K Apr 3 19:39 dhcpd.leases~
This was noticed in an environment that is running 4 parallel deployments with 3 to 4 LXC containers per deployment in a constant loop, 24/7. |
maas-cluster-celery service eats CPU time, as one or more celeryd processes consumes 100% of the cpu. This looks to be due to the dhcpd.lease file being very large, as the time it takes to complete the upload_dhcp_leases file is excessive.
[2014-04-09 14:05:28,405: INFO/MainProcess] Task provisioningserver.tasks.upload_dhcp_leases[f8470913-fff7-49a6-a651-0e2b45901cbb] succeeded in 147.969564829s: None
ubuntu@maas-trusty:~$ ls -lh /var/lib/maas/dhcp
total 18M
-rw-r--r-- 1 root root 17M Apr 9 14:06 dhcpd.leases
-rw-r--r-- 1 root root 366K Apr 3 19:39 dhcpd.leases~
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
34283 maas 20 0 1646232 1.464g 1896 R 99.9 18.9 12:15.58 celeryd
34284 maas 20 0 1970464 1.773g 1896 R 99.9 22.9 14:49.22 celeryd
34259 maas 20 0 140572 17980 1912 R 99.5 0.2 3:32.11 celeryd
36722 maas 20 0 1592204 1.414g 2444 R 99.5 18.2 4:25.09 celeryd
37773 maas 20 0 591952 482116 2380 R 99.2 5.9 0:43.97 celeryd
This was noticed in an environment that is running 4 parallel deployments with 3 to 4 LXC containers per deployment in a constant loop, 24/7. |
|
| 2014-04-10 20:00:05 |
Gavin Panella |
branch linked |
|
lp:~allenap/maas/dhcp-leases-parsing |
|
| 2014-04-30 20:34:58 |
Gavin Panella |
maas: status |
Triaged |
In Progress |
|
| 2014-04-30 20:35:01 |
Gavin Panella |
maas: assignee |
|
Gavin Panella (allenap) |
|
| 2014-05-08 03:28:26 |
Julian Edwards |
nominated for series |
|
maas/1.5 |
|
| 2014-05-08 03:28:26 |
Julian Edwards |
bug task added |
|
maas/1.5 |
|
| 2014-05-08 03:28:33 |
Julian Edwards |
maas/1.5: status |
New |
Triaged |
|
| 2014-05-08 03:28:35 |
Julian Edwards |
maas/1.5: importance |
Undecided |
Critical |
|
| 2014-05-08 03:28:37 |
Julian Edwards |
maas/1.5: milestone |
|
1.5.1 |
|
| 2014-05-09 02:08:20 |
Julian Edwards |
maas/1.5: milestone |
1.5.1 |
|
|
| 2014-05-19 15:06:37 |
Gavin Panella |
branch linked |
|
lp:~allenap/maas/dhcp-leases-parsing-wire-up |
|
| 2014-05-19 15:15:50 |
Gavin Panella |
branch linked |
|
lp:~allenap/maas/dhcp-leases-parsing--1.5 |
|
| 2014-05-19 15:18:01 |
Gavin Panella |
nominated for series |
|
maas/trunk |
|
| 2014-05-19 15:18:01 |
Gavin Panella |
bug task added |
|
maas/trunk |
|
| 2014-05-19 15:18:17 |
Gavin Panella |
maas/1.5: status |
Triaged |
In Progress |
|
| 2014-05-19 15:18:20 |
Gavin Panella |
maas/1.5: assignee |
|
Gavin Panella (allenap) |
|
| 2014-05-19 16:43:56 |
Gavin Panella |
branch linked |
|
lp:~allenap/maas/dhcp-leases-parsing-wire-up--1.5 |
|
| 2014-05-19 16:44:48 |
Gavin Panella |
maas/trunk: status |
In Progress |
Fix Committed |
|
| 2014-05-19 16:44:54 |
Gavin Panella |
maas/trunk: milestone |
|
1.6.0 |
|
| 2014-05-23 00:59:25 |
Julian Edwards |
maas/1.5: milestone |
|
1.5.2 |
|
| 2014-05-23 00:59:28 |
Julian Edwards |
maas/1.5: status |
In Progress |
Fix Committed |
|
| 2014-05-26 02:37:37 |
Julian Edwards |
maas/trunk: milestone |
1.6.0 |
|
|
| 2014-05-26 02:44:44 |
Julian Edwards |
maas/trunk: status |
Fix Committed |
Fix Released |
|
| 2014-05-26 02:44:46 |
Julian Edwards |
maas/1.5: status |
Fix Committed |
Fix Released |
|
| 2014-06-02 18:58:26 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/utopic-proposed/maas |
|
| 2014-06-06 17:15:44 |
Greg Lutostanski |
description |
maas-cluster-celery service eats CPU time, as one or more celeryd processes consumes 100% of the cpu. This looks to be due to the dhcpd.lease file being very large, as the time it takes to complete the upload_dhcp_leases file is excessive.
[2014-04-09 14:05:28,405: INFO/MainProcess] Task provisioningserver.tasks.upload_dhcp_leases[f8470913-fff7-49a6-a651-0e2b45901cbb] succeeded in 147.969564829s: None
ubuntu@maas-trusty:~$ ls -lh /var/lib/maas/dhcp
total 18M
-rw-r--r-- 1 root root 17M Apr 9 14:06 dhcpd.leases
-rw-r--r-- 1 root root 366K Apr 3 19:39 dhcpd.leases~
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
34283 maas 20 0 1646232 1.464g 1896 R 99.9 18.9 12:15.58 celeryd
34284 maas 20 0 1970464 1.773g 1896 R 99.9 22.9 14:49.22 celeryd
34259 maas 20 0 140572 17980 1912 R 99.5 0.2 3:32.11 celeryd
36722 maas 20 0 1592204 1.414g 2444 R 99.5 18.2 4:25.09 celeryd
37773 maas 20 0 591952 482116 2380 R 99.2 5.9 0:43.97 celeryd
This was noticed in an environment that is running 4 parallel deployments with 3 to 4 LXC containers per deployment in a constant loop, 24/7. |
maas-cluster-celery service eats CPU time, as one or more celeryd processes consumes 100% of the cpu. This looks to be due to the dhcpd.lease file being very large, as the time it takes to complete the upload_dhcp_leases file is excessive.
[2014-04-09 14:05:28,405: INFO/MainProcess] Task provisioningserver.tasks.upload_dhcp_leases[f8470913-fff7-49a6-a651-0e2b45901cbb] succeeded in 147.969564829s: None
ubuntu@maas-trusty:~$ ls -lh /var/lib/maas/dhcp
total 18M
-rw-r--r-- 1 root root 17M Apr 9 14:06 dhcpd.leases
-rw-r--r-- 1 root root 366K Apr 3 19:39 dhcpd.leases~
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
34283 maas 20 0 1646232 1.464g 1896 R 99.9 18.9 12:15.58 celeryd
34284 maas 20 0 1970464 1.773g 1896 R 99.9 22.9 14:49.22 celeryd
34259 maas 20 0 140572 17980 1912 R 99.5 0.2 3:32.11 celeryd
36722 maas 20 0 1592204 1.414g 2444 R 99.5 18.2 4:25.09 celeryd
37773 maas 20 0 591952 482116 2380 R 99.2 5.9 0:43.97 celeryd
This was noticed in an environment that is running 4 parallel deployments with 3 to 4 LXC containers per deployment in a constant loop, 24/7.
[Impact]
* More efficient lease parser to work on larger deployments -- previously would be CPU bound in these cases and sometimes causing timeouts
[Testcase]
* Install maas; ensure new lease parser works by detecting nodes ip addresses correctly for the maas managed dhcp provider
[Regression Potential]
* Large at first blush, but this has had lots of testing; both from unittests as well as maasCI and Openstack Interoperability Lab |
|
| 2014-06-06 19:54:26 |
Andres Rodriguez |
bug task added |
|
maas (Ubuntu) |
|
| 2014-06-06 19:54:33 |
Andres Rodriguez |
nominated for series |
|
Ubuntu Trusty |
|
| 2014-06-06 19:54:33 |
Andres Rodriguez |
bug task added |
|
maas (Ubuntu Trusty) |
|
| 2014-06-06 19:54:39 |
Andres Rodriguez |
maas (Ubuntu): status |
New |
Fix Released |
|
| 2014-06-07 03:07:54 |
Adam Conrad |
maas (Ubuntu Trusty): status |
New |
Fix Committed |
|
| 2014-06-07 03:07:57 |
Adam Conrad |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2014-06-07 03:08:01 |
Adam Conrad |
bug |
|
|
added subscriber SRU Verification |
| 2014-06-07 03:08:04 |
Adam Conrad |
tags |
|
verification-needed |
|
| 2014-06-07 03:33:34 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/trusty-proposed/maas |
|
| 2014-06-09 16:06:41 |
Jason Hobbs |
tags |
verification-needed |
verification-done |
|
| 2014-06-20 18:20:17 |
Stéphane Graber |
tags |
verification-done |
|
|
| 2014-06-20 18:20:18 |
Stéphane Graber |
tags |
|
verification-needed |
|
| 2014-07-01 18:47:43 |
Blake Rouse |
tags |
verification-needed |
verification-done |
|
| 2014-07-03 16:05:26 |
Launchpad Janitor |
maas (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2014-07-03 16:06:11 |
Stéphane Graber |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
