MAAS slow performance + growing database
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Fix Released
|
High
|
Unassigned |
Bug Description
On a long running installation I'm seeing performance problems when commissioning nodes.
The installation has 130+ nodes and is running MAAS 2.3.5 (6511-gf466fdb-
When commissioning 4 nodes the web interface will become unresponsive (taking a node overview page to render 15s+), and a "maas <profile> node read xxx" taking up to 9s instead of 2s for the idle case.
One thing I'm noticing is a lot of "maas@maasdb ERROR: could not serialize access due to concurrent update" lines in postgres log.
Looking at the database seems that some tables in postgres have become quite big, esp. the maasserver_event table plus index:
maasdb=# SELECT nspname || '.' || relname AS "relation",
maasdb-# pg_size_
maasdb-# FROM pg_class C
maasdb-# LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
maasdb-# WHERE nspname NOT IN ('pg_catalog', 'information_
maasdb-# ORDER BY pg_relation_
maasdb-# LIMIT 20;
-------
public.
public.
public.
public.
public.
public.
maasdb=# SELECT *, pg_size_
, pg_size_
, pg_size_
, pg_size_
FROM (
SELECT *, total_bytes-
SELECT c.oid,nspname AS table_schema, relname AS TABLE_NAME
, c.reltuples AS row_estimate
, pg_total_
, pg_indexes_
, pg_total_
FROM pg_class c
LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE relkind = 'r' AND relname like 'maasserver_event%'
) a
) a;
oid | table_schema | table_name | row_estimate | total_bytes | index_bytes | toast_bytes | table_bytes | total | index | toast | table
-------
19125 | public | maasserver_event | 1.64169e+07 | 5180358656 | 2587885568 | 8192 | 2592464896 | 4940 MB | 2468 MB | 8192 bytes | 2472 MB
19136 | public | maasserver_
From inspecting that table it seems that this table keeps all events since initial installation. Also, that table defines quite a few btree indexes, which could also impact performance on updates/inserts.
Do you think this could explain the sluggishness when commissioning? Should those tables be trimmed as part of a regular maintenance?
Related branches
- Blake Rouse (community): Approve
-
Diff: 992 lines (+325/-152)16 files modifiedsrc/maasserver/third_party_drivers.py (+3/-2)
src/maasserver/websockets/base.py (+8/-4)
src/maasserver/websockets/handlers/controller.py (+17/-8)
src/maasserver/websockets/handlers/device.py (+15/-7)
src/maasserver/websockets/handlers/event.py (+1/-1)
src/maasserver/websockets/handlers/machine.py (+36/-30)
src/maasserver/websockets/handlers/node.py (+69/-50)
src/maasserver/websockets/handlers/notification.py (+2/-2)
src/maasserver/websockets/handlers/sshkey.py (+1/-1)
src/maasserver/websockets/handlers/switch.py (+7/-3)
src/maasserver/websockets/handlers/tests/test_controller.py (+69/-1)
src/maasserver/websockets/handlers/tests/test_device.py (+19/-4)
src/maasserver/websockets/handlers/tests/test_machine.py (+54/-15)
src/maasserver/websockets/handlers/tests/test_switch.py (+1/-20)
src/maasserver/websockets/handlers/user.py (+3/-3)
src/maasserver/websockets/tests/test_base.py (+20/-1)
- Blake Rouse (community): Approve
-
Diff: 415 lines (+189/-40)9 files modifiedsrc/maasserver/static/js/angular/controllers/node_results.js (+21/-10)
src/maasserver/static/js/angular/controllers/tests/test_node_results.js (+46/-0)
src/maasserver/static/js/angular/factories/node_results.js (+10/-1)
src/maasserver/static/js/angular/factories/tests/test_node_results.js (+22/-1)
src/maasserver/static/partials/script-results-list.html (+7/-1)
src/maasserver/websockets/handlers/node_result.py (+21/-12)
src/maasserver/websockets/handlers/tests/test_node_result.py (+32/-11)
src/metadataserver/models/scriptresult.py (+4/-2)
src/metadataserver/models/tests/test_scriptresult.py (+26/-2)
- Alberto Donato (community): Approve
-
Diff: 117 lines (+69/-3)2 files modifiedsrc/maasserver/websockets/handlers/node.py (+17/-3)
src/maasserver/websockets/handlers/tests/test_machine.py (+52/-0)
- Blake Rouse (community): Approve
-
Diff: 358 lines (+95/-54)6 files modifiedsrc/maasserver/websockets/base.py (+8/-6)
src/maasserver/websockets/handlers/device.py (+9/-1)
src/maasserver/websockets/handlers/machine.py (+0/-11)
src/maasserver/websockets/handlers/node.py (+30/-25)
src/maasserver/websockets/handlers/tests/test_device.py (+1/-1)
src/maasserver/websockets/handlers/tests/test_machine.py (+47/-10)
- Blake Rouse (community): Approve
-
Diff: 949 lines (+436/-308)7 files modifiedsrc/maasserver/static/js/angular/controllers/node_details.js (+0/-125)
src/maasserver/static/js/angular/controllers/node_results.js (+141/-11)
src/maasserver/static/js/angular/controllers/tests/test_node_details.js (+1/-145)
src/maasserver/static/js/angular/controllers/tests/test_node_results.js (+230/-0)
src/maasserver/static/js/angular/factories/nodes.js (+13/-1)
src/maasserver/static/js/angular/factories/tests/test_nodes.js (+32/-0)
src/maasserver/static/partials/node-details.html (+19/-26)
- Blake Rouse (community): Approve
- Alberto Donato (community): Approve
-
Diff: 512 lines (+104/-236)5 files modifiedsrc/maasserver/websockets/handlers/controller.py (+3/-1)
src/maasserver/websockets/handlers/machine.py (+3/-1)
src/maasserver/websockets/handlers/node.py (+53/-93)
src/maasserver/websockets/handlers/switch.py (+4/-1)
src/maasserver/websockets/handlers/tests/test_machine.py (+41/-140)
Changed in maas: | |
status: | New → In Progress |
importance: | Undecided → High |
assignee: | nobody → Lee Trager (ltrager) |
milestone: | none → 2.3.6 |
Changed in maas: | |
status: | In Progress → Fix Committed |
Changed in maas: | |
assignee: | Lee Trager (ltrager) → nobody |
I'm uploading logs here (sorry Canonical only): /private- fileshare. canonical. com/~sabaini/ lp1830365/
https:/
Also the above queries with slightly less horrible formatting: /paste. ubuntu. com/p/nSB5kPv3n 3/
https:/