Unexpectedly high number of DB queries when hitting /MAAS/api/2.0/machines/
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Invalid
|
High
|
Unassigned |
Bug Description
While working on MAAS integration with a LXD related project, we've noticed that a simple:
GET /MAAS/api/
Over your REST API is taking upwards of 15s to get us a response.
Looking at MAAS, we can see a regiond process running at 100% of CPU usage during the entire time.
Then looking at traffic on the loopback device, I noticed that it's doing over 1Mb/s of postgresql traffic during that entire time.
Enabling query logging in postgresql and extracting the table accessed for that one API query, I'm getting:
1976 "maasserver_
933 "maasserver_
846 "maasserver_
414 "maasserver_
164 "maasserver_vlan"
103 "maasserver_
94 "maasserver_node"
80 "maasserver_
72 "maasserver_
69 "maasserver_
63 "maasserver_
56 "maasserver_
53 "maasserver_
52 "maasserver_
48 maasserver_node
48 "metadataserver
48 "maasserver_event"
42 "maasserver_
40 "maasserver_config"
36 "maasserver_
23 "maasserver_
12 "maasserver_fabric"
11 "maasserver_subnet"
11 "maasserver_space"
11 "maasserver_
4 "piston3_consumer"
4 "maasserver_
4 "maasserver_
3 "auth_user"
2 "piston3_token"
2 "piston3_nonce"
2 "maasserver_
2 "maasserver_bmc"
1 "maasserver_
1 "maasserver_
1 "maasserver_tag"
1 "maasserver_
1 "maasserver_
1 "maasserver_
1 "maasserver_
1 "maasserver_domain"
1 "maasserver_
1 "maasserver_
1 "maasserver_
This is the number of QUERIES hitting a each table by looking at the FROM in each logged query.
It means that just getting the list of machines caused around 5340 SQL queries!
The MAAS in question only has around 50 machines in 4 resource pools, so it's not exactly massive.
As an additional datapoint, MAAS is also hitting "get_machine_
Tracing that get_machine_
This is mAAS 3.1.0 from 3.1/stable snap. With one region daemon and 4 rack daemons.
Related branches
- MAAS Lander: Approve
- Jack Lloyd-Walters: Approve
-
Diff: 13 lines (+1/-1)1 file modifiedsrc/provisioningserver/prometheus/metrics.py (+1/-1)
Changed in maas: | |
status: | Incomplete → New |
status: | New → Incomplete |
summary: |
- Insane amount of DB queries when hitting /MAAS/api/2.0/machines/ + Unexpectedly high number of DB queries when hitting + /MAAS/api/2.0/machines/ |
Hi there, this is something we are/were aware of and addressed in 3.2.0, which has an RC out as of right now. Can you try with the release candidate?