nova list --all-tenants fetches all instance faults but uses only latest
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Roman Podoliaka |
Bug Description
Description
===========
There are 15 instances. I excute the command: nova --debug list --all-tenants,
But it took more than 40 seconds. I read the nova api code, it sends a Get request
and read the instance_faults table for detail information. The instance_faults table has
more than tens of thousands of records. Each instance has many records.
GET /v2/433288e1244
I think instance_faults table needs to be optimized. A large number of records in the instance_faults
table are useless, only leaving the last three records should be on it, others can be deleted.
There are any other optimization program?
Steps to reproduce
==================
A chronological list of steps which will bring off the
issue you noticed:
* I excute the command: nova --debug list --all-tenants
A list of openstack client commands (with correct argument value)
$ nova --debug list --all-tenants
Expected result
===============
I expect to be back soon within 10 seconds
Actual result
=============
But the query took more than 40 seconds.
Environment
===========
1. version
Mitaka
2. Which hypervisor did you use?
Libvirt + KVM
tags: | added: all-tenants list slow |
Changed in nova: | |
importance: | Undecided → Wishlist |
Changed in nova: | |
assignee: | nobody → Sivasathurappan Radhakrishnan (siva-radhakrishnan) |
tags: |
added: db removed: all-tenants list slow |
Changed in nova: | |
status: | New → Confirmed |
importance: | Wishlist → Medium |
summary: |
- The command "nova list --all-tenants" query is slow + We fetch all instance faults, then throw away all but the last ones on + "nova list --all-tenants", potentinally making the command slow |
summary: |
- We fetch all instance faults, then throw away all but the last ones on - "nova list --all-tenants", potentinally making the command slow + nova list --all-tenants fetches all instance faults but uses only latest |
Changed in nova: | |
assignee: | Sivasathurappan Radhakrishnan (siva-radhakrishnan) → Jay Pipes (jaypipes) |
Changed in nova: | |
assignee: | Jay Pipes (jaypipes) → Roman Podoliaka (rpodolyaka) |
description: | updated |
Hi Tina Kevin!
I tried reproducing the issue but I wasn't able to do so. I just dug into this issue in more detail and below are my findings.
(a) In instance_faults table we are soft deleting the records by setting the column to numeric value which can be later archived.
(b) Even if you have thousand of records in the table, the query selects the records based only deleted flag set
(c) instance_faults has column 'detail' which is of mediumtext format which can possibly slown down the query. Since we just put stack trace in detail column so I assume innodb wouldn't use external blob storage page to store it.
Can you please let me know how did you confirm if this method was the reason for the query being too slow ?
Thanks,
Siva