Nova cache is constantly being re-created, causing 5s delay per item listing.

Bug #1943872 reported by Mirek
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Nova Cloud Controller Charm
New
Undecided
Unassigned

Bug Description

A strange bug in my deployment, when cache is enabled I've noticed each instance created is adding about about 5sec when trying to list them, which lead to Horizon timeout if you get more then 6 instances. e.g.
openstack --timing server list
+--------------------------------------+---------------+--------+------------------------------------------+-------+---------------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------------+--------+------------------------------------------+-------+---------------------+
| 1ec25121-ada7-4c5d-be70-2a0f58dae92d | win2019_test1 | ACTIVE | priv=10.10.100.88; public01=10.10.50.166 | | m1.winserver.medium |
| d1321873-03bc-4f51-a5d1-e1b7f8985819 | win2016_test1 | ACTIVE | priv=10.10.100.77; public01=10.10.50.170 | | m1.winserver.medium |
| 80dd005b-cc2d-4cc9-b23c-49b5997cc014 | win10_test1 | ACTIVE | priv=10.10.100.58; public01=10.10.50.112 | | m1.winserver.medium |
| 9b3068db-bd4d-4cc3-a6ff-38af45994b70 | test-ext-1 | ACTIVE | priv=10.10.100.23; public01=10.10.50.201 | | m1.micro |
+--------------------------------------+---------------+--------+------------------------------------------+-------+---------------------+

+------------------------------------------------------------------+--------------------+
| URL | Seconds |
+------------------------------------------------------------------+--------------------+
| GET https://10.10.40.203:5000/v3 | 0.290954 |
| POST https://10.10.40.203:5000/v3/auth/tokens | 0.671316 |
| POST https://10.10.40.203:5000/v3/auth/tokens | 0.693663 |
| GET https://10.10.40.205:8774/v2.1/servers/detail | 25.016857 |
| GET https://10.10.40.205:8774/v2.1/flavors/detail?is_public=None | 0.088543 |
| Total | 26.761333000000004 |
+------------------------------------------------------------------+--------------------+

I've managed to work around it with config change in nova-cc

[cache]
enabled = false

And I've also traced it down to the following piece of code:
/usr/lib/python3/dist-packages/nova/availability_zones.py:185
 196 cache = _get_cache()
 197 -> az = cache.get(cache_key)

What happens is nova is checking if cache exists and somehow it always returns not and tries to create a cache entry per item, which takes about 5s in my environment hence the timeout in Horizon if I get more than 6 instances. My OpenStack is a 3 Zone dMAAS deployment and then the same 3 zones are created in Nova. Never tried to re-created in single zone deployment so don't know if it's an AZ issue or not. My deployment file is attached.

Tags: cache
Revision history for this message
Mirek (mirek186) wrote :
summary: - nova cache constatnly being re-cretead casuing 5s delya per item
+ Nova cache is constantly being re-created, causing 5s delay per item
listing.
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.