This looks like our locking is broken:
$ grep -i 'lock "compute_resources"' n-cpu.log | egrep -e 'req-e892fc24-0651-4c08-be7c-551c1ecc7bc7|req-e1b62bbc-91d8-46ca-bba4-451391c6b1a1' 2022-09-01T13:52:02.144032+0000 devstack0a nova-compute[79776]: DEBUG oslo_concurrency.lockutils [None req-e892fc24-0651-4c08-be7c-551c1ecc7bc7 admin admin] Acquiring lock "compute_resources" by "nova.compute.resource_tracker.ResourceTracker.rebuild_claim" {{(pid=79776) inner /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:385}} 2022-09-01T13:52:02.144786+0000 devstack0a nova-compute[79776]: DEBUG oslo_concurrency.lockutils [None req-e892fc24-0651-4c08-be7c-551c1ecc7bc7 admin admin] Lock "compute_resources" acquired by "nova.compute.resource_tracker.ResourceTracker.rebuild_claim" :: waited 0.001s {{(pid=79776) inner /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:390}} 2022-09-01T13:52:02.147508+0000 devstack0a nova-compute[79776]: DEBUG oslo_concurrency.lockutils [None req-e1b62bbc-91d8-46ca-bba4-451391c6b1a1 admin admin] Acquiring lock "compute_resources" by "nova.compute.resource_tracker.ResourceTracker.rebuild_claim" {{(pid=79776) inner /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:385}} 2022-09-01T13:52:02.148136+0000 devstack0a nova-compute[79776]: DEBUG oslo_concurrency.lockutils [None req-e1b62bbc-91d8-46ca-bba4-451391c6b1a1 admin admin] Lock "compute_resources" acquired by "nova.compute.resource_tracker.ResourceTracker.rebuild_claim" :: waited 0.001s {{(pid=79776) inner /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:390}} 2022-09-01T13:52:02.388504+0000 devstack0a nova-compute[79776]: DEBUG oslo_concurrency.lockutils [None req-e1b62bbc-91d8-46ca-bba4-451391c6b1a1 admin admin] Lock "compute_resources" "released" by "nova.compute.resource_tracker.ResourceTracker.rebuild_claim" :: held 0.240s {{(pid=79776) inner /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:404}} 2022-09-01T13:52:02.455376+0000 devstack0a nova-compute[79776]: DEBUG oslo_concurrency.lockutils [None req-e892fc24-0651-4c08-be7c-551c1ecc7bc7 admin admin] Lock "compute_resources" "released" by "nova.compute.resource_tracker.ResourceTracker.rebuild_claim" :: held 0.311s {{(pid=79776) inner /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:404}} 2022-09-01T13:52:07.640135+0000 devstack0a nova-compute[79776]: DEBUG oslo_concurrency.lockutils [None req-e892fc24-0651-4c08-be7c-551c1ecc7bc7 admin admin] Acquiring lock "compute_resources" by "nova.compute.resource_tracker.ResourceTracker.finish_evacuation" {{(pid=79776) inner /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:385}} 2022-09-01T13:52:07.640764+0000 devstack0a nova-compute[79776]: DEBUG oslo_concurrency.lockutils [None req-e892fc24-0651-4c08-be7c-551c1ecc7bc7 admin admin] Lock "compute_resources" acquired by "nova.compute.resource_tracker.ResourceTracker.finish_evacuation" :: waited 0.001s {{(pid=79776) inner /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:390}} 2022-09-01T13:52:08.053607+0000 devstack0a nova-compute[79776]: DEBUG oslo_concurrency.lockutils [None req-e892fc24-0651-4c08-be7c-551c1ecc7bc7 admin admin] Lock "compute_resources" "released" by "nova.compute.resource_tracker.ResourceTracker.finish_evacuation" :: held 0.413s {{(pid=79776) inner /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:404}}
This looks like our locking is broken:
$ grep -i 'lock "compute_ resources" ' n-cpu.log | egrep -e 'req-e892fc24- 0651-4c08- be7c-551c1ecc7b c7|req- e1b62bbc- 91d8-46ca- bba4-451391c6b1 a1' 01T13:52: 02.144032+ 0000 devstack0a nova-compute[ 79776]: DEBUG oslo_concurrenc y.lockutils [None req-e892fc24- 0651-4c08- be7c-551c1ecc7b c7 admin admin] Acquiring lock "compute_resources" by "nova.compute. resource_ tracker. ResourceTracker .rebuild_ claim" {{(pid=79776) inner /usr/local/ lib/python3. 10/dist- packages/ oslo_concurrenc y/lockutils. py:385} } 01T13:52: 02.144786+ 0000 devstack0a nova-compute[ 79776]: DEBUG oslo_concurrenc y.lockutils [None req-e892fc24- 0651-4c08- be7c-551c1ecc7b c7 admin admin] Lock "compute_resources" acquired by "nova.compute. resource_ tracker. ResourceTracker .rebuild_ claim" :: waited 0.001s {{(pid=79776) inner /usr/local/ lib/python3. 10/dist- packages/ oslo_concurrenc y/lockutils. py:390} } 01T13:52: 02.147508+ 0000 devstack0a nova-compute[ 79776]: DEBUG oslo_concurrenc y.lockutils [None req-e1b62bbc- 91d8-46ca- bba4-451391c6b1 a1 admin admin] Acquiring lock "compute_resources" by "nova.compute. resource_ tracker. ResourceTracker .rebuild_ claim" {{(pid=79776) inner /usr/local/ lib/python3. 10/dist- packages/ oslo_concurrenc y/lockutils. py:385} } 01T13:52: 02.148136+ 0000 devstack0a nova-compute[ 79776]: DEBUG oslo_concurrenc y.lockutils [None req-e1b62bbc- 91d8-46ca- bba4-451391c6b1 a1 admin admin] Lock "compute_resources" acquired by "nova.compute. resource_ tracker. ResourceTracker .rebuild_ claim" :: waited 0.001s {{(pid=79776) inner /usr/local/ lib/python3. 10/dist- packages/ oslo_concurrenc y/lockutils. py:390} } 01T13:52: 02.388504+ 0000 devstack0a nova-compute[ 79776]: DEBUG oslo_concurrenc y.lockutils [None req-e1b62bbc- 91d8-46ca- bba4-451391c6b1 a1 admin admin] Lock "compute_resources" "released" by "nova.compute. resource_ tracker. ResourceTracker .rebuild_ claim" :: held 0.240s {{(pid=79776) inner /usr/local/ lib/python3. 10/dist- packages/ oslo_concurrenc y/lockutils. py:404} } 01T13:52: 02.455376+ 0000 devstack0a nova-compute[ 79776]: DEBUG oslo_concurrenc y.lockutils [None req-e892fc24- 0651-4c08- be7c-551c1ecc7b c7 admin admin] Lock "compute_resources" "released" by "nova.compute. resource_ tracker. ResourceTracker .rebuild_ claim" :: held 0.311s {{(pid=79776) inner /usr/local/ lib/python3. 10/dist- packages/ oslo_concurrenc y/lockutils. py:404} } 01T13:52: 07.640135+ 0000 devstack0a nova-compute[ 79776]: DEBUG oslo_concurrenc y.lockutils [None req-e892fc24- 0651-4c08- be7c-551c1ecc7b c7 admin admin] Acquiring lock "compute_resources" by "nova.compute. resource_ tracker. ResourceTracker .finish_ evacuation" {{(pid=79776) inner /usr/local/ lib/python3. 10/dist- packages/ oslo_concurrenc y/lockutils. py:385} } 01T13:52: 07.640764+ 0000 devstack0a nova-compute[ 79776]: DEBUG oslo_concurrenc y.lockutils [None req-e892fc24- 0651-4c08- be7c-551c1ecc7b c7 admin admin] Lock "compute_resources" acquired by "nova.compute. resource_ tracker. ResourceTracker .finish_ evacuation" :: waited 0.001s {{(pid=79776) inner /usr/local/ lib/python3. 10/dist- packages/ oslo_concurrenc y/lockutils. py:390} } 01T13:52: 08.053607+ 0000 devstack0a nova-compute[ 79776]: DEBUG oslo_concurrenc y.lockutils [None req-e892fc24- 0651-4c08- be7c-551c1ecc7b c7 admin admin] Lock "compute_resources" "released" by "nova.compute. resource_ tracker. ResourceTracker .finish_ evacuation" :: held 0.413s {{(pid=79776) inner /usr/local/ lib/python3. 10/dist- packages/ oslo_concurrenc y/lockutils. py:404} }
2022-09-
2022-09-
2022-09-
2022-09-
2022-09-
2022-09-
2022-09-
2022-09-
2022-09-