Yes, this is a duplicate of https://bugs.launchpad.net/starlingx/+bug/1822116
Looking at system in question: nova-compute on compute-1 is stuck and one of the placements pods is not responsive.
controller-1 placement logs: 2019-04-01 09:55:50.793763 Traceback (most recent call last): 2019-04-01 09:55:50.793802 File "/var/lib/openstack/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 460, in fire_timers 2019-04-01 09:55:50.793995 timer() 2019-04-01 09:55:50.794004 File "/var/lib/openstack/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 59, in __call__ 2019-04-01 09:55:50.794051 cb(*args, **kw) 2019-04-01 09:55:50.794060 File "/var/lib/openstack/lib/python2.7/site-packages/eventlet/semaphore.py", line 147, in _do_acquire 2019-04-01 09:55:50.794136 waiter.switch() 2019-04-01 09:55:50.794150 error: cannot switch to a different thread
nova-compute-1 logs, shows delete of VM but no other activity: </capabilities>
2019-04-01 09:55:44,471.471 47950 INFO nova.compute.manager [req-879c43a8-a86f-46fc-a2fa-e2f36d6ca94b - - - - -] [instance: 6cbfe64e-1b50-457e-bee9-544f997b1e16] Deleting instance as it has been evacuated from this host 2019-04-01 09:55:45,663.663 47950 INFO nova.virt.libvirt.driver [-] [instance: 6cbfe64e-1b50-457e-bee9-544f997b1e16] Instance destroyed successfully. 2019-04-01 09:55:50,348.348 47950 INFO nova.virt.libvirt.driver [req-879c43a8-a86f-46fc-a2fa-e2f36d6ca94b - - - - -] [instance: 6cbfe64e-1b50-457e-bee9-544f997b1e16] Deleting instance files /var/lib/nova/instances/6cbfe64e-1b50-457e-bee9-544f997b1e16_del 2019-04-01 09:55:50,350.350 47950 INFO nova.virt.libvirt.driver [req-879c43a8-a86f-46fc-a2fa-e2f36d6ca94b - - - - -] [instance: 6cbfe64e-1b50-457e-bee9-544f997b1e16] Deletion of /var/lib/nova/instances/6cbfe64e-1b50-457e-bee9-544f997b1e16_del complete
Resource tracker not running on compute-1 since time of reboot:[wrsroot@controller-1 ~(keystone_admin)]$ date; kubectl exec -it -n openstack mariadb-server-0 -- bash -c "mysql --password=\$MYSQL_ROOT_PASSWORD --user=root nova -e 'select host,updated_at,vcpus_used from compute_nodes;'" Mon Apr 1 18:10:08 UTC 2019 +-----------+---------------------+------------+ | host | updated_at | vcpus_used | +-----------+---------------------+------------+ | compute-2 | 2019-04-01 15:31:02 | 0 | | compute-0 | 2019-04-01 09:52:40 | 2 | | compute-1 | 2019-04-01 09:49:56 | 4 | +-----------+---------------------+------------+
Yes, this is a duplicate of https:/ /bugs.launchpad .net/starlingx/ +bug/1822116
Looking at system in question:
nova-compute on compute-1 is stuck and one of the placements pods is not responsive.
controller-1 placement logs: openstack/ lib/python2. 7/site- packages/ eventlet/ hubs/hub. py", line 460, in fire_timers openstack/ lib/python2. 7/site- packages/ eventlet/ hubs/timer. py", line 59, in __call__ openstack/ lib/python2. 7/site- packages/ eventlet/ semaphore. py", line 147, in _do_acquire
2019-04-01 09:55:50.793763 Traceback (most recent call last):
2019-04-01 09:55:50.793802 File "/var/lib/
2019-04-01 09:55:50.793995 timer()
2019-04-01 09:55:50.794004 File "/var/lib/
2019-04-01 09:55:50.794051 cb(*args, **kw)
2019-04-01 09:55:50.794060 File "/var/lib/
2019-04-01 09:55:50.794136 waiter.switch()
2019-04-01 09:55:50.794150 error: cannot switch to a different thread
nova-compute-1 logs, shows delete of VM but no other activity:
</capabilities>
2019-04-01 09:55:44,471.471 47950 INFO nova.compute. manager [req-879c43a8- a86f-46fc- a2fa-e2f36d6ca9 4b - - - - -] [instance: 6cbfe64e- 1b50-457e- bee9-544f997b1e 16] Deleting instance as it has been evacuated from this host libvirt. driver [-] [instance: 6cbfe64e- 1b50-457e- bee9-544f997b1e 16] Instance destroyed successfully. libvirt. driver [req-879c43a8- a86f-46fc- a2fa-e2f36d6ca9 4b - - - - -] [instance: 6cbfe64e- 1b50-457e- bee9-544f997b1e 16] Deleting instance files /var/lib/ nova/instances/ 6cbfe64e- 1b50-457e- bee9-544f997b1e 16_del libvirt. driver [req-879c43a8- a86f-46fc- a2fa-e2f36d6ca9 4b - - - - -] [instance: 6cbfe64e- 1b50-457e- bee9-544f997b1e 16] Deletion of /var/lib/ nova/instances/ 6cbfe64e- 1b50-457e- bee9-544f997b1e 16_del complete
2019-04-01 09:55:45,663.663 47950 INFO nova.virt.
2019-04-01 09:55:50,348.348 47950 INFO nova.virt.
2019-04-01 09:55:50,350.350 47950 INFO nova.virt.
Resource tracker not running on compute-1 since time of reboot: [wrsroot@ controller- 1 ~(keystone_admin)]$ date; kubectl exec -it -n openstack mariadb-server-0 -- bash -c "mysql --password= \$MYSQL_ ROOT_PASSWORD --user=root nova -e 'select host,updated_ at,vcpus_ used from compute_nodes;'" -----+- ------- ------- ------+ ------- -----+ -----+- ------- ------- ------+ ------- -----+ -----+- ------- ------- ------+ ------- -----+
Mon Apr 1 18:10:08 UTC 2019
+------
| host | updated_at | vcpus_used |
+------
| compute-2 | 2019-04-01 15:31:02 | 0 |
| compute-0 | 2019-04-01 09:52:40 | 2 |
| compute-1 | 2019-04-01 09:49:56 | 4 |
+------