Comment 2 for bug 1822631

Revision history for this message
Gerry Kopec (gerry-kopec) wrote :

Yes, this is a duplicate of https://bugs.launchpad.net/starlingx/+bug/1822116

Looking at system in question:
nova-compute on compute-1 is stuck and one of the placements pods is not responsive.

controller-1 placement logs:
2019-04-01 09:55:50.793763 Traceback (most recent call last):
2019-04-01 09:55:50.793802 File "/var/lib/openstack/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 460, in fire_timers
2019-04-01 09:55:50.793995 timer()
2019-04-01 09:55:50.794004 File "/var/lib/openstack/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 59, in __call__
2019-04-01 09:55:50.794051 cb(*args, **kw)
2019-04-01 09:55:50.794060 File "/var/lib/openstack/lib/python2.7/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
2019-04-01 09:55:50.794136 waiter.switch()
2019-04-01 09:55:50.794150 error: cannot switch to a different thread

nova-compute-1 logs, shows delete of VM but no other activity:
</capabilities>

2019-04-01 09:55:44,471.471 47950 INFO nova.compute.manager [req-879c43a8-a86f-46fc-a2fa-e2f36d6ca94b - - - - -] [instance: 6cbfe64e-1b50-457e-bee9-544f997b1e16] Deleting instance as it has been evacuated from this host
2019-04-01 09:55:45,663.663 47950 INFO nova.virt.libvirt.driver [-] [instance: 6cbfe64e-1b50-457e-bee9-544f997b1e16] Instance destroyed successfully.
2019-04-01 09:55:50,348.348 47950 INFO nova.virt.libvirt.driver [req-879c43a8-a86f-46fc-a2fa-e2f36d6ca94b - - - - -] [instance: 6cbfe64e-1b50-457e-bee9-544f997b1e16] Deleting instance files /var/lib/nova/instances/6cbfe64e-1b50-457e-bee9-544f997b1e16_del
2019-04-01 09:55:50,350.350 47950 INFO nova.virt.libvirt.driver [req-879c43a8-a86f-46fc-a2fa-e2f36d6ca94b - - - - -] [instance: 6cbfe64e-1b50-457e-bee9-544f997b1e16] Deletion of /var/lib/nova/instances/6cbfe64e-1b50-457e-bee9-544f997b1e16_del complete

Resource tracker not running on compute-1 since time of reboot:[wrsroot@controller-1 ~(keystone_admin)]$ date; kubectl exec -it -n openstack mariadb-server-0 -- bash -c "mysql --password=\$MYSQL_ROOT_PASSWORD --user=root nova -e 'select host,updated_at,vcpus_used from compute_nodes;'"
Mon Apr 1 18:10:08 UTC 2019
+-----------+---------------------+------------+
| host | updated_at | vcpus_used |
+-----------+---------------------+------------+
| compute-2 | 2019-04-01 15:31:02 | 0 |
| compute-0 | 2019-04-01 09:52:40 | 2 |
| compute-1 | 2019-04-01 09:49:56 | 4 |
+-----------+---------------------+------------+