In testrun: https://solutions.qa.canonical.com/testruns/testRun/194ee801-04bf-4ed9-83cf-ac2e19669bfc
Yoga builds correctly but fails on tempest: https://oil-jenkins.canonical.com/artifacts/194ee801-04bf-4ed9-83cf-ac2e19669bfc/generated/generated/openstack/tempest_result.html
In this test, tempest stop and starts a server, but it does not come back up.
In the nova logs we see: -------------------------------------------- 2022-04-12 20:48:26.281 1820614 INFO nova.virt.libvirt.driver [-] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Instance destroyed successfully. 2022-04-12 20:48:27.310 1820614 WARNING nova.compute.manager [req-577b95c6-b3ec-49eb-8d63-b40f9bb7e5ce a6a714097fbd450fa917189d6e89b073 - - dda5fe657938420a947ac87de5ae0b29 dda5fe657938420a947ac87de5ae0b29] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Received unexpected event network-vif-unplugged-ee0dab67-3fd9-4943-a70a-b53a1edafffc for instance with vm_state stopped and task_state None. 2022-04-12 20:48:30.655 1820614 WARNING nova.compute.manager [req-6bce62da-4634-471e-a504-13bbe3270d61 a6a714097fbd450fa917189d6e89b073 - - dda5fe657938420a947ac87de5ae0b29 dda5fe657938420a947ac87de5ae0b29] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Received unexpected event network-vif-plugged-ee0dab67-3fd9-4943-a70a-b53a1edafffc for instance with vm_state stopped and task_state None. 2022-04-12 20:48:33.254 1820614 INFO nova.virt.libvirt.driver [-] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Instance destroyed successfully. 2022-04-12 20:48:34.588 1820614 WARNING nova.compute.manager [req-f6586bba-4012-4eaa-b000-1a56e22c45a0 a6a714097fbd450fa917189d6e89b073 - - dda5fe657938420a947ac87de5ae0b29 dda5fe657938420a947ac87de5ae0b29] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Received unexpected event network-vif-plugged-ee0dab67-3fd9-4943-a70a-b53a1edafffc for instance with vm_state stopped and task_state powering-on. 2022-04-12 20:48:36.722 1820614 WARNING nova.compute.manager [req-c8345295-08bd-4e32-a210-8ab99c7f25b9 a6a714097fbd450fa917189d6e89b073 - - dda5fe657938420a947ac87de5ae0b29 dda5fe657938420a947ac87de5ae0b29] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Received unexpected event network-vif-plugged-ee0dab67-3fd9-4943-a70a-b53a1edafffc for instance with vm_state stopped and task_state powering-on. 2022-04-12 20:48:40.986 1820614 INFO nova.compute.manager [-] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] VM Stopped (Lifecycle Event) 2022-04-12 20:49:04.836 1820614 INFO nova.compute.manager [-] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] During sync_power_state the instance has a pending task (powering-on). Skip. <uuid>79ae643f-c915-4557-82f8-f17da1c44755</uuid> <entry name='serial'>79ae643f-c915-4557-82f8-f17da1c44755</entry> <entry name='uuid'>79ae643f-c915-4557-82f8-f17da1c44755</entry> <source file='/var/lib/nova/instances/79ae643f-c915-4557-82f8-f17da1c44755/disk'/> <log file='/var/lib/nova/instances/79ae643f-c915-4557-82f8-f17da1c44755/console.log' append='off'/> <log file='/var/lib/nova/instances/79ae643f-c915-4557-82f8-f17da1c44755/console.log' append='off'/> 2022-04-12 20:49:04.847 1820614 ERROR nova.virt.libvirt.driver [req-cbfa0bed-3e3e-4402-a071-76c9da1f5661 c3202ca4eb774fa8a0f357564a4dd572 - - f6ca11d407354d8f9af85c0b58e31019 f6ca11d407354d8f9af85c0b58e31019] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Failed to start libvirt guest: libvirt.libvirtError: error from service: Timeout was reached 2022-04-12 20:49:04.974 1820614 INFO nova.compute.manager [req-abaa98f3-840b-4e5f-8833-a18043823319 - - - - -] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] During sync_power_state the instance has a pending task (powering-on). Skip. 2022-04-12 20:49:05.079 1820614 INFO nova.compute.manager [req-cbfa0bed-3e3e-4402-a071-76c9da1f5661 c3202ca4eb774fa8a0f357564a4dd572 - - f6ca11d407354d8f9af85c0b58e31019 f6ca11d407354d8f9af85c0b58e31019] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Successfully reverted task state from powering-on on failure for instance. 2022-04-12 20:49:05.785 1820614 WARNING nova.compute.manager [req-13b96ce5-9969-49fb-a5e1-73bd06190227 a6a714097fbd450fa917189d6e89b073 - - dda5fe657938420a947ac87de5ae0b29 dda5fe657938420a947ac87de5ae0b29] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Received unexpected event network-vif-unplugged-ee0dab67-3fd9-4943-a70a-b53a1edafffc for instance with vm_state stopped and task_state None. 2022-04-12 20:49:08.229 1820614 WARNING nova.compute.manager [req-24e1c96f-04bd-46be-9ce1-5468a0bf2d7e a6a714097fbd450fa917189d6e89b073 - - dda5fe657938420a947ac87de5ae0b29 dda5fe657938420a947ac87de5ae0b29] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Received unexpected event network-vif-plugged-ee0dab67-3fd9-4943-a70a-b53a1edafffc for instance with vm_state stopped and task_state None. 2022-04-12 20:59:06.499 1820614 INFO nova.compute.manager [-] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] During _sync_instance_power_state the DB power_state (4) does not match the vm_power_state from the hypervisor (0). Updating power_state in the DB to match the hypervisor. 2022-04-12 21:08:32.088 1820614 INFO nova.compute.manager [req-11d57952-8002-450d-9122-2739af9e25dd c3202ca4eb774fa8a0f357564a4dd572 - - f6ca11d407354d8f9af85c0b58e31019 f6ca11d407354d8f9af85c0b58e31019] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Terminating instance 2022-04-12 21:08:32.094 1820614 INFO nova.virt.libvirt.driver [-] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Instance destroyed successfully. 2022-04-12 21:08:32.148 1820614 INFO nova.virt.libvirt.driver [req-11d57952-8002-450d-9122-2739af9e25dd c3202ca4eb774fa8a0f357564a4dd572 - - f6ca11d407354d8f9af85c0b58e31019 f6ca11d407354d8f9af85c0b58e31019] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Deleting instance files /var/lib/nova/instances/79ae643f-c915-4557-82f8-f17da1c44755_del 2022-04-12 21:08:32.148 1820614 INFO nova.virt.libvirt.driver [req-11d57952-8002-450d-9122-2739af9e25dd c3202ca4eb774fa8a0f357564a4dd572 - - f6ca11d407354d8f9af85c0b58e31019 f6ca11d407354d8f9af85c0b58e31019] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Deletion of /var/lib/nova/instances/79ae643f-c915-4557-82f8-f17da1c44755_del complete 2022-04-12 21:08:32.231 1820614 INFO nova.compute.manager [req-11d57952-8002-450d-9122-2739af9e25dd c3202ca4eb774fa8a0f357564a4dd572 - - f6ca11d407354d8f9af85c0b58e31019 f6ca11d407354d8f9af85c0b58e31019] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Took 0.14 seconds to destroy the instance on the hypervisor. 2022-04-12 21:08:34.605 1820614 INFO nova.compute.manager [req-3f77554c-6cbf-48ed-9598-8854d7687267 a6a714097fbd450fa917189d6e89b073 - - dda5fe657938420a947ac87de5ae0b29 dda5fe657938420a947ac87de5ae0b29] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Neutron deleted interface ee0dab67-3fd9-4943-a70a-b53a1edafffc; detaching it from the instance and deleting it from the info cache 2022-04-12 21:08:34.641 1820614 INFO nova.compute.manager [-] [instance: 79ae643f-c915-4557-82f8-f17da1c44755] Took 2.41 seconds to deallocate network for instance. 2022-04-12 21:08:34.910 1820614 INFO nova.scheduler.client.report [req-11d57952-8002-450d-9122-2739af9e25dd c3202ca4eb774fa8a0f357564a4dd572 - - f6ca11d407354d8f9af85c0b58e31019 f6ca11d407354d8f9af85c0b58e31019] Deleted allocations for instance 79ae643f-c915-4557-82f8-f17da1c44755 --------------------------------------------
So there seem to be some network problems preventing the server from starting again.
Reproducers: Deploy the bundle https://oil-jenkins.canonical.com/artifacts/194ee801-04bf-4ed9-83cf-ac2e19669bfc/generated/generated/openstack/bundle.yaml run the test tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_stop_start_server[id-af8eafd4-38a7-4a4b-bdbc-75145a580560]
Link to crashdumps etc.: https://oil-jenkins.canonical.com/artifacts/194ee801-04bf-4ed9-83cf-ac2e19669bfc/index.html
In testrun: /solutions. qa.canonical. com/testruns/ testRun/ 194ee801- 04bf-4ed9- 83cf-ac2e19669b fc
https:/
Yoga builds correctly but fails on tempest: /oil-jenkins. canonical. com/artifacts/ 194ee801- 04bf-4ed9- 83cf-ac2e19669b fc/generated/ generated/ openstack/ tempest_ result. html
https:/
In this test, tempest stop and starts a server, but it does not come back up.
In the nova logs we see: ------- ------- ------- ------- ------- -- libvirt. driver [-] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Instance destroyed successfully. manager [req-577b95c6- b3ec-49eb- 8d63-b40f9bb7e5 ce a6a714097fbd450 fa917189d6e89b0 73 - - dda5fe657938420 a947ac87de5ae0b 29 dda5fe657938420 a947ac87de5ae0b 29] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Received unexpected event network- vif-unplugged- ee0dab67- 3fd9-4943- a70a-b53a1edaff fc for instance with vm_state stopped and task_state None. manager [req-6bce62da- 4634-471e- a504-13bbe3270d 61 a6a714097fbd450 fa917189d6e89b0 73 - - dda5fe657938420 a947ac87de5ae0b 29 dda5fe657938420 a947ac87de5ae0b 29] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Received unexpected event network- vif-plugged- ee0dab67- 3fd9-4943- a70a-b53a1edaff fc for instance with vm_state stopped and task_state None. libvirt. driver [-] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Instance destroyed successfully. manager [req-f6586bba- 4012-4eaa- b000-1a56e22c45 a0 a6a714097fbd450 fa917189d6e89b0 73 - - dda5fe657938420 a947ac87de5ae0b 29 dda5fe657938420 a947ac87de5ae0b 29] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Received unexpected event network- vif-plugged- ee0dab67- 3fd9-4943- a70a-b53a1edaff fc for instance with vm_state stopped and task_state powering-on. manager [req-c8345295- 08bd-4e32- a210-8ab99c7f25 b9 a6a714097fbd450 fa917189d6e89b0 73 - - dda5fe657938420 a947ac87de5ae0b 29 dda5fe657938420 a947ac87de5ae0b 29] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Received unexpected event network- vif-plugged- ee0dab67- 3fd9-4943- a70a-b53a1edaff fc for instance with vm_state stopped and task_state powering-on. manager [-] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] VM Stopped (Lifecycle Event) manager [-] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] During sync_power_state the instance has a pending task (powering-on). Skip. 79ae643f- c915-4557- 82f8-f17da1c447 55</uuid> >79ae643f- c915-4557- 82f8-f17da1c447 55</entry> >79ae643f- c915-4557- 82f8-f17da1c447 55</entry> var/lib/ nova/instances/ 79ae643f- c915-4557- 82f8-f17da1c447 55/disk' /> var/lib/ nova/instances/ 79ae643f- c915-4557- 82f8-f17da1c447 55/console. log' append='off'/> var/lib/ nova/instances/ 79ae643f- c915-4557- 82f8-f17da1c447 55/console. log' append='off'/> libvirt. driver [req-cbfa0bed- 3e3e-4402- a071-76c9da1f56 61 c3202ca4eb774fa 8a0f357564a4dd5 72 - - f6ca11d407354d8 f9af85c0b58e310 19 f6ca11d407354d8 f9af85c0b58e310 19] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Failed to start libvirt guest: libvirt. libvirtError: error from service: Timeout was reached manager [req-abaa98f3- 840b-4e5f- 8833-a180438233 19 - - - - -] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] During sync_power_state the instance has a pending task (powering-on). Skip. manager [req-cbfa0bed- 3e3e-4402- a071-76c9da1f56 61 c3202ca4eb774fa 8a0f357564a4dd5 72 - - f6ca11d407354d8 f9af85c0b58e310 19 f6ca11d407354d8 f9af85c0b58e310 19] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Successfully reverted task state from powering-on on failure for instance. manager [req-13b96ce5- 9969-49fb- a5e1-73bd061902 27 a6a714097fbd450 fa917189d6e89b0 73 - - dda5fe657938420 a947ac87de5ae0b 29 dda5fe657938420 a947ac87de5ae0b 29] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Received unexpected event network- vif-unplugged- ee0dab67- 3fd9-4943- a70a-b53a1edaff fc for instance with vm_state stopped and task_state None. manager [req-24e1c96f- 04bd-46be- 9ce1-5468a0bf2d 7e a6a714097fbd450 fa917189d6e89b0 73 - - dda5fe657938420 a947ac87de5ae0b 29 dda5fe657938420 a947ac87de5ae0b 29] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Received unexpected event network- vif-plugged- ee0dab67- 3fd9-4943- a70a-b53a1edaff fc for instance with vm_state stopped and task_state None. manager [-] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] During _sync_instance_ power_state the DB power_state (4) does not match the vm_power_state from the hypervisor (0). Updating power_state in the DB to match the hypervisor. manager [req-11d57952- 8002-450d- 9122-2739af9e25 dd c3202ca4eb774fa 8a0f357564a4dd5 72 - - f6ca11d407354d8 f9af85c0b58e310 19 f6ca11d407354d8 f9af85c0b58e310 19] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Terminating instance libvirt. driver [-] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Instance destroyed successfully. libvirt. driver [req-11d57952- 8002-450d- 9122-2739af9e25 dd c3202ca4eb774fa 8a0f357564a4dd5 72 - - f6ca11d407354d8 f9af85c0b58e310 19 f6ca11d407354d8 f9af85c0b58e310 19] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Deleting instance files /var/lib/ nova/instances/ 79ae643f- c915-4557- 82f8-f17da1c447 55_del libvirt. driver [req-11d57952- 8002-450d- 9122-2739af9e25 dd c3202ca4eb774fa 8a0f357564a4dd5 72 - - f6ca11d407354d8 f9af85c0b58e310 19 f6ca11d407354d8 f9af85c0b58e310 19] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Deletion of /var/lib/ nova/instances/ 79ae643f- c915-4557- 82f8-f17da1c447 55_del complete manager [req-11d57952- 8002-450d- 9122-2739af9e25 dd c3202ca4eb774fa 8a0f357564a4dd5 72 - - f6ca11d407354d8 f9af85c0b58e310 19 f6ca11d407354d8 f9af85c0b58e310 19] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Took 0.14 seconds to destroy the instance on the hypervisor. manager [req-3f77554c- 6cbf-48ed- 9598-8854d76872 67 a6a714097fbd450 fa917189d6e89b0 73 - - dda5fe657938420 a947ac87de5ae0b 29 dda5fe657938420 a947ac87de5ae0b 29] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Neutron deleted interface ee0dab67- 3fd9-4943- a70a-b53a1edaff fc; detaching it from the instance and deleting it from the info cache manager [-] [instance: 79ae643f- c915-4557- 82f8-f17da1c447 55] Took 2.41 seconds to deallocate network for instance. client. report [req-11d57952- 8002-450d- 9122-2739af9e25 dd c3202ca4eb774fa 8a0f357564a4dd5 72 - - f6ca11d407354d8 f9af85c0b58e310 19 f6ca11d407354d8 f9af85c0b58e310 19] Deleted allocations for instance 79ae643f- c915-4557- 82f8-f17da1c447 55 ------- ------- ------- ------- ------- --
-------
2022-04-12 20:48:26.281 1820614 INFO nova.virt.
2022-04-12 20:48:27.310 1820614 WARNING nova.compute.
2022-04-12 20:48:30.655 1820614 WARNING nova.compute.
2022-04-12 20:48:33.254 1820614 INFO nova.virt.
2022-04-12 20:48:34.588 1820614 WARNING nova.compute.
2022-04-12 20:48:36.722 1820614 WARNING nova.compute.
2022-04-12 20:48:40.986 1820614 INFO nova.compute.
2022-04-12 20:49:04.836 1820614 INFO nova.compute.
<uuid>
<entry name='serial'
<entry name='uuid'
<source file='/
<log file='/
<log file='/
2022-04-12 20:49:04.847 1820614 ERROR nova.virt.
2022-04-12 20:49:04.974 1820614 INFO nova.compute.
2022-04-12 20:49:05.079 1820614 INFO nova.compute.
2022-04-12 20:49:05.785 1820614 WARNING nova.compute.
2022-04-12 20:49:08.229 1820614 WARNING nova.compute.
2022-04-12 20:59:06.499 1820614 INFO nova.compute.
2022-04-12 21:08:32.088 1820614 INFO nova.compute.
2022-04-12 21:08:32.094 1820614 INFO nova.virt.
2022-04-12 21:08:32.148 1820614 INFO nova.virt.
2022-04-12 21:08:32.148 1820614 INFO nova.virt.
2022-04-12 21:08:32.231 1820614 INFO nova.compute.
2022-04-12 21:08:34.605 1820614 INFO nova.compute.
2022-04-12 21:08:34.641 1820614 INFO nova.compute.
2022-04-12 21:08:34.910 1820614 INFO nova.scheduler.
-------
So there seem to be some network problems preventing the server from starting again.
Reproducers: /oil-jenkins. canonical. com/artifacts/ 194ee801- 04bf-4ed9- 83cf-ac2e19669b fc/generated/ generated/ openstack/ bundle. yaml api.compute. servers. test_server_ actions. ServerActionsTe stJSON. test_stop_ start_server[ id-af8eafd4- 38a7-4a4b- bdbc-75145a5805 60]
Deploy the bundle https:/
run the test tempest.
Link to crashdumps etc.: /oil-jenkins. canonical. com/artifacts/ 194ee801- 04bf-4ed9- 83cf-ac2e19669b fc/index. html
https:/