OpenStack Compute (nova)

  1. Series ocata
  2. Bug #1713783
  3. Activity log

Activity log for bug #1713783

Date Who What changed Old value New value Message
2017-08-29 17:17:15 Elod Illes bug added bug
2017-08-29 17:23:27 Balazs Gibizer description Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so the manager tries to delete the instance. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs: 2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling NoValidHost: No valid host was found. There are not enough hosts available. 2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1) 2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1 2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0 2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0 2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[] 2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1 2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0 2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2) 2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1 2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0 2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0 2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[] 2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1 2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0 2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics 2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198 2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618 2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442 2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0) 2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host Description =========== In case of a failed evacuation attempt the status of the migration is 'accepted' instead of 'failed' so when source compute is recovered the compute manager tries to delete the instance from the source host. However a secondary fault prevents deleting the allocation in placement so the actual deletion of the instance fails as well. Steps to reproduce ================== The following functional test reproduces the bug: https://review.openstack.org/#/c/498482/ What it does: initiate evacuation when no valid host is available and evacuation fails, but nova manager still tries to delete the instance. Logs:     2017-08-29 19:11:15,751 ERROR [oslo_messaging.rpc.server] Exception during message handling     NoValidHost: No valid host was found. There are not enough hosts available.     2017-08-29 19:11:16,103 INFO [nova.tests.functional.test_servers] Running periodic for compute1 (host1)     2017-08-29 19:11:16,115 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,120 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,131 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/allocations" status: 200 len: 152 microversion: 1.0     2017-08-29 19:11:16,138 INFO [nova.compute.resource_tracker] Final resource view: name=host1 phys_ram=8192MB used_ram=1024MB phys_disk=1028GB used_disk=1GB total_vcpus=10 used_vcpus=1 pci_stats=[]     2017-08-29 19:11:16,146 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,151 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/4e8e23ff-0c52-4cf7-8356-d9fa88536316/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,152 INFO [nova.tests.functional.test_servers] Running periodic for compute2 (host2)     2017-08-29 19:11:16,163 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,168 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,176 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/allocations" status: 200 len: 54 microversion: 1.0     2017-08-29 19:11:16,184 INFO [nova.compute.resource_tracker] Final resource view: name=host2 phys_ram=8192MB used_ram=512MB phys_disk=1028GB used_disk=0GB total_vcpus=10 used_vcpus=0 pci_stats=[]     2017-08-29 19:11:16,192 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/aggregates" status: 200 len: 18 microversion: 1.1     2017-08-29 19:11:16,197 INFO [nova.api.openstack.placement.requestlog] 127.0.0.1 "GET /placement/resource_providers/531b1ce8-def1-455d-95b3-4140665d956f/inventories" status: 200 len: 401 microversion: 1.0     2017-08-29 19:11:16,198 INFO [nova.tests.functional.test_servers] Finished with periodics     2017-08-29 19:11:16,255 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/servers/5058200c-478e-4449-88c1-906fdd572662" status: 200 len: 1875 microversion: 2.53 time: 0.056198     2017-08-29 19:11:16,262 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/6f70656e737461636b20342065766572/os-migrations" status: 200 len: 373 microversion: 2.53 time: 0.004618     2017-08-29 19:11:16,280 INFO [nova.api.openstack.requestlog] 127.0.0.1 "PUT /v2.1/6f70656e737461636b20342065766572/os-services/c269bc74-4720-4de4-a6e5-889080b892a0" status: 200 len: 245 microversion: 2.53 time: 0.016442     2017-08-29 19:11:16,281 INFO [nova.service] Starting compute node (version 16.0.0)     2017-08-29 19:11:16,296 INFO [nova.compute.manager] Deleting instance as it has been evacuated from this host
2017-08-29 17:24:01 Balazs Gibizer tags evacuate placement
2017-08-29 17:25:03 Balazs Gibizer summary Failed evacuation also tries to delete the VM After failed evacuation the recovered source compute tries to delete the instance
2017-08-29 17:25:45 Balazs Gibizer bug added subscriber Balazs Gibizer
2017-08-31 13:09:46 Matt Riedemann nominated for series nova/ocata
2017-08-31 13:09:46 Matt Riedemann bug task added nova/ocata
2017-08-31 13:09:46 Matt Riedemann nominated for series nova/pike
2017-08-31 13:09:46 Matt Riedemann bug task added nova/pike
2017-08-31 13:10:00 Matt Riedemann nova: importance Undecided High
2017-08-31 13:10:36 Matt Riedemann nova/ocata: importance Undecided High
2017-08-31 13:10:38 Matt Riedemann nova/pike: importance Undecided High
2017-08-31 13:10:42 Matt Riedemann nova: status New Triaged
2017-08-31 13:10:47 Matt Riedemann nominated for series nova/newton
2017-08-31 13:10:47 Matt Riedemann bug task added nova/newton
2017-08-31 13:11:05 Matt Riedemann information type Public Private Security
2017-08-31 13:11:11 Matt Riedemann nova/newton: importance Undecided High
2017-08-31 13:11:14 Matt Riedemann nova/newton: status New Triaged
2017-08-31 13:11:15 Matt Riedemann nova/ocata: status New Triaged
2017-08-31 13:11:18 Matt Riedemann nova/pike: status New Triaged
2017-08-31 13:14:04 Matt Riedemann removed subscriber Balazs Gibizer
2017-08-31 14:48:12 Tristan Cacqueray bug added subscriber Nova Core security contacts
2017-08-31 18:31:55 Matt Riedemann information type Private Security Public
2017-08-31 19:04:34 Jeremy Stanley tags evacuate placement evacuate placement security
2017-08-31 19:04:49 Jeremy Stanley bug task added ossa
2017-08-31 19:05:09 Jeremy Stanley ossa: status New Won't Fix
2017-09-01 14:45:56 OpenStack Infra nova: status Triaged In Progress
2017-09-01 14:45:56 OpenStack Infra nova: assignee Illes Elod (elod-illes)
2017-09-01 16:34:46 OpenStack Infra nova: assignee Illes Elod (elod-illes) Matt Riedemann (mriedem)
2017-09-18 14:50:53 OpenStack Infra nova: status In Progress Fix Released
2017-09-18 14:55:19 Matt Riedemann nova: assignee Matt Riedemann (mriedem) Illes Elod (elod-illes)
2017-09-19 00:14:14 OpenStack Infra nova/pike: status Triaged In Progress
2017-09-19 00:14:14 OpenStack Infra nova/pike: assignee Matt Riedemann (mriedem)
2017-09-19 02:58:52 OpenStack Infra tags evacuate placement security evacuate in-stable-pike placement security
2017-09-19 16:03:14 OpenStack Infra nova/pike: status In Progress Fix Committed
2017-11-09 13:42:48 OpenStack Infra nova/ocata: status Triaged In Progress
2017-11-09 13:42:48 OpenStack Infra nova/ocata: assignee Balazs Gibizer (balazs-gibizer)
2018-03-28 14:08:22 OpenStack Infra nova/ocata: assignee Balazs Gibizer (balazs-gibizer) Illes Elod (elod-illes)
2018-04-16 19:58:02 Matt Riedemann bug task deleted nova/newton
2018-04-17 16:17:08 OpenStack Infra tags evacuate in-stable-pike placement security evacuate in-stable-ocata in-stable-pike placement security
2018-04-17 16:32:48 OpenStack Infra nova/ocata: assignee Illes Elod (elod-illes) Matt Riedemann (mriedem)
2018-04-17 16:33:15 Matt Riedemann nova/ocata: assignee Matt Riedemann (mriedem) Balazs Gibizer (balazs-gibizer)
2018-04-20 05:55:58 OpenStack Infra nova/ocata: status In Progress Fix Committed
2019-01-16 13:44:14 Oleksiy Molchanov bug task added fuel
2019-01-16 13:44:23 Oleksiy Molchanov fuel: assignee Oleksiy Molchanov (omolchanov)
2019-01-16 13:44:25 Oleksiy Molchanov fuel: importance Undecided High
2019-01-16 13:44:29 Oleksiy Molchanov fuel: status New Confirmed
2019-01-16 13:44:32 Oleksiy Molchanov fuel: milestone 9.2-mu-11
2019-03-27 14:23:46 Denis Meltsaykin fuel: status Confirmed Fix Committed
2019-04-03 10:15:50 Pavel Glazov fuel: status Fix Committed Fix Released