nova evacuate fails with neutron

[Mathieu Rohon]

When I deploy nova with a shared storage and with neutron/ML2/linuxbridge, I have an error when I want to to use "nova evacuate"

command :
# nova evacuate 1fa486f3-259c-4e1e-ae82-8b52606f1efd devstack2 --on-shared-storage

here are the logs on the compute node (devstack2) which will host the VM after the evacuation:

2014-02-25 16:08:39.918 ERROR nova.compute.manager [req-8307b6e1-b6ee-423e-9baf-d05a0ac5e91d admin admin] [ins
tance:
1fa486f3-259c-4e1e-ae82-8b52606f1efd] Setting instance vm_state to ERROR
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] Traceback (most recent call last):
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/nova/nova/compute/manager.py", line 5261, in _error_out_instance_on_exception
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] yield
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/nova/nova/compute/manager.py", line 2267, in rebuild_instance
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] extra_usage_info=extra_usage_info)
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/nova/nova/conductor/api.py", line 271, in notify_usage_exists
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] system_metadata, extra_usage_info)
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/nova/nova/conductor/rpcapi.py", line 428, in notify_usage_exists
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] extra_usage_info=extra_usage_info_p)
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/oslo.messaging/oslo/messaging/rpc/client.py", line 150, in call
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] wait_for_reply=True, timeout=timeout)
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/oslo.messaging/oslo/messaging/transport.py", line 87, in _send
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] timeout=timeout)
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/oslo.messaging/oslo/messaging/_drivers/amqpdriver.py", line 393, in send
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/oslo.messaging/oslo/messaging/_drivers/amqpdriver.py", line 386, in _send
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] raise result
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] TypeError: 'NoneType' object is not iterable
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] Traceback (most recent call last):
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd]
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] incoming.message))
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd]
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] return self._do_dispatch(endpoint, method, ctxt, args)
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd]
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/oslo.messaging/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] result = getattr(endpoint, method)(ctxt, **new_args)
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd]
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/nova/nova/conductor/manager.py", line 502, in notify_usage_exists
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] system_metadata, extra_usage_info)
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd]
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/nova/nova/compute/utils.py", line 276, in notify_usage_exists
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] ignore_missing_network_data)
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd]
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] File "/opt/stack/nova/nova/notifications.py", line 288, in bandwidth_usage
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] macs = [vif['address'] for vif in nw_info]
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd]
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd] TypeError: 'NoneType' object is not iterable
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd]
2014-02-25 16:08:39.918 12410 TRACE nova.compute.manager [instance: 1fa486f3-259c-4e1e-ae82-8b52606f1efd]

[Alex Xu (xuhj)]

Error message from nova conductor

2014-03-14 16:12:20.972 15060 ERROR nova.notifications [req-c0bf13ea-db7f-43ee-bef4-dcf176fcc85a 32572797748d4737908ee5e091b58982 0db3961ebbaf4dca9a111079e1c85
766] [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] Failed to get nw_info
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] Traceback (most recent call last):
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] File "/home/os/work-code/nova/nova/notifications.py", line 267, in _get_nwinfo_old_skool
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] instance_ref)
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] File "/home/os/work-code/nova/nova/network/api.py", line 48, in wrapper
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] res = f(self, context, *args, **kwargs)
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] File "/home/os/work-code/nova/nova/network/neutronv2/api.py", line 456, in get_instance_nw_info
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] port_ids)
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] File "/home/os/work-code/nova/nova/network/neutronv2/api.py", line 465, in _get_instance_nw_info
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] port_ids)
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] File "/home/os/work-code/nova/nova/network/neutronv2/api.py", line 1085, in _build_network_info_model
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] context, instance, networks, port_ids)
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] File "/home/os/work-code/nova/nova/network/neutronv2/api.py", line 480, in _gather_port_ids_and_networks
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] if isinstance(instance['info_cache']['network_info'], six.text_type):
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1] KeyError: 'info_cache'
2014-03-14 16:12:20.972 15060 TRACE nova.notifications [instance: 00d927b8-2907-4294-a103-85e6d4081ef1]
2014-03-14 16:12:20.973 15060 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: 'NoneType' object is not iterable

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/80511

[Aaron Rosen (arosen)]

No hits here: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwicmV0dXJuIHNlcnZlci5uZXR3b3Jrc1tuZXR3b3JrX25hbWVdWzBdXCIgQU5EIGZpbGVuYW1lOlwiY29uc29sZS5odG1sXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTMxMDM2OTUyODR9 for the past 7 days. CLosing.

[Aaron Rosen (arosen)]

Whoops sorry wrong bug :(

[Leandro Ignacio Costantino (leandro-i-costantino)]

i tested it and seems to work.

I've another exception on the target compute node here when it's rebuilding

/opt/stack/nova/nova/network/linux_net.py

def delete_ovs_vif_port(bridge, dev):
    _ovs_vsctl(['del-port', bridge, dev])
    delete_net_dev(dev)

Looks like trying to delete a bridge that do not exists in this host, but maybe was due to information retrieved from neutron?
Where you able to do a full evacuation?

[Pavel Kirpichyov (pavel-kirpichyov)]

https://review.openstack.org/#/c/88301/

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/80511
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4c68f19cd8d1f045bde02be7eec50c34c1d87932
Submitter: Jenkins
Branch: master

commit 4c68f19cd8d1f045bde02be7eec50c34c1d87932
Author: He Jie Xu <email address hidden>
Date: Fri Mar 14 15:59:12 2014 +0800

    Add info_cache as expected attribute when evacuate instance

    Nova evacuates instance failed with neutron. The reason is
    when nova-conductor try to get network info for instance, it
    need instance include 'info_cache', but the instance dict which
    passed didn't include it. This patch adds 'info_cache' as the
    expected attribute for get instance.

    Change-Id: I6f110c607e32b0ab536916741605b25d572331dc
    Closes-Bug: #1284709

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/96066

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/96066
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=43f04375adbe15b50b74b0ecc83d661b7571e544
Submitter: Jenkins
Branch: stable/icehouse

commit 43f04375adbe15b50b74b0ecc83d661b7571e544
Author: He Jie Xu <email address hidden>
Date: Fri Mar 14 15:59:12 2014 +0800

    Add info_cache as expected attribute when evacuate instance

    Nova evacuates instance failed with neutron. The reason is
    when nova-conductor try to get network info for instance, it
    need instance include 'info_cache', but the instance dict which
    passed didn't include it. This patch adds 'info_cache' as the
    expected attribute for get instance.

    Change-Id: I6f110c607e32b0ab536916741605b25d572331dc
    Closes-Bug: #1284709
    (cherry picked from commit 4c68f19cd8d1f045bde02be7eec50c34c1d87932)