nova-lxd

Error on delete interface when destroying instance

Bug #1808819 reported by Sahid Orentino
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
nova-lxd
Fix Committed
Undecided
Sahid Orentino

Bug Description

An instance may be destroyed prior nova-compute receives a vif-delete-event. detach_interface() will so try to get the lxd profile related to the instance which does not exist anymore, the process will fail with:

Dec 14 12:26:42.331921 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: DEBUG nova.network.base_api [-] [instance: 08fcff77-2118-4cd3-828f-daeb619fc3bd] Updating instance_info_cache with network_info: [] {{(pid=27625) update_instance_cache_with_nw_info /opt/stack/new/nova/nova/network/base_api.py:48}}
Dec 14 12:26:42.352509 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: INFO nova.compute.manager [-] [instance: 08fcff77-2118-4cd3-828f-daeb619fc3bd] Took 1.22 seconds to deallocate network for instance.
Dec 14 12:26:42.358135 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: DEBUG nova.compute.manager [req-4661f7c1-fcc8-48d8-9c2f-30b246cb331a req-a0f6e191-042f-481c-ae20-a300f7c1cde7 service nova] [instance: 08fcff77-2118-4cd3-828f-daeb619fc3bd] Received event network-vif-deleted-be4dbfa3-3fcc-4dbc-bd2f-d1226321ab2e {{(pid=27625) external_instance_event /opt/stack/new/nova/nova/compute/manager.py:8074}}
Dec 14 12:26:42.358434 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: INFO nova.compute.manager [req-4661f7c1-fcc8-48d8-9c2f-30b246cb331a req-a0f6e191-042f-481c-ae20-a300f7c1cde7 service nova] [instance: 08fcff77-2118-4cd3-828f-daeb619fc3bd] Neutron deleted interface be4dbfa3-3fcc-4dbc-bd2f-d1226321ab2e; detaching it from the instance and deleting it from the info cache
Dec 14 12:26:42.358843 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: DEBUG nova.network.base_api [req-4661f7c1-fcc8-48d8-9c2f-30b246cb331a req-a0f6e191-042f-481c-ae20-a300f7c1cde7 service nova] [instance: 08fcff77-2118-4cd3-828f-daeb619fc3bd] Updating instance_info_cache with network_info: [] {{(pid=27625) update_instance_cache_with_nw_info /opt/stack/new/nova/nova/network/base_api.py:48}}
Dec 14 12:26:42.454631 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server [req-4661f7c1-fcc8-48d8-9c2f-30b246cb331a req-a0f6e191-042f-481c-ae20-a300f7c1cde7 service nova] Exception during message handling: NotFound: not found
Dec 14 12:26:42.454872 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Dec 14 12:26:42.455022 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Dec 14 12:26:42.455156 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Dec 14 12:26:42.455291 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Dec 14 12:26:42.455432 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Dec 14 12:26:42.455574 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Dec 14 12:26:42.455807 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Dec 14 12:26:42.456014 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception_wrapper.py", line 79, in wrapped
Dec 14 12:26:42.456161 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb)
Dec 14 12:26:42.456350 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Dec 14 12:26:42.456498 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server self.force_reraise()
Dec 14 12:26:42.456644 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Dec 14 12:26:42.456805 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Dec 14 12:26:42.456941 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/exception_wrapper.py", line 69, in wrapped
Dec 14 12:26:42.457078 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
Dec 14 12:26:42.457218 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 8090, in external_instance_event
Dec 14 12:26:42.457361 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server event.tag)
Dec 14 12:26:42.457496 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova/nova/compute/manager.py", line 7995, in _process_instance_vif_deleted_event
Dec 14 12:26:42.457626 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server self.driver.detach_interface(context, instance, vif)
Dec 14 12:26:42.457771 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/nova-lxd/nova/virt/lxd/driver.py", line 803, in detach_interface
Dec 14 12:26:42.457902 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server profile = self.client.profiles.get(instance.name)
Dec 14 12:26:42.458042 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pylxd/models/profile.py", line 38, in get
Dec 14 12:26:42.458175 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server response = client.api.profiles[name].get()
Dec 14 12:26:42.458311 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pylxd/client.py", line 148, in get
Dec 14 12:26:42.458440 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server is_api=is_api)
Dec 14 12:26:42.458578 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/pylxd/client.py", line 103, in _assert_response
Dec 14 12:26:42.458707 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server raise exceptions.NotFound(response)
Dec 14 12:26:42.458889 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server NotFound: not found
Dec 14 12:26:42.459028 ubuntu-xenial-vexxhost-sjc1-0001200467 nova-compute[27625]: ERROR oslo_messaging.rpc.server

Sahid Orentino (sahid-ferdjaoui)
Changed in nova-lxd:
assignee: nobody → sahid (sahid-ferdjaoui)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova-lxd (master)

Fix proposed to branch: master
Review: https://review.openstack.org/625616

Changed in nova-lxd:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova-lxd (master)

Reviewed: https://review.openstack.org/625616
Committed: https://git.openstack.org/cgit/openstack/nova-lxd/commit/?id=bf3e123e1f7c09c64beecf0ff092c7e6b84590b4
Submitter: Zuul
Branch: master

commit bf3e123e1f7c09c64beecf0ff092c7e6b84590b4
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Mon Dec 17 09:54:48 2018 -0500

    fix error on delete interface when destroying instance

    An instance may be destroyed prior nova-compute receives a
    vif-delete-event. detach_interface() will so try to get the lxd
    profile related to the instance which does not exist anymore, the
    process will fail with a NotFound exception raised by lxclib.

    In this commit we solve the issue by handling the exception and so
    avoiding on updatating the lxc profile. We still continue the process
    of vif_driver.un_plug() to ensure all got cleaned.

    Change-Id: If7fd5dd17323ec625df3518311dfb012302b2711
    Closes-Bug: #1808819
    Signed-off-by: Sahid Orentino Ferdjaoui <email address hidden>

Changed in nova-lxd:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.