Can't detach interface from VM (if VM has two interface with same mac addresses)

[Eugene Frolov]

How to reproduce:

1. Run any VM.
2. Create two networks.
3. Create two ports for each network with same mac addresses
4. Attach those ports to VM
5. Try to detach any interface.

Expected result:
The interface should be detached from VM.

Actual result:
We don't get any errors (via API) on previous steps but an interface still attached to VM

Environment:
* fuel_release: 9.0
* fuel_openstack_version: mitaka-9.0
* libvirt0: 1.2.9.3-9~u14.04+mos10
* hypervisor: Libvirt + KVM

Example:

(OpenStack-venv)agent@laptop ~/projects $ nova interface-list c5ae5a9a-54a2-47b8-800e-619b8bc286f5
+------------+--------------------------------------+--------------------------------------+----------------+-------------------+
| Port State | Port ID | Net ID | IP addresses | MAC Addr |
+------------+--------------------------------------+--------------------------------------+----------------+-------------------+
| ACTIVE | 0d9377b4-4f8f-467a-bae7-54d0d70e5262 | 3f5adcaa-d3e5-4caf-be8f-474751de5589 | 192.168.0.1 | fa:16:3e:60:46:1f |
| ACTIVE | 13cac036-7b6d-4188-879f-650c8d9e1f63 | 28939866-7379-4279-800c-b64c2776e1e0 | 192.168.111.82 | fa:16:3e:24:0d:a4 |
| ACTIVE | 310b9883-806d-4038-a095-1625abecbcb1 | 311c5a7e-5cb0-47e2-8aa0-20d74c4ee8c2 | 192.168.0.1 | fa:16:3e:60:46:1f |
+------------+--------------------------------------+--------------------------------------+----------------+-------------------+
(OpenStack-venv)agent@laptop ~/projects $ nova interface-detach c5ae5a9a-54a2-47b8-800e-619b8bc286f5 0d9377b4-4f8f-467a-bae7-54d0d70e5262
(OpenStack-venv)agent@laptop ~/projects $ nova interface-list c5ae5a9a-54a2-47b8-800e-619b8bc286f5
+------------+--------------------------------------+--------------------------------------+----------------+-------------------+
| Port State | Port ID | Net ID | IP addresses | MAC Addr |
+------------+--------------------------------------+--------------------------------------+----------------+-------------------+
| ACTIVE | 0d9377b4-4f8f-467a-bae7-54d0d70e5262 | 3f5adcaa-d3e5-4caf-be8f-474751de5589 | 192.168.0.1 | fa:16:3e:60:46:1f |
| ACTIVE | 13cac036-7b6d-4188-879f-650c8d9e1f63 | 28939866-7379-4279-800c-b64c2776e1e0 | 192.168.111.82 | fa:16:3e:24:0d:a4 |
| ACTIVE | 310b9883-806d-4038-a095-1625abecbcb1 | 311c5a7e-5cb0-47e2-8aa0-20d74c4ee8c2 | 192.168.0.1 | fa:16:3e:60:46:1f |
+------------+--------------------------------------+--------------------------------------+----------------+-------------------+

logs from compute:

libvirt:
<11>Sep 7 12:18:00 node-9 libvirtd: 11320: error : virDomainNetFindIdx:11005 : operation failed: multiple devices matching mac address fa:16:3e:60:46:1f found

nova compute:
<183>Sep 6 17:58:04 node-7 nova-compute: 2016-09-06 17:58:04.348 7652 DEBUG nova.objects.instance [req-db381757-3fbf-4bb8-a4df-160e1422a005 2b96e098d62147d8b9a15f635e0dd51a 4b867602d6974059afb2489a71dfaabb - - -] Lazy-loading 'flavor' on Instance uuid d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a obj_load_attr /usr/lib/python2.7/dist-packages/nova/objects/instance.py:895
<183>Sep 6 17:58:04 node-7 nova-compute: 2016-09-06 17:58:04.430 7652 DEBUG nova.virt.libvirt.vif [req-db381757-3fbf-4bb8-a4df-160e1422a005 2b96e098d62147d8b9a15f635e0dd51a 4b867602d6974059afb2489a71dfaabb - - -] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=False,availability_zone=None,cell_name=None,cleaned=False,config_drive='True',created_at=2016-09-06T17:41:02Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,disable_terminate=False,display_description='4894d01a-e464-4d17-b309-3c39a9875147',display_name='4894d01a-e464-4d17-b309-3c39a9875147',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(211),host='node-7.domain.tld',hostname='4894d01a-e464-4d17-b309-3c39a9875147',id=277,image_ref='9213a377-e2c2-4cc1-b1e8-483369db03fa',info_cache=InstanceInfoCache,instance_type_id=211,kernel_id='',key_data=None,key_name=None,launch_index=0,launched_at=2016-09-06T17:41:08Z,launched_on='node-7.domain.tld',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='node-7.domain.tld',numa_topology=<?>,old_flavor=None,os_type=None,pci_devices=<?>,pci_requests=<?>,power_state=1,progress=0,project_id='4b867602d6974059afb2489a71dfaabb',ramdisk_id='',reservation_id='r-ybcxxyxm',root_device_name='/dev/hda',root_gb=1,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={image_base_image_ref='9213a377-e2c2-4cc1-b1e8-483369db03fa',image_container_format='bare',image_disk_format='iso',image_min_disk='1',image_min_ram='0'},tags=<?>,task_state=None,terminated_at=None,updated_at=2016-09-06T17:41:08Z,user_data=None,user_id='2b96e098d62147d8b9a15f635e0dd51a',uuid=d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a,vcpu_model=<?>,vcpus=1,vm_mode=None,vm_state='active') vif=VIF({'profile': {}, 'ovs_interfaceid': u'4461eba1-a2cd-48f4-9289-86fc8d33bf12', 'preserve_on_delete': True, 'network': Network({'bridge': u'br-int', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'versio
<183>Sep 6 17:58:04 node-7 nova-compute: 2016-09-06 17:58:04.432 7652 DEBUG nova.virt.libvirt.vif [req-db381757-3fbf-4bb8-a4df-160e1422a005 2b96e098d62147d8b9a15f635e0dd51a 4b867602d6974059afb2489a71dfaabb - - -] vif_type=ovs instance=Instance(access_ip_v4=None,access_ip_v6=None,architecture=None,auto_disk_config=False,availability_zone=None,cell_name=None,cleaned=False,config_drive='True',created_at=2016-09-06T17:41:02Z,default_ephemeral_device=None,default_swap_device=None,deleted=False,deleted_at=None,disable_terminate=False,display_description='4894d01a-e464-4d17-b309-3c39a9875147',display_name='4894d01a-e464-4d17-b309-3c39a9875147',ec2_ids=<?>,ephemeral_gb=0,ephemeral_key_uuid=None,fault=<?>,flavor=Flavor(211),host='node-7.domain.tld',hostname='4894d01a-e464-4d17-b309-3c39a9875147',id=277,image_ref='9213a377-e2c2-4cc1-b1e8-483369db03fa',info_cache=InstanceInfoCache,instance_type_id=211,kernel_id='',key_data=None,key_name=None,launch_index=0,launched_at=2016-09-06T17:41:08Z,launched_on='node-7.domain.tld',locked=False,locked_by=None,memory_mb=512,metadata={},migration_context=<?>,new_flavor=None,node='node-7.domain.tld',numa_topology=<?>,old_flavor=None,os_type=None,pci_devices=<?>,pci_requests=<?>,power_state=1,progress=0,project_id='4b867602d6974059afb2489a71dfaabb',ramdisk_id='',reservation_id='r-ybcxxyxm',root_device_name='/dev/hda',root_gb=1,security_groups=SecurityGroupList,services=<?>,shutdown_terminate=False,system_metadata={image_base_image_ref='9213a377-e2c2-4cc1-b1e8-483369db03fa',image_container_format='bare',image_disk_format='iso',image_min_disk='1',image_min_ram='0'},tags=<?>,task_state=None,terminated_at=None,updated_at=2016-09-06T17:41:08Z,user_data=None,user_id='2b96e098d62147d8b9a15f635e0dd51a',uuid=d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a,vcpu_model=<?>,vcpus=1,vm_mode=None,vm_state='active') vif=VIF({'profile': {}, 'ovs_interfaceid': u'4461eba1-a2cd-48f4-9289-86fc8d33bf12', 'preserve_on_delete': True, 'network': Network({'bridge': u'br-int', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'versio
<183>Sep 6 17:58:04 node-7 nova-compute: 2016-09-06 17:58:04.436 7652 DEBUG nova.virt.libvirt.config [req-db381757-3fbf-4bb8-a4df-160e1422a005 2b96e098d62147d8b9a15f635e0dd51a 4b867602d6974059afb2489a71dfaabb - - -] Generated XML ('<interface type="bridge">\n <mac address="fa:16:3e:74:b0:c8"/>\n <model type="virtio"/>\n <source bridge="br-int"/>\n <target dev="tap4461eba1-a2"/>\n <virtualport type="openvswitch">\n <parameters interfaceid="4461eba1-a2cd-48f4-9289-86fc8d33bf12"/>\n </virtualport>\n</interface>\n',) to_xml /usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py:82
<179>Sep 6 17:58:04 node-7 nova-compute: 2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [req-db381757-3fbf-4bb8-a4df-160e1422a005 2b96e098d62147d8b9a15f635e0dd51a 4b867602d6974059afb2489a71dfaabb - - -] [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] detaching network adapter failed.
2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] Traceback (most recent call last):
2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1539, in detach_interface
2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] guest.detach_device(cfg, persistent=True, live=live)
2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 357, in detach_device
2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] self._domain.detachDeviceFlags(conf.to_xml(), flags=flags)
2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] rv = execute(f, *args, **kwargs)
2016-09-06 17:58:04.441 7652 ERROR nova.virt.libvirt.driver [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] File "/usr/lib/python2.7/dist-packages/eventlet/
<180>Sep 6 17:58:04 node-7 nova-compute: 2016-09-06 17:58:04.473 7652 WARNING nova.compute.manager [req-db381757-3fbf-4bb8-a4df-160e1422a005 2b96e098d62147d8b9a15f635e0dd51a 4b867602d6974059afb2489a71dfaabb - - -] [instance: d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a] Detach interface failed, port_id=4461eba1-a2cd-48f4-9289-86fc8d33bf12, reason: Failed to detach network adapter device from d95ab3d6-6b8f-42f0-8bf8-a553e99ee41a

[Matt Riedemann (mriedem)]

What version of nova/neutron are you using?

[Matt Riedemann (mriedem)]

What is the use case for this? I've never heard of anyone trying to do this in nova and not sure if it's supposed to be supported.

[Matt Riedemann (mriedem)]

This is where it blows up:

https://github.com/openstack/nova/blob/d0775c50d0c2bd50a62ccd49ea7063948af6c3b3/nova/virt/libvirt/driver.py#L1400

https://github.com/openstack/nova/blob/d0775c50d0c2bd50a62ccd49ea7063948af6c3b3/nova/virt/libvirt/guest.py#L404

Can you recreate this with DEBUG logging and post the xml output from this?

https://github.com/openstack/nova/blob/d0775c50d0c2bd50a62ccd49ea7063948af6c3b3/nova/virt/libvirt/guest.py#L403

[Eugene Frolov (efrolov)]

> What version of nova/neutron are you using?

The bug can be reproduced on any mitaka version. I tried latest code from github.
nova: commit fb3f1706c68ea5b58f05ea810c6339f2449959de
neutron: commit 5e6168fab36415d6d677c32501e64371f74a6992

>What is the use case for this? I've never heard of anyone trying to do this in nova and not sure if it's supposed to be supported.

if it is not supported then I should get an error when I connecting port to instance. Otherwise I should detach interface.

> This is where it blows up:
> https://github.com/openstack/nova/blob/d0775c50d0c2bd50a62ccd49ea7063948af6c3b3/nova/virt/libvirt/driver.py#L1400
> https://github.com/openstack/nova/blob/d0775c50d0c2bd50a62ccd49ea7063948af6c3b3/nova/virt/libvirt/guest.py#L404

I think, yes. Also I have trace from libvirt:
 Traceback (most recent call last):
   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1522, in detach_interface
     guest.detach_device(cfg, persistent=True, live=live)
   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 357, in detach_device
     self._domain.detachDeviceFlags(conf.to_xml(), flags=flags)
   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
     result = proxy_call(self._autowrap, f, *args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
     rv = execute(f, *args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
     six.reraise(c, e, tb)
   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
     rv = meth(*args, **kwargs)
   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 985, in detachDeviceFlags
     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
 libvirtError: operation failed: multiple devices matching mac address fa:16:3e:b1:8a:7c found

> Can you recreate this with DEBUG logging and post the xml output from this?

Generated XML ('<interface type="bridge">\n <mac address="fa:16:3e:b1:8a:7c"/>\n <model type="virtio"/>\n <source bridge="qbr418139e8-e2"/>\n <target dev="tap418139e8-e2"/>\n</interface>\n',)

[Matt Riedemann (mriedem)]

(9:29:36 AM) danpb: mriedem: this is still nova's fault
(9:29:57 AM) danpb: mriedem: the API contract for virDomainDetachDevice says you must pass the full XML for the device shown in the current guest XML
(9:30:13 AM) danpb: mriedem: nova doesn't do this - it re-creates the minimal <interface> XML from scratch
(9:30:31 AM) danpb: mriedem: as a result it is missing the PCI device ID libvirt would normally use to uniquely identify the device
(9:30:40 AM) danpb: mriedem: so libvirt has no option but to try and find it using mac address
(9:30:54 AM) mriedem: ok, do we have any existing code to build that interface xml from the vif?
(9:31:01 AM) mriedem: seems like a simple fix
(9:31:24 AM) mriedem: it's a latent bug, so not an rc issue, just wanted your ack on it
(9:31:33 AM) danpb: yeah, its been broken forever

[Daniel Berrange (berrange)]

This is due to nova violating the libvirt XML contract for virDomainDetachDevice.

You are required to provide the full <interface> XML as currently shown in the *live* guest XML.

Nova doesn't do this, instead it builds new minimal <interface> XML from scatch. Thus nova is missing the PCI device information that libvirt would use to uniquely identify devices.

[leehom (feli5)]

Yes. Without PCI id when interface has the same mac address, libvirt cannot correctly find out the right interface to detach.
By generate full <interface> XML and pass it to libvirt can solve this bug.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/372243

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/372243
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a5c38cc861f052b89bc1ce182f68784c493c723e
Submitter: Jenkins
Branch: master

commit a5c38cc861f052b89bc1ce182f68784c493c723e
Author: Leehom Li (feli5) <email address hidden>
Date: Mon Sep 19 13:43:46 2016 +0800

    Support detach interface with same MAC from instance

    When detach_interface nova uses <interface> XML generated
    from scatch, which is missing the PCI device information that
    libvirt would use to uniquely identify devices.

    In case instance has mutiple interfaces with same MAC address.
    Libvirt will failed with below error message:
        libvirtError: operation failed: multiple devices matching
        mac address fa:16:3e:60:46:1f found

    This patch fixes this problem by provide a new function
    get_interface_by_cfg, this function uses cfg generated by
    nova.virt.libvirt.vif.get_config as parameter,
    return a LibvirtConfigGuestInterface object.

    Also added function format_dom for
    LibvirtConfigGuestDeviceAddressPCI, which will be used to
    generate pci address for LibvirtConfigGuestInterface.

    Change-Id: I8acae90c9d2111ed35f58f374f321d64f01ba563
    Closes-Bug: #1621076

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/470347

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/470347
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=059c5298e25361d5820842b10e5a1880fe7b18b3
Submitter: Jenkins
Branch: stable/newton

commit 059c5298e25361d5820842b10e5a1880fe7b18b3
Author: Leehom Li (feli5) <email address hidden>
Date: Mon Sep 19 13:43:46 2016 +0800

    Support detach interface with same MAC from instance

    When detach_interface nova uses <interface> XML generated
    from scatch, which is missing the PCI device information that
    libvirt would use to uniquely identify devices.

    In case instance has mutiple interfaces with same MAC address.
    Libvirt will failed with below error message:
        libvirtError: operation failed: multiple devices matching
        mac address fa:16:3e:60:46:1f found

    This patch fixes this problem by provide a new function
    get_interface_by_cfg, this function uses cfg generated by
    nova.virt.libvirt.vif.get_config as parameter,
    return a LibvirtConfigGuestInterface object.

    Also added function format_dom for
    LibvirtConfigGuestDeviceAddressPCI, which will be used to
    generate pci address for LibvirtConfigGuestInterface.

    Conflicts:
          nova/tests/unit/virt/libvirt/test_driver.py

    NOTE(mriedem): The conflict is due to change
    I5c461a8242c51994d12ce9c6774d5f956232f950 not being in Newton.

    Change-Id: I8acae90c9d2111ed35f58f374f321d64f01ba563
    Closes-Bug: #1621076
    (cherry picked from commit a5c38cc861f052b89bc1ce182f68784c493c723e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.8 release.