StarlingX

Interface attach cli rejected after 15 vnics attach, detach and attach again

Bug #1790968 reported by Senthil Mukundakumar on 2018-09-05

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	Medium	Jim Somerville

Bug Description

Brief Description
-----------------
Interface attach cli rejected after attach 15 vnics, live migrate, detach 15 vnics, attach again at vnic13

Severity
--------
Major

Steps to Reproduce
------------------
        - Boot a vm with only mgmt interface
        - Attach an 15 vifs to vm with given net-id and vif_model:virtio
        - Bring up the interface from vm
        - ping between base_vm and vm_under_test over mgmt & tenant network
        - Perform VM action - live migrate
        - Verify ping between base_vm and vm_under_test over mgmt & tenant network after vm operation
        - detach all the tenant interface
        - Again attach an 15 vifs to vm with given net-id and vif_model:virtio

Expected Behavior
------------------
All 15 vifs should be attachable

Actual Behavior
----------------
After 13 vif attachment then the cli rejected.
E Details: CLI 'nova --os-username 'tenant2' --os-password 'Li69nux*' --os-project-name tenant2 --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne interface-attach --net-id 0b736d87-413a-4fb8-a6c0-58800f9ba0c4 --wrs-if:vif_model virtio 9ec7cfa8-18f4-4baa-9801-d980f25fb34a' failed to execute. Output: ERROR (ClientException): Failed to attach network adapter device to 9ec7cfa8-18f4-4baa-9801-d980f25fb34a (HTTP 500) (Request-ID: req-a09e4e55-6e89-47d7-a0d4-b45bc2ba69c5)

Reproducibility
---------------
Reproducible (2 out of 2 times)

System Configuration
--------------------
Any

Branch/Pull Time/Commit
-----------------------
StarlingX master as of 2018-08-31_20-18-00

See original description

Tags:

Ghada Khalil (gkhalil) on 2018-09-06

tags:

added: stx.networking

Senthil Mukundakumar (smukunda) on 2018-09-07

description:

updated

Revision history for this message

Joseph Richard (josephrichard) wrote on 2018-09-07:

Download full text (27.5 KiB)

I looked through the logs briefly. The failure is with the request from nova-compute to libvirt.

2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [req-a09e4e55-6e89-47d7-a0d4-b45bc2ba69c5 204c0a61b28f40b3bcbd1805179a8ae5 bb5b3bb7440d4cbc8b279e1f895551e2 - default default] [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] attaching network adapter failed.: libvirtError: internal error: No more available PCI slots
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] Traceback (most recent call last):
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1642, in attach_interface
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] guest.attach_device(cfg, persistent=True, live=live)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 322, in attach_device
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] self._domain.attachDeviceFlags(device_xml, flags=flags)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] result = proxy_call(self._autowrap, f, *args, **kwargs)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] rv = execute(f, *args, **kwargs)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] six.reraise(c, e, tb)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] rv = meth(*args, **kwargs)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 587, in attachDeviceFlags
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] lib...

I looked through the logs briefly.  The failure is with the request from nova-compute to libvirt.

2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [req-a09e4e55-6e89-47d7-a0d4-b45bc2ba69c5 204c0a61b28f40b3bcbd1805179a8ae5 bb5b3bb7440d4cbc8b279e1f895551e2 - default default] [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] attaching network adapter failed.: libvirtError: internal error: No more available PCI slots
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] Traceback (most recent call last):
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1642, in attach_interface
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]     guest.attach_device(cfg, persistent=True, live=live)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 322, in attach_device
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]     rv = execute(f, *args, **kwargs)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]     six.reraise(c, e, tb)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]     rv = meth(*args, **kwargs)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 587, in attachDeviceFlags
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] libvirtError: internal error: No more available PCI slots

I see that the detaches were failing in libvirt, but unfortunately nova ignores this and removes it.
2018-09-02 00:58:11.850 57065 WARNING nova.virt.libvirt.driver [req-faaca6c7-48bd-4b42-8907-78c98819d18d 204c0a61b28f40b3bcbd1805179a8ae5 bb5b3bb7440d4cbc8b279e1f895551e2 - default default] [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] Detaching interface fa:16:3e:7a:52:7a failed because the device is no longer found on the guest.
2018-09-02 00:58:12.524 57065 INFO nova.compute.manager [req-29196fd3-9d22-4255-ab8b-d9a16a68e4f8 c3c9ebd4b37e4ee2b3c93c2b25d490e0 fd5046f6d69a442982ae832c6d0a8e15 - default default] [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] Neutron deleted interface 88f17a31-7e1e-4c6c-a4a0-3340fac00f7c; detaching it from the instance and deleting it from the info cache

VM information as seen from virsh:

compute-0:/home/wrsroot# virsh qemu-monitor-command --hmp instance-0000003d "info pci"
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:1237
      id ""
  Bus  0, device   1, function 0:
    ISA bridge: PCI device 8086:7000
      id ""
  Bus  0, device   1, function 1:
    IDE controller: PCI device 8086:7010
      BAR4: I/O at 0xc0a0 [0xc0af].
      id ""
  Bus  0, device   1, function 2:
    USB controller: PCI device 8086:7020
      IRQ 11.
      BAR4: I/O at 0xc040 [0xc05f].
      id "usb"
  Bus  0, device   1, function 3:
    Bridge: PCI device 8086:7113
      IRQ 9.
      id ""
  Bus  0, device   2, function 0:
    VGA controller: PCI device 1013:00b8
      BAR0: 32 bit prefetchable memory at 0xfc000000 [0xfdffffff].
      BAR1: 32 bit memory at 0xfebd0000 [0xfebd0fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id "video0"
  Bus  0, device   3, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 11.
      BAR0: I/O at 0xc060 [0xc07f].
      BAR1: 32 bit memory at 0xfebd1000 [0xfebd1fff].
      BAR4: 64 bit prefetchable memory at 0xfe000000 [0xfe003fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net0"
  Bus  0, device   4, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc000 [0xc03f].
      BAR1: 32 bit memory at 0xfebd2000 [0xfebd2fff].
      BAR4: 64 bit prefetchable memory at 0xfe004000 [0xfe007fff].
      id "virtio-disk0"
  Bus  0, device   5, function 0:
    Class 0255: PCI device 1af4:1002
      IRQ 10.
      BAR0: I/O at 0xc080 [0xc09f].
      BAR4: 64 bit prefetchable memory at 0xfe008000 [0xfe00bfff].
      id "balloon0"
  Bus  0, device   6, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1000 [0x101f].
      BAR1: 32 bit memory at 0x40044000 [0x40044fff].
      BAR4: 64 bit prefetchable memory at 0x40040000 [0x40043fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net1"
  Bus  0, device   7, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1020 [0x103f].
      BAR1: 32 bit memory at 0x40045000 [0x40045fff].
      BAR4: 64 bit prefetchable memory at 0x40048000 [0x4004bfff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net2"
  Bus  0, device   8, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1040 [0x105f].
      BAR1: 32 bit memory at 0x40046000 [0x40046fff].
      BAR4: 64 bit prefetchable memory at 0x4004c000 [0x4004ffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net3"
  Bus  0, device   9, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1060 [0x107f].
      BAR1: 32 bit memory at 0x40047000 [0x40047fff].
      BAR4: 64 bit prefetchable memory at 0x40050000 [0x40053fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net4"
  Bus  0, device  10, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1080 [0x109f].
      BAR1: 32 bit memory at 0x40058000 [0x40058fff].
      BAR4: 64 bit prefetchable memory at 0x40054000 [0x40057fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net5"
  Bus  0, device  11, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x10a0 [0x10bf].
      BAR1: 32 bit memory at 0x40059000 [0x40059fff].
      BAR4: 64 bit prefetchable memory at 0x4005c000 [0x4005ffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net6"
  Bus  0, device  12, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x10c0 [0x10df].
      BAR1: 32 bit memory at 0x4005a000 [0x4005afff].
      BAR4: 64 bit prefetchable memory at 0x40060000 [0x40063fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net7"
  Bus  0, device  13, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x10e0 [0x10ff].
      BAR1: 32 bit memory at 0x4005b000 [0x4005bfff].
      BAR4: 64 bit prefetchable memory at 0x40064000 [0x40067fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net8"
  Bus  0, device  14, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1400 [0x141f].
      BAR1: 32 bit memory at 0x4006c000 [0x4006cfff].
      BAR4: 64 bit prefetchable memory at 0x40068000 [0x4006bfff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net9"
  Bus  0, device  15, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1420 [0x143f].
      BAR1: 32 bit memory at 0x4006d000 [0x4006dfff].
      BAR4: 64 bit prefetchable memory at 0x40070000 [0x40073fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net10"
  Bus  0, device  16, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1440 [0x145f].
      BAR1: 32 bit memory at 0x4006e000 [0x4006efff].
      BAR4: 64 bit prefetchable memory at 0x40074000 [0x40077fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net11"
  Bus  0, device  17, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1460 [0x147f].
      BAR1: 32 bit memory at 0x4006f000 [0x4006ffff].
      BAR4: 64 bit prefetchable memory at 0x40078000 [0x4007bfff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net12"
  Bus  0, device  18, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1480 [0x149f].
      BAR1: 32 bit memory at 0x40380000 [0x40380fff].
      BAR4: 64 bit prefetchable memory at 0x4007c000 [0x4007ffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net13"
  Bus  0, device  19, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x14a0 [0x14bf].
      BAR1: 32 bit memory at 0x40381000 [0x40381fff].
      BAR4: 64 bit prefetchable memory at 0x40384000 [0x40387fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net14"
  Bus  0, device  20, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x14c0 [0x14df].
      BAR1: 32 bit memory at 0x40382000 [0x40382fff].
      BAR4: 64 bit prefetchable memory at 0x40388000 [0x4038bfff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net15"
  Bus  0, device  21, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x14e0 [0x14ff].
      BAR1: 32 bit memory at 0x40383000 [0x40383fff].
      BAR4: 64 bit prefetchable memory at 0x4038c000 [0x4038ffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net16"
  Bus  0, device  22, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1800 [0x181f].
      BAR1: 32 bit memory at 0x40394000 [0x40394fff].
      BAR4: 64 bit prefetchable memory at 0x40390000 [0x40393fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net17"
  Bus  0, device  23, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1820 [0x183f].
      BAR1: 32 bit memory at 0x40395000 [0x40395fff].
      BAR4: 64 bit prefetchable memory at 0x40398000 [0x4039bfff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net18"
  Bus  0, device  24, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1840 [0x185f].
      BAR1: 32 bit memory at 0x40396000 [0x40396fff].
      BAR4: 64 bit prefetchable memory at 0x4039c000 [0x4039ffff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net19"
  Bus  0, device  25, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1860 [0x187f].
      BAR1: 32 bit memory at 0x40397000 [0x40397fff].
      BAR4: 64 bit prefetchable memory at 0x403a0000 [0x403a3fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net20"
  Bus  0, device  26, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1880 [0x189f].
      BAR1: 32 bit memory at 0x403a8000 [0x403a8fff].
      BAR4: 64 bit prefetchable memory at 0x403a4000 [0x403a7fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net21"
  Bus  0, device  27, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x18a0 [0x18bf].
      BAR1: 32 bit memory at 0x403a9000 [0x403a9fff].
      BAR4: 64 bit prefetchable memory at 0x403ac000 [0x403affff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net22"
  Bus  0, device  28, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x18c0 [0x18df].
      BAR1: 32 bit memory at 0x403aa000 [0x403aafff].
      BAR4: 64 bit prefetchable memory at 0x403b0000 [0x403b3fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net23"
  Bus  0, device  29, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x18e0 [0x18ff].
      BAR1: 32 bit memory at 0x403ab000 [0x403abfff].
      BAR4: 64 bit prefetchable memory at 0x403b4000 [0x403b7fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net24"
  Bus  0, device  30, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1c00 [0x1c1f].
      BAR1: 32 bit memory at 0x403bc000 [0x403bcfff].
      BAR4: 64 bit prefetchable memory at 0x403b8000 [0x403bbfff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net25"
  Bus  0, device  31, function 0:
    Ethernet controller: PCI device 1af4:1000
      IRQ 0.
      BAR0: I/O at 0x1c20 [0x1c3f].
      BAR1: 32 bit memory at 0x403bd000 [0x403bdfff].
      BAR4: 64 bit prefetchable memory at 0x40700000 [0x40703fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id "net26"

compute-0:/home/wrsroot# virsh dumpxml instance-0000003d
<domain type='kvm' id='11'>
  <name>instance-0000003d</name>
  <uuid>7a3df365-5a01-41d2-bf6e-a4839195fecb</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="16.0.2-1.tis.5"/>
      <nova:name>tenant2-if_attach_tenant-2</nova:name>
      <nova:creationTime>2018-09-07 19:35:37</nova:creationTime>
      <nova:flavor name="flavor-1">
        <nova:memory>1024</nova:memory>
        <nova:disk>2</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="222f031c753f40a3a91b1d9b7072ef62">tenant2</nova:user>
        <nova:project uuid="340a1201df5a43b2960137c40456f524">tenant2</nova:project>
      </nova:owner>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB' nodeset='0'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4-23,52-71'/>
    <emulatorpin cpuset='4-23,52-71'/>
  </cputune>
  <numatune>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Fedora Project</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>16.0.2-1.tis.5</entry>
      <entry name='serial'>669bedc8-48c1-47de-ba20-d7f323614df5</entry>
      <entry name='uuid'>7a3df365-5a01-41d2-bf6e-a4839195fecb</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.4.0'>hvm</type>
    <boot dev='hd'/>
    <bios rebootTimeout='5000'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu>
    <topology sockets='1' cores='1' threads='1'/>
    <numa>
      <cell id='0' cpus='0' memory='1048576' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdc'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>17851e43-e880-4e2b-aea5-27eb3dd0d7c8</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:f3:57:95'/>
      <source type='unix' path='/var/run/openvswitch/vhu21543a75-82' mode='server'/>
      <target dev='vhu21543a75-82'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:31:56:c4'/>
      <source type='unix' path='/var/run/openvswitch/vhu9af4698f-2d' mode='server'/>
      <target dev='vhu9af4698f-2d'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:b9:c1:91'/>
      <source type='unix' path='/var/run/openvswitch/vhu019f7da4-55' mode='server'/>
      <target dev='vhu019f7da4-55'/>
      <model type='virtio'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:f1:1f:52'/>
      <source type='unix' path='/var/run/openvswitch/vhub7e5c1ae-31' mode='server'/>
      <target dev='vhub7e5c1ae-31'/>
      <model type='virtio'/>
      <alias name='net3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:52:21:8c'/>
      <source type='unix' path='/var/run/openvswitch/vhu3222d70c-05' mode='server'/>
      <target dev='vhu3222d70c-05'/>
      <model type='virtio'/>
      <alias name='net4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:7d:23:7f'/>
      <source type='unix' path='/var/run/openvswitch/vhuf2c34258-4a' mode='server'/>
      <target dev='vhuf2c34258-4a'/>
      <model type='virtio'/>
      <alias name='net5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:70:ee:98'/>
      <source type='unix' path='/var/run/openvswitch/vhu827ef24b-b5' mode='server'/>
      <target dev='vhu827ef24b-b5'/>
      <model type='virtio'/>
      <alias name='net6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:66:34:6b'/>
      <source type='unix' path='/var/run/openvswitch/vhu8388da8c-e7' mode='server'/>
      <target dev='vhu8388da8c-e7'/>
      <model type='virtio'/>
      <alias name='net7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:b8:f5:11'/>
      <source type='unix' path='/var/run/openvswitch/vhu18f2dd08-e6' mode='server'/>
      <target dev='vhu18f2dd08-e6'/>
      <model type='virtio'/>
      <alias name='net8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:49:dd:31'/>
      <source type='unix' path='/var/run/openvswitch/vhu67eade13-ee' mode='server'/>
      <target dev='vhu67eade13-ee'/>
      <model type='virtio'/>
      <alias name='net9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:f9:e0:b9'/>
      <source type='unix' path='/var/run/openvswitch/vhu39e8d10a-81' mode='server'/>
      <target dev='vhu39e8d10a-81'/>
      <model type='virtio'/>
      <alias name='net10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:f3:f6:c9'/>
      <source type='unix' path='/var/run/openvswitch/vhu2dd5189e-91' mode='server'/>
      <target dev='vhu2dd5189e-91'/>
      <model type='virtio'/>
      <alias name='net11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:a7:d5:bb'/>
      <source type='unix' path='/var/run/openvswitch/vhude564613-74' mode='server'/>
      <target dev='vhude564613-74'/>
      <model type='virtio'/>
      <alias name='net12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x11' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:a1:e3:13'/>
      <source type='unix' path='/var/run/openvswitch/vhu1a26e20b-ef' mode='server'/>
      <target dev='vhu1a26e20b-ef'/>
      <model type='virtio'/>
      <alias name='net13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x12' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:36:c8:18'/>
      <source type='unix' path='/var/run/openvswitch/vhu27ce5145-26' mode='server'/>
      <target dev='vhu27ce5145-26'/>
      <model type='virtio'/>
      <alias name='net14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x13' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:9d:2d:7e'/>
      <source type='unix' path='/var/run/openvswitch/vhu8b961c0f-b0' mode='server'/>
      <target dev='vhu8b961c0f-b0'/>
      <model type='virtio'/>
      <alias name='net15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x14' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:5a:38:66'/>
      <source type='unix' path='/var/run/openvswitch/vhuae2b9f31-fe' mode='server'/>
      <model type='virtio'/>
      <alias name='net16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x15' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:81:e5:5d'/>
      <source type='unix' path='/var/run/openvswitch/vhu4eac11cd-21' mode='server'/>
      <model type='virtio'/>
      <alias name='net17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x16' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:1b:15:a5'/>
      <source type='unix' path='/var/run/openvswitch/vhu8dcd4eea-9f' mode='server'/>
      <model type='virtio'/>
      <alias name='net18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x17' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:b1:41:96'/>
      <source type='unix' path='/var/run/openvswitch/vhud89afdbe-53' mode='server'/>
      <model type='virtio'/>
      <alias name='net19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x18' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:14:71:c1'/>
      <source type='unix' path='/var/run/openvswitch/vhu162393a7-b1' mode='server'/>
      <model type='virtio'/>
      <alias name='net20'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:73:1b:2d'/>
      <source type='unix' path='/var/run/openvswitch/vhuaf73b2be-93' mode='server'/>
      <model type='virtio'/>
      <alias name='net21'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:21:5e:14'/>
      <source type='unix' path='/var/run/openvswitch/vhu659d4531-c7' mode='server'/>
      <model type='virtio'/>
      <alias name='net22'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:35:2b:e6'/>
      <source type='unix' path='/var/run/openvswitch/vhu1631d0cd-4c' mode='server'/>
      <model type='virtio'/>
      <alias name='net23'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1c' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:3a:f8:8e'/>
      <source type='unix' path='/var/run/openvswitch/vhu218f84d5-7d' mode='server'/>
      <model type='virtio'/>
      <alias name='net24'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:54:da:3f'/>
      <source type='unix' path='/var/run/openvswitch/vhue3a99291-8c' mode='server'/>
      <model type='virtio'/>
      <alias name='net25'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='fa:16:3e:04:bc:9d'/>
      <source type='unix' path='/var/run/openvswitch/vhu3ad9a908-06' mode='server'/>
      <model type='virtio'/>
      <alias name='net26'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <log file='/etc/nova/instances/7a3df365-5a01-41d2-bf6e-a4839195fecb/console.log' append='off'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <log file='/etc/nova/instances/7a3df365-5a01-41d2-bf6e-a4839195fecb/console.log' append='off'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2018-09-07:

This is a libvirt issue. Given that we are in the process of upgrading libvirt to a new version, there is no plan to investigate this issue.

The upgrade is tracked via:
https://storyboard.openstack.org/#!/story/2003396

Changed in starlingx:
importance:	Undecided → Low
assignee:	nobody → Joseph Richard (josephrichard)
status:	New → Won't Fix

Ghada Khalil (gkhalil) on 2018-10-05

Changed in starlingx:
status:	Won't Fix → New

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2018-10-05:

Re-opening as Senthil has reported that he saw the issue again with the upgraded version of libvirt.

Seen on StarlingX master as of 2018-10-03_20-18-00
Reproducible (5/5 times)

Changed in starlingx:
importance:	Low → Medium
assignee:	Joseph Richard (josephrichard) → Jim Somerville (jsomervi)

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2018-10-05:

Not gating stx.2018.10 as it's a specific scenario with interface attach/detach. Targeting stx.2019.03

Changed in starlingx:
status:	New → Triaged
tags:	added: stx.2019.03

Ken Young (kenyis) on 2019-01-18

tags:

added: stx.2019.05
removed: stx.2019.03

Revision history for this message

Jim Somerville (jsomervi) wrote on 2019-02-27:

The problem is not with libvirt but is with nova. This upstream nova commit fixes it, and we'll get it once we cut over to stein.

commit a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121
Author: arches <email address hidden>
Date: Thu Dec 27 17:25:48 2018 +0200

Skip checking of target_dev for vhostuser

    Nova skips detaching of ovs dpdk interfaces
    thinking that it's already detached because
    get_interface_by_cfg() return no inteface.
    This is due to _set_config_VIFVHostUser()
    not setting target_dev in configuration while
    LibvirtConfigGuestInterface sets target_dev
    if tag "target" is found in the interface.

    As target_dev is not a valid value for
    vhostuser interface, it will not be checked
    for vhostuser type.

Change-Id: Iaf185b98c236df47e44cda0732ee0aed1fd6323d
Closes-Bug: #1807340

Revision history for this message

Jim Somerville (jsomervi) wrote on 2019-02-27:

Issue fixed in nova, we will get it once we cut over to stein.

Changed in starlingx:
status:	Triaged → Fix Committed

Ghada Khalil (gkhalil) on 2019-03-01

tags:

added: stx.distro.openstack

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-03-05:

Marking as Fix Released. StarlingX was cutover to Stein on March 1/2019.

Changed in starlingx:
status:	Fix Committed → Fix Released

Ken Young (kenyis) on 2019-04-05

tags:

added: stx.2.0
removed: stx.2019.05

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.