Interface attach cli rejected after 15 vnics attach, detach and attach again

Bug #1790968 reported by Senthil Mukundakumar on 2018-09-05
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Medium
Jim Somerville

Bug Description

Brief Description
-----------------
Interface attach cli rejected after attach 15 vnics, live migrate, detach 15 vnics, attach again at vnic13

Severity
--------
Major

Steps to Reproduce
------------------
        - Boot a vm with only mgmt interface
        - Attach an 15 vifs to vm with given net-id and vif_model:virtio
        - Bring up the interface from vm
        - ping between base_vm and vm_under_test over mgmt & tenant network
        - Perform VM action - live migrate
        - Verify ping between base_vm and vm_under_test over mgmt & tenant network after vm operation
        - detach all the tenant interface
        - Again attach an 15 vifs to vm with given net-id and vif_model:virtio

Expected Behavior
------------------
All 15 vifs should be attachable

Actual Behavior
----------------
After 13 vif attachment then the cli rejected.
E Details: CLI 'nova --os-username 'tenant2' --os-password 'Li69nux*' --os-project-name tenant2 --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-region-name RegionOne interface-attach --net-id 0b736d87-413a-4fb8-a6c0-58800f9ba0c4 --wrs-if:vif_model virtio 9ec7cfa8-18f4-4baa-9801-d980f25fb34a' failed to execute. Output: ERROR (ClientException): Failed to attach network adapter device to 9ec7cfa8-18f4-4baa-9801-d980f25fb34a (HTTP 500) (Request-ID: req-a09e4e55-6e89-47d7-a0d4-b45bc2ba69c5)

Reproducibility
---------------
Reproducible (2 out of 2 times)

System Configuration
--------------------
Any

Branch/Pull Time/Commit
-----------------------
StarlingX master as of 2018-08-31_20-18-00

Ghada Khalil (gkhalil) on 2018-09-06
tags: added: stx.networking
description: updated
Joseph Richard (josephrichard) wrote :
Download full text (27.5 KiB)

I looked through the logs briefly. The failure is with the request from nova-compute to libvirt.

2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [req-a09e4e55-6e89-47d7-a0d4-b45bc2ba69c5 204c0a61b28f40b3bcbd1805179a8ae5 bb5b3bb7440d4cbc8b279e1f895551e2 - default default] [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] attaching network adapter failed.: libvirtError: internal error: No more available PCI slots
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] Traceback (most recent call last):
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1642, in attach_interface
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] guest.attach_device(cfg, persistent=True, live=live)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 322, in attach_device
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] self._domain.attachDeviceFlags(device_xml, flags=flags)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] result = proxy_call(self._autowrap, f, *args, **kwargs)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] rv = execute(f, *args, **kwargs)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] six.reraise(c, e, tb)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] rv = meth(*args, **kwargs)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 587, in attachDeviceFlags
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2018-09-02 01:01:36.627 57065 ERROR nova.virt.libvirt.driver [instance: 9ec7cfa8-18f4-4baa-9801-d980f25fb34a] lib...

Ghada Khalil (gkhalil) wrote :

This is a libvirt issue. Given that we are in the process of upgrading libvirt to a new version, there is no plan to investigate this issue.

The upgrade is tracked via:
https://storyboard.openstack.org/#!/story/2003396

Changed in starlingx:
importance: Undecided → Low
assignee: nobody → Joseph Richard (josephrichard)
status: New → Won't Fix
Ghada Khalil (gkhalil) on 2018-10-05
Changed in starlingx:
status: Won't Fix → New
Ghada Khalil (gkhalil) wrote :

Re-opening as Senthil has reported that he saw the issue again with the upgraded version of libvirt.

Seen on StarlingX master as of 2018-10-03_20-18-00
Reproducible (5/5 times)

Changed in starlingx:
importance: Low → Medium
assignee: Joseph Richard (josephrichard) → Jim Somerville (jsomervi)
Ghada Khalil (gkhalil) wrote :

Not gating stx.2018.10 as it's a specific scenario with interface attach/detach. Targeting stx.2019.03

Changed in starlingx:
status: New → Triaged
tags: added: stx.2019.03
Ken Young (kenyis) on 2019-01-18
tags: added: stx.2019.05
removed: stx.2019.03
Jim Somerville (jsomervi) wrote :

The problem is not with libvirt but is with nova. This upstream nova commit fixes it, and we'll get it once we cut over to stein.

commit a19c38a6ab13cdf2509a1f9f9d39c7f0a70ba121
Author: arches <email address hidden>
Date: Thu Dec 27 17:25:48 2018 +0200

    Skip checking of target_dev for vhostuser

    Nova skips detaching of ovs dpdk interfaces
    thinking that it's already detached because
    get_interface_by_cfg() return no inteface.
    This is due to _set_config_VIFVHostUser()
    not setting target_dev in configuration while
    LibvirtConfigGuestInterface sets target_dev
    if tag "target" is found in the interface.

    As target_dev is not a valid value for
    vhostuser interface, it will not be checked
    for vhostuser type.

    Change-Id: Iaf185b98c236df47e44cda0732ee0aed1fd6323d
    Closes-Bug: #1807340

Jim Somerville (jsomervi) wrote :

Issue fixed in nova, we will get it once we cut over to stein.

Changed in starlingx:
status: Triaged → Fix Committed
Ghada Khalil (gkhalil) on 2019-03-01
tags: added: stx.distro.openstack
Ghada Khalil (gkhalil) wrote :

Marking as Fix Released. StarlingX was cutover to Stein on March 1/2019.

Changed in starlingx:
status: Fix Committed → Fix Released
Ken Young (kenyis) on 2019-04-05
tags: added: stx.2.0
removed: stx.2019.05
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers