[qa][fuel-devops] Environment creation fails: 'libvirt: error : Unable to create tap device virnet0: Device or resource busy'

Bug #1590419 reported by Artem Panchenko
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Anton Studenov

Bug Description

fuel-devops 3.0 (master branch)

Environment (master node) creation fails with the following error:

======================================================================
ERROR: Create environment and set up master node
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/fuel-devops-3/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/fuel-devops-3/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/fuel-devops-3/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0.fuel.baremetal.beta/fuelweb_test/helpers/decorators.py", line 121, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.fuel.baremetal.beta/fuelweb_test/tests/base_test_case.py", line 358, in setup_master
    self.env.setup_environment()
  File "/home/jenkins/workspace/9.0.fuel.baremetal.beta/fuelweb_test/models/environment.py", line 399, in setup_environment
    self.d_env.start([admin])
  File "/home/jenkins/fuel-devops-3/local/lib/python2.7/site-packages/devops/models/environment.py", line 180, in start
    group.start_nodes(nodes)
  File "/home/jenkins/fuel-devops-3/local/lib/python2.7/site-packages/devops/models/group.py", line 106, in start_nodes
    node.start()
  File "/home/jenkins/fuel-devops-3/local/lib/python2.7/site-packages/devops/driver/libvirt/libvirt_driver.py", line 950, in start
    self.create()
  File "/home/jenkins/fuel-devops-3/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/fuel-devops-3/local/lib/python2.7/site-packages/devops/driver/libvirt/libvirt_driver.py", line 955, in create
    self._libvirt_node.create()
  File "/home/jenkins/fuel-devops-3/local/lib/python2.7/site-packages/libvirt.py", line 870, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: Unable to create tap device virnet0: Device or resource busy

fuel-devops tries to use 'virnet0' device name, but it's already used by another environment.

The problem is here:

https://github.com/openstack/fuel-devops/blob/8667612c798ee461e3f3da677cd4a3b884922fca/devops/driver/libvirt/libvirt_driver.py#L261-L264

For net_virnet* device listCaps() returns an empty list:

>>> c
<libvirt.virNodeDevice object at 0x7feeaaf62f90>
>>> print c.XMLDesc()
<device>
  <name>net_virnet0_fe_19_31_c7_76_9c</name>
  <path>/sys/devices/virtual/net/virnet0</path>
  <parent>computer</parent>
  <capability type='net'>
    <interface>virnet0</interface>
    <address>fe:19:31:c7:76:9c</address>
    <capability type='80203'/>
  </capability>
</device>

>>> c.listCaps()
[]

>>> env.nodes().admin.driver.get_allocated_device_names()
['virbr0', 'virbr1', 'virbr2', 'virbr3', 'virbr4', 'virbr5', 'virbr6', 'virbr7', 'virbr8', 'virbr9', 'virbr10']

After commenting those lines everything works fine:

http://paste.openstack.org/show/508916/

>>> env.nodes().admin.driver.get_allocated_device_names()
['eno1', 'eno2', 'lo', 'virbr1-nic', 'virbr2-nic', 'virbr3-nic', 'virbr4-nic', 'virbr5-nic', 'virnet0', 'virbr6-nic', 'virbr7-nic', 'virbr8-nic', 'virbr9-nic', 'virbr10-nic', 'virbr0', 'virbr1', 'virbr2', 'virbr3', 'virbr4', 'virbr5', 'virbr6', 'virbr7', 'virbr8', 'virbr9', 'virbr10']

Tags: area-qa
Changed in fuel:
status: New → Confirmed
Revision history for this message
Anton Studenov (astudenov) wrote :

Fuel-devops 3.0 requires libvirt >=1.2.3. With libvirt 1.2.2 there is impossible
to run more than one environment at the same time because of conflicting names of
network interfaces (virnet*). It is fixed on libvirt >=1.2.3 , but will require
to re-create already existing environments.

Duplicate of https://bugs.launchpad.net/fuel/+bug/1586385

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-devops (master)

Fix proposed to branch: master
Review: https://review.openstack.org/328295

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Anton Studenov (astudenov)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-devops (master)

Reviewed: https://review.openstack.org/328295
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=0953908ca61b10b046eb4a1cec535bf734d4ed5e
Submitter: Jenkins
Branch: master

commit 0953908ca61b10b046eb4a1cec535bf734d4ed5e
Author: Anton Studenov <email address hidden>
Date: Fri Jun 10 16:23:29 2016 +0300

    Fix get_allocated_device_names

    - remove check for network devices.

    Change-Id: I78359cebaacd58feaec7be70c882815fc474307d
    Closes-Bug: #1590419
    Closes-Bug: #1586385

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Sergey Yudin (tsipa740) wrote :

Does that mean devops 3.0 is not compatible with libvirt 1.2.2 or this commit is supposed to fix the issue?

I'm still observing the same error

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-devops (master)

Fix proposed to branch: master
Review: https://review.openstack.org/333805

Revision history for this message
Sergey Yudin (tsipa740) wrote :

Even with last patchset applied i'm observing this issue.

Changed in fuel:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-devops (release/2.9)

Related fix proposed to branch: release/2.9
Review: https://review.openstack.org/336057

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-devops (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/336064

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-devops (master)

Change abandoned by Anton Studenov (<email address hidden>) on branch: master
Review: https://review.openstack.org/333805

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-devops (release/2.9)

Reviewed: https://review.openstack.org/336057
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=1d0d2655a8f71d4a9deb39725abac8ff1d5181f7
Submitter: Jenkins
Branch: release/2.9

commit 1d0d2655a8f71d4a9deb39725abac8ff1d5181f7
Author: Anton Studenov <email address hidden>
Date: Thu Jun 30 17:09:15 2016 +0300

    Use libvirt generated bridge and interafece names

    - This patch fixes race condition issue with creating of multiple
      environments at once.
    - Removed all code that looks for available device names

    Change-Id: I38287ced50beb0c2a484021b6c96d1a970f8dd4b
    Related-Bug: #1590419
    Implements: blueprint fuel-devops-multiple-databases

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-devops (release/2.9)

Related fix proposed to branch: release/2.9
Review: https://review.openstack.org/339811

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-devops (release/2.9)

Reviewed: https://review.openstack.org/339811
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=1955f25c4b531cd13d588ec82840801c46e20162
Submitter: Jenkins
Branch: release/2.9

commit 1955f25c4b531cd13d588ec82840801c46e20162
Author: Anton Studenov <email address hidden>
Date: Fri Jul 8 23:42:34 2016 +0300

    Set network bridge names manually

    - libvirt bridge name generator allows to create only 256 names
      so this patch returns some logic for manual generation of
      bridge names.
    - Node network interface names are still generated by libvirt
      (and limited to 65536 names)

    Change-Id: Ife31acd300466f660f7525e8f4fd6927ae1e2158
    Related-Bug: #1590419
    Implements: blueprint fuel-devops-multiple-databases

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-devops (master)

Reviewed: https://review.openstack.org/336064
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=561ec34363ac8d42475cc441175c3298dcdda457
Submitter: Jenkins
Branch: master

commit 561ec34363ac8d42475cc441175c3298dcdda457
Author: Anton Studenov <email address hidden>
Date: Fri Jul 8 21:37:07 2016 +0300

    Use libvirt generated interafece names

    - This patch fixes race condition issue with creating of multiple
      environments at once.

    Change-Id: I349ece62509c3e9da974dcb68c389789b1baba66
    Related-Bug: #1590419
    Implements: blueprint fuel-devops-multiple-databases

Changed in fuel:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.