Deployment of master node failed with 'storage volume exists already'

Bug #1566966 reported by Alexandr Kostrikov on 2016-04-06
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Dmitry Belyaninov
Mitaka
High
Dmitry Belyaninov

Bug Description

Detailed information:
On swarm run [0] we have a failure in installation of master node at first step:
        Snapshot: empty
libvirt: Storage Driver error : cannot unlink file '(null)': Bad address
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
2016-04-05 21:03:21,154 - ERROR __init__.py:66 -- get_admin_node_ip raised: libvirtError("storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already",)
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_1/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_1/fuelweb_test/models/environment.py", line 144, in get_admin_node_ip
    self.d_env.nodes(
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_1/fuelweb_test/models/environment.py", line 216, in d_env
    boot_from=settings.ADMIN_BOOT_DEVICE)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 244, in describe_environment
    environment = cls.create_environment(config)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 283, in create_environment
    environment.create_node(config_node)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 385, in create_node
    new_vol.define()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/volume.py", line 34, in define
    self.driver.volume_define(self)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/driver/libvirt/libvirt_driver.py", line 897, in volume_define
    self.xml_builder.build_volume_xml(volume), 0)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/libvirt.py", line 2846, in createXML
    if ret is None:raise libvirtError('virStorageVolCreateXML() failed', pool=self)
libvirtError: storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already

Changed in fuel:
importance: Undecided → High
Changed in fuel:
status: New → Confirmed
Changed in fuel:
milestone: 9.0 → 10.0
Andrey Lavrentyev (alavrentyev) wrote :

Swarm failure with the same symptoms: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.bonding_ha_one_controller/39/testReport/%28root%29/setup_master/

It affects 9.1 version

9.1 snapshot #177

Setting status back to Confirmed

Changed in fuel:
status: Invalid → Confirmed
Alexander Gubanov (ogubanov) wrote :

I faced with the same issue on MOS QA CI http://cz7776.bud.mirantis.net:8080/jenkins/job/9.x_Ironic/83/consoleFull

9.1 snapshot #193

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Fuel Sustaining (fuel-sustaining-team)

The issue was fixed on MOS CI by updating the images, Alexander Gubanov knows more details.

Alexander Gubanov (ogubanov) wrote :

I removed files from /var/lib/libvirt/images/ after that deploy is passed.

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Fuel Infra Apps (fuel-infra-apps)
Alexander Charykov (acharykov) wrote :

Doesn't belongs to infra-apps-team.

Changed in fuel:
assignee: Fuel Infra Apps (fuel-infra-apps) → nobody

The issue affects automated tests and doesn't affect the product.

tags: added: non-release
Dmitry Pyzhov (dpyzhov) on 2016-12-05
Changed in fuel:
assignee: nobody → Fuel QA Team (fuel-qa)

Fix was merged to "stable/mitaka"
We need some time to check that the bug will not be reproduced somewhere.

Reviewed: https://review.openstack.org/414450
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=1a5c0468e2b3a53a074fcd335a4b615c440e7c26
Submitter: Jenkins
Branch: stable/newton

commit 1a5c0468e2b3a53a074fcd335a4b615c440e7c26
Author: Dmitry Belyaninov <email address hidden>
Date: Thu Dec 8 14:43:24 2016 +0000

    Environment creation procedure modification

    Sometimes installation of environment failed
    because storage volume(s) exists already.
    The issue is hard-to-reproduce. Some logging and
    setting internal attribute _virt_env were added.

    Change-Id: Ib49c904ca7c961617f0f328230c2658a13bfdecf
    Related-Bug: 1566966
    (cherry picked from commit 1b2d807336f41d0d319f273790c708ce2929d1c5)

tags: added: in-stable-newton

Reviewed: https://review.openstack.org/426280
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=f6dbf49b7b7821c07d33ba8504a1b189bb2686ae
Submitter: Jenkins
Branch: master

commit f6dbf49b7b7821c07d33ba8504a1b189bb2686ae
Author: Dmitry Belyaninov <email address hidden>
Date: Thu Dec 8 14:43:24 2016 +0000

    Environment creation procedure modification

    Sometimes installation of environment failed
    because storage volume(s) exists already.
    The issue is hard-to-reproduce. Some logging and
    setting internal attribute _virt_env were added.

    Change-Id: Ib49c904ca7c961617f0f328230c2658a13bfdecf
    Related-Bug: 1566966
    (cherry picked from commit 1b2d807336f41d0d319f273790c708ce2929d1c5)

Changed in fuel:
status: Confirmed → Fix Committed
assignee: Fuel QA Team (fuel-qa) → Dmitry Belyaninov (dbelyaninov)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers