Fuel for OpenStack

Deployment of master node failed with 'storage volume exists already'

Bug #1566966 reported by Alexandr Kostrikov on 2016-04-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Dmitry Belyaninov	Fuel for OpenStack 10.0
	Mitaka	Fix Committed	High	Dmitry Belyaninov	Fuel for OpenStack 9.2

Bug Description

Detailed information:
On swarm run [0] we have a failure in installation of master node at first step:
        Snapshot: empty
libvirt: Storage Driver error : cannot unlink file '(null)': Bad address
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
libvirt: Storage Driver error : storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already
2016-04-05 21:03:21,154 - ERROR __init__.py:66 -- get_admin_node_ip raised: libvirtError("storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already",)
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_1/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_1/fuelweb_test/models/environment.py", line 144, in get_admin_node_ip
    self.d_env.nodes(
  File "/home/jenkins/workspace/9.0.system_test.ubuntu.thread_1/fuelweb_test/models/environment.py", line 216, in d_env
    boot_from=settings.ADMIN_BOOT_DEVICE)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 244, in describe_environment
    environment = cls.create_environment(config)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 283, in create_environment
    environment.create_node(config_node)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 385, in create_node
    new_vol.define()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/volume.py", line 34, in define
    self.driver.volume_define(self)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/driver/libvirt/libvirt_driver.py", line 897, in volume_define
    self.xml_builder.build_volume_xml(volume), 0)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/libvirt.py", line 2846, in createXML
    if ret is None:raise libvirtError('virStorageVolCreateXML() failed', pool=self)
libvirtError: storage volume '9.0.system_test.ubuntu.thread_1.69.69_slave-06-swift' exists already

Tags:

Revision history for this message

Alexandr Kostrikov (akostrikov-mirantis) wrote on 2016-04-06:

Possible duplicate of https://bugs.launchpad.net/fuel/+bug/1326457

Changed in fuel:
importance:	Undecided → High

Oleksiy Molchanov (omolchanov) on 2016-04-07

Changed in fuel:
status:	New → Confirmed

Fuel Devops McRobotson (fuel-devops-robot) on 2016-04-19

Changed in fuel:
milestone:	9.0 → 10.0

Revision history for this message

Nastya Urlapova (aurlapova) wrote on 2016-05-13:

Passed on ISO 9.0-mos-354: https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.thread_1/107/testReport/(root)/setup_master/

Changed in fuel:
status:	Confirmed → Invalid

Revision history for this message

Andrey Lavrentyev (alavrentyev) wrote on 2016-08-25:

Swarm failure with the same symptoms: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.bonding_ha_one_controller/39/testReport/%28root%29/setup_master/

It affects 9.1 version

9.1 snapshot #177

Setting status back to Confirmed

Changed in fuel:
status:	Invalid → Confirmed

Revision history for this message

Alexander Gubanov (ogubanov) wrote on 2016-08-29:

I faced with the same issue on MOS QA CI http://cz7776.bud.mirantis.net:8080/jenkins/job/9.x_Ironic/83/consoleFull

9.1 snapshot #193

Timur Nurlygayanov (tnurlygayanov) on 2016-08-30

Changed in fuel:
assignee:	Fuel QA Team (fuel-qa) → Fuel Sustaining (fuel-sustaining-team)

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-08-30:

The issue was fixed on MOS CI by updating the images, Alexander Gubanov knows more details.

Revision history for this message

Alexander Gubanov (ogubanov) wrote on 2016-08-30:

I removed files from /var/lib/libvirt/images/ after that deploy is passed.

Kyrylo Galanov (kgalanov) on 2016-08-31

Changed in fuel:
assignee:	Fuel Sustaining (fuel-sustaining-team) → Fuel Infra Apps (fuel-infra-apps)

Revision history for this message

Alexander Charykov (acharykov) wrote on 2016-09-22:

Doesn't belongs to infra-apps-team.

Changed in fuel:
assignee:	Fuel Infra Apps (fuel-infra-apps) → nobody

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-10-04:

The issue affects automated tests and doesn't affect the product.

tags:

added: non-release

Revision history for this message

Alexandra (aallakhverdieva) wrote on 2016-10-05:

Occurs on 9.1 snapshot #347
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.multirole/84/testReport/(root)/setup_master/

Revision history for this message

Alexandra (aallakhverdieva) wrote on 2016-10-06:

#10

New fail:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.thread_3/85/testReport/(root)/setup_master/

Revision history for this message

Alexandra (aallakhverdieva) wrote on 2016-10-17:

#11

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.bonding_ha_one_controller/148/testReport/(root)/setup_master/

Dmitry Pyzhov (dpyzhov) on 2016-12-05

Changed in fuel:
assignee:	nobody → Fuel QA Team (fuel-qa)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-12-08: Related fix proposed to fuel-qa (stable/mitaka)

#12

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/408649

Revision history for this message

Dmitry Belyaninov (dbelyaninov) wrote on 2016-12-14:

#13

Fix was merged to "stable/mitaka"
We need some time to check that the bug will not be reproduced somewhere.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-12-23: Related fix proposed to fuel-qa (stable/newton)

#14

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/414450

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-01-27: Related fix proposed to fuel-qa (master)

#15

Related fix proposed to branch: master
Review: https://review.openstack.org/426280

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-08: Related fix merged to fuel-qa (stable/newton)

#16

Reviewed: https://review.openstack.org/414450
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=1a5c0468e2b3a53a074fcd335a4b615c440e7c26
Submitter: Jenkins
Branch: stable/newton

commit 1a5c0468e2b3a53a074fcd335a4b615c440e7c26
Author: Dmitry Belyaninov <email address hidden>
Date: Thu Dec 8 14:43:24 2016 +0000

Environment creation procedure modification

    Sometimes installation of environment failed
    because storage volume(s) exists already.
    The issue is hard-to-reproduce. Some logging and
    setting internal attribute _virt_env were added.

    Change-Id: Ib49c904ca7c961617f0f328230c2658a13bfdecf
    Related-Bug: 1566966
    (cherry picked from commit 1b2d807336f41d0d319f273790c708ce2929d1c5)

tags:

added: in-stable-newton

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-08: Related fix merged to fuel-qa (master)

#17

Reviewed: https://review.openstack.org/426280
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=f6dbf49b7b7821c07d33ba8504a1b189bb2686ae
Submitter: Jenkins
Branch: master

commit f6dbf49b7b7821c07d33ba8504a1b189bb2686ae
Author: Dmitry Belyaninov <email address hidden>
Date: Thu Dec 8 14:43:24 2016 +0000

Environment creation procedure modification

    Sometimes installation of environment failed
    because storage volume(s) exists already.
    The issue is hard-to-reproduce. Some logging and
    setting internal attribute _virt_env were added.

    Change-Id: Ib49c904ca7c961617f0f328230c2658a13bfdecf
    Related-Bug: 1566966
    (cherry picked from commit 1b2d807336f41d0d319f273790c708ce2929d1c5)

Dmitry Belyaninov (dbelyaninov) on 2017-02-09

Changed in fuel:
status:	Confirmed → Fix Committed
assignee:	Fuel QA Team (fuel-qa) → Dmitry Belyaninov (dbelyaninov)

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.