ImageNotFound: error opening image /var/lib/nova/instances/_base/

Bug #1284193 reported by Egor Kotko on 2014-02-24
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Fuel Library (Deprecated)
4.1.x
Medium
Fuel Library (Deprecated)

Bug Description

{"build_id": "2014-02-24_01-17-30", "mirantis": "yes", "build_number": "181", "nailgun_sha": "f786786894acc331a4b53b31f33e373ef95ccdfc", "ostf_sha": "b8f16a0288cbf39e11e0b4a41a3f63e6b87dcc4b", "fuelmain_sha": "9a8f918f0f5113970888c43147cc0c3cd46c956e", "astute_sha": "3d43abeefb60677ce6cae83d31ebbba1ff3cdbe2", "release": "4.1", "fuellib_sha": "e3ea44c3b607f37401a268a91956c9d222a81bab"}

Steps to reproduce:
1. Create environment:
Centos HA 3Controllers, 2 compute+ ceph, Ceph for volumes + images, neutron VLAN
2. Create instance from image
3. Migrate instance via horizon

Actual result:
Instance change status on "ERROR"
Expected result:
Instance migrate.

nova-all.log:

<0>Feb 24 15:20:54 node-5 <BF><182>nova-nova.virt.libvirt.driver INFO: Creating image
<0>Feb 24 15:20:54 node-5 <BF><179>nova-nova.virt.libvirt.imagebackend ERROR: error opening rbd image /var/lib/nova/instances/_base/38d944bd3703c01a49fc408ed9d1af0584fd8244
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 467, in __init__
    read_only=read_only)
  File "/usr/lib/python2.6/site-packages/rbd.py", line 351, in __init__
    raise make_ex(ret, 'error opening image %s at snapshot %s' % (name, snapshot))
ImageNotFound: error opening image /var/lib/nova/instances/_base/38d944bd3703c01a49fc408ed9d1af0584fd8244 at snapshot None
<0>Feb 24 15:20:54 node-5 <BF><179>nova-nova.compute.manager ERROR: Setting instance vm_state to ERROR
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3160, in finish_resize
    disk_info, image)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3128, in _finish_resize
    block_device_info, power_on)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 4627, in finish_migration
    block_device_info=None, inject_files=False)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2395, in _create_image
    project_id=instance['project_id'])
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 177, in cache
    *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 638, in create_image
    self.verify_base_size(base, size)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 218, in verify_base_size
    base_size = self.get_disk_size(base)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 622, in get_disk_size
    with RBDVolumeProxy(self, name) as vol:
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 467, in __init__
    read_only=read_only)
  File "/usr/lib/python2.6/site-packages/rbd.py", line 351, in __init__
    raise make_ex(ret, 'error opening image %s at snapshot %s' % (name, snapshot))
ImageNotFound: error opening image /var/lib/nova/instances/_base/38d944bd3703c01a49fc408ed9d1af0584fd8244 at snapshot None

Egor Kotko (ykotko) wrote :
Changed in fuel:
importance: Undecided → High
assignee: nobody → Fuel Library Team (fuel-library)
Andrew Woodward (xarses) on 2014-02-24
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dmitry Borodaenko (dborodaenko)
Dmitry Borodaenko (angdraug) wrote :

According to the diagnostic snapshot, corosync failed to install on the primary controller:
2014-02-24T13:57:56.081829+00:00 err: (/Stage[corosync_setup]/Corosync/Package[corosync]/ensure) change from absent to present failed: Execution of '/usr/bin/yum -d 0 -e 0 -y install corosync' returned 1:

Dmitry Borodaenko (angdraug) wrote :

corosync failure is irrelevant, there seem to be a logic error in Nova: Image.cache() in imagebackend.py:173 creates image name as a file path, and that eventually ends up being passed to RBD in place of an rbd:// URI.

Changed in fuel:
status: New → Confirmed
Dmitry Borodaenko (angdraug) wrote :

Live migration via Nova CLI with ISO #195 deployed in the same configuration as described in this bug works fine, so the problem is limited to cold migration via Horizon. I think we need to note this as a limitation in 4.1 and postpone resolution of this bug to 5.0.

Changed in fuel:
milestone: 4.1 → 5.0
tags: added: release-notes
Changed in fuel:
assignee: Dmitry Borodaenko (dborodaenko) → Fuel Library Team (fuel-library)
Dmitry Borodaenko (angdraug) wrote :

Icehouse version of Horizon in 5.0 offers a separate "Live Migrate" option in Instances tab for the admin user.

Changed in fuel:
status: Confirmed → Fix Committed
Changed in fuel:
status: Fix Committed → Fix Released
Meg McRoberts (dreidellhasa) wrote :

Noted in "Other Enhancements" in 5.0 Release Notes

Egor Kotko (ykotko) wrote :

{"build_id": "2014-05-29_00-52-53", "mirantis": "yes", "build_number": "330", "nailgun_sha": "0828946af6a2e757fb018c2f69f7f81f0ea4b7f1", "ostf_sha": "3690709d95c1f42506b6d955b44221910cec8527", "fuelmain_sha": "7fbe99c86a4fffcb5ebc5b86046dcf1c20776f61", "astute_sha": "55df06b2e84fa5d71a1cc0e78dbccab5db29d968", "release": "4.1B", "fuellib_sha": "c64ffdc5e7bfe7fa8eb971be1fea2260ee849ab5"}
This bug was reproduced on 4.1.1

Egor Kotko (ykotko) wrote :
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers