regression in cloning raw image type with ceph

[Sam Yaple]

Exists in 12.0.0 and master right now. An image of type 'raw' cannot be booted if it tries to clone due to a regression introduced in 12.0.0.0-rc3

2015-10-20 22:27:34.185 1 DEBUG nova.virt.libvirt.imagebackend [req-8ba3f810-3c6a-4d9c-8848-657308310df8 30c19408c93247c9824e13980f3c14aa 41358b17076645ccbdd3738ca6b02a77 - - -] Image locations are: [{'url': u'rbd://5fba2fbc-551d-11e5-a8ce-01ef4c5cf93c/glance/eb1e102a-dc1d-4084-aa5a-309d02b1f170/snap', 'metadata': {}}] clone /usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py:829
2015-10-20 22:27:34.198 1 DEBUG oslo_concurrency.lockutils [req-8ba3f810-3c6a-4d9c-8848-657308310df8 30c19408c93247c9824e13980f3c14aa 41358b17076645ccbdd3738ca6b02a77 - - -] Lock "51f355cbdaa5d01f681ddff0aec17c526c07c9de" released by "nova.virt.libvirt.imagebackend.fetch_func_sync" :: held 0.068s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:265
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [req-8ba3f810-3c6a-4d9c-8848-657308310df8 30c19408c93247c9824e13980f3c14aa 41358b17076645ccbdd3738ca6b02a77 - - -] [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] Instance failed to spawn
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] Traceback (most recent call last):
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2172, in _build_resources
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] yield resources
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2019, in _build_and_run_instance
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] block_device_info=block_device_info)
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2437, in spawn
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] admin_pass=admin_password)
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2861, in _create_image
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] instance, size, fallback_from_host)
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6282, in _try_fetch_image_cache
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] size=size)
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 249, in cache
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] *args, **kwargs)
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 803, in create_image
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] prepare_template(target=base, max_size=size, *args, **kwargs)
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] return f(*args, **kwargs)
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 239, in fetch_func_sync
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] fetch_func(target=target, *args, **kwargs)
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2853, in clone_fallback_to_fetch
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] backend.clone(context, disk_images['image_id'])
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 837, in clone
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] if self.driver.is_cloneable(location, image_meta):
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 193, in is_cloneable
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] if image_meta.disk_format != 'raw':
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94] AttributeError: 'dict' object has no attribute 'disk_format'
2015-10-20 22:27:34.199 1 ERROR nova.compute.manager [instance: 5f35f469-4ef5-471f-bdbc-ecb1835e1e94]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/237799

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/237801

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sam Yaple (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/237801
Reason: yea i jumped the gun. been doing to much today. will wait for master merge

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237799
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=835ff1bf73d36a311168283f8a4ef5bec03d5a7e
Submitter: Jenkins
Branch: master

commit 835ff1bf73d36a311168283f8a4ef5bec03d5a7e
Author: Sam Yaple <email address hidden>
Date: Tue Oct 20 22:42:36 2015 +0000

    Fix attibute error when cloning raw images in Ceph

    A regression introduced in If03b166d3ecc3e7fa6b7f1a0c69f8ab1cc7b1972
    causes a glance image with the type 'raw' in ceph to fail to CoW
    clone which prevents launching at all in this case.

    The original patch did not update imagebackend.py [1] to use the new
    nova.objects.ImageMeta object so the clone() function passes a normal
    dict to the is_cloneable() function.

    Additionally add another test to ensure that not passing disk_format
    will fail, but not result in an exception. It will simply mean that
    the function returns that it cannot take advantage of the CoW cloning
    ceph provides.

    [1] https://github.com/openstack/nova/blob/f2e2a5891d5f5ff9346e6dc8e4dd0e994485245c/nova/virt/libvirt/imagebackend.py#L824

    Change-Id: I7ae107bfccab7ff66c09d96856722dd6e60fdd96
    Closes-Bug: #1508230

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237801
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=474e7bb3f356e7e7592a249dcf2053659fe8cf38
Submitter: Jenkins
Branch: stable/liberty

commit 474e7bb3f356e7e7592a249dcf2053659fe8cf38
Author: Sam Yaple <email address hidden>
Date: Tue Oct 20 22:42:36 2015 +0000

    Fix attibute error when cloning raw images in Ceph

    A regression introduced in If03b166d3ecc3e7fa6b7f1a0c69f8ab1cc7b1972
    causes a glance image with the type 'raw' in ceph to fail to CoW
    clone which prevents launching at all in this case.

    The original patch did not update imagebackend.py [1] to use the new
    nova.objects.ImageMeta object so the clone() function passes a normal
    dict to the is_cloneable() function.

    Additionally add another test to ensure that not passing disk_format
    will fail, but not result in an exception. It will simply mean that
    the function returns that it cannot take advantage of the CoW cloning
    ceph provides.

    [1] https://github.com/openstack/nova/blob/f2e2a5891d5f5ff9346e6dc8e4dd0e994485245c/nova/virt/libvirt/imagebackend.py#L824

    Change-Id: I7ae107bfccab7ff66c09d96856722dd6e60fdd96
    Closes-Bug: #1508230
    (cherry picked from commit 835ff1bf73d36a311168283f8a4ef5bec03d5a7e)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b1 development milestone.

[wangqiang.sheng (wangqiangsheng)]

查找问题定位：/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py

3，解决问题bug:
  修改前：
    if image_meta.disk_format != 'raw':
  修改后：
    if image_meta['disk_format'] != 'raw':
179
180 def is_cloneable(self, image_location, image_meta):
181 import ipdb;ipdb.set_trace()
182 url = image_location['url']
183 try:
184 fsid, pool, image, snapshot = self.parse_url(url)
185 except exception.ImageUnacceptable as e:
186 LOG.debug('not cloneable: %s', e)
187 return False
188
189 if self._get_fsid() != fsid:
190 reason = '%s is in a different ceph cluster' % url
191 LOG.debug(reason)
192 return False
193
194 # Add william , image_meta.disk_format != 'raw'
195 # so ...
196 if image_meta['disk_format'] != 'raw':
197 reason = ("rbd image clone requires image format to be "
198 "'raw' but image {0} is '{1}'").format(
199 url, image_meta.disk_format)
200 LOG.debug(reason)
201 return False

http://swq499809608.blog.51cto.com/797714/1722459
详情查看链接！

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Page (james-page)]

Ubuntu SRU information

[Impact]
Deployments using the ceph rbd backend for Nova libvirt are unable to boot instances.

[Test Case]
Deploy openstack with ceph rbd backend for nova enabled
Attempt to boot an instance

[Regression Potential]
Minimal - in development release and landed in stable branch.

[James Page (james-page)]

For those interested in testing, I've uploaded a fixed package for nova to:

  https://launchpad.net/~james-page/+archive/ubuntu/liberty-sru

Alongside all current liberty stable point releases we're about to start SRU'ing.

[James Page (james-page)]

Additional note for SRU - the image type must be 'raw' - a qcow2 image format works just fine as it does not use the cow cloning code path.

[Florian Haas (fghaas)]

Sweet, thanks James!

[Brian Murray (brian-murray)]

Hello Sam, or anyone else affected,

Accepted nova into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:12.0.0-0ubuntu2.15.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/nova 12.0.1 release.