RBD image backend tries to flatten images even if they are already flat

Bug #1860990 reported by Vladyslav Drok
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Vladyslav Drok
Queens
Fix Released
Medium
Lee Yarwood
Rocky
Fix Released
Medium
Lee Yarwood
Stein
Fix Released
Medium
Lee Yarwood
Train
Fix Released
Medium
Lee Yarwood

Bug Description

When [DEFAULT]show_multiple_locations option is not set in glance, and both glance and nova use ceph as their backend, with properly configured accesses, nova will fail with the following exception:

2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [req-8021fd76-d5ab-4a9b-bd17-f5eb4d4faf62 0e96a04f360644818632b7e46fe8d3e7 ac01daacc7424a40b8b464a163902dcb - default default] [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] Instance failed to spawn: rbd.InvalidArgument: [errno 22] error flattening b'fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6_disk'
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] Traceback (most recent call last):
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/manager.py", line 5757, in _unshelve_instance
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] block_device_info=block_device_info)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3457, in spawn
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] block_device_info=block_device_info)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3832, in _create_image
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] fallback_from_host)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3923, in _create_and_inject_local_root
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] instance, size, fallback_from_host)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 9267, in _try_fetch_image_cache
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] image.flatten()
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/imagebackend.py", line 983, in flatten
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] self.driver.flatten(self.rbd_name, pool=self.driver.pool)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 290, in flatten
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] vol.flatten()
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] rv = execute(f, *args, **kwargs)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] six.reraise(c, e, tb)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] raise value
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "/var/lib/openstack/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] rv = meth(*args, **kwargs)
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] File "rbd.pyx", line 2207, in rbd.Image.flatten
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6] rbd.InvalidArgument: [errno 22] error flattening b'fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6_disk'
2020-01-23 14:36:43.617 8647 ERROR nova.compute.manager [instance: fa9e4118-1bb1-4d52-a2e1-9f61b0e20dc6]

During unshelve, when nova fails to determine the parent pool for the image (because glance api does not return rbd image location), it downloads the image through the glance api and puts it into its pool. Such image will be already flat. And nova will try to flatten it again and fail. It might be better to make flatten idempotent and just be a noop for already flat images.

Tags: ceph
Changed in nova:
assignee: nobody → Vladyslav Drok (vdrok)
status: New → In Progress
Revision history for this message
Vladyslav Drok (vdrok) wrote :
melanie witt (melwitt)
Changed in nova:
importance: Undecided → Medium
tags: added: ceph
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/704330
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=65825ebfbd58920adac5e8594891eec8e9cec41f
Submitter: Zuul
Branch: master

commit 65825ebfbd58920adac5e8594891eec8e9cec41f
Author: Vladyslav Drok <email address hidden>
Date: Mon Jan 27 15:31:53 2020 +0100

    Make RBD imagebackend flatten method idempotent

    If glance and nova are both configured with RBD backend, but glance
    does not return location information from the API, nova will fail to
    clone the image from glance pool and will download it from the API.
    In this case, image will be already flat, and subsequent flatten call
    will fail.

    This commit makes flatten call idempotent, so that it ignores already
    flat images by catching ImageUnacceptable when requesting parent info
    from ceph.

    Closes-Bug: 1860990
    Change-Id: Ia6c184c31a980e4728b7309b2afaec4d9f494ac3

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/707650

Revision history for this message
Jacolex (jacolex) wrote :

The same situation with show_multiple_locations = True.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/708615

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/708616

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/708617

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/train)

Reviewed: https://review.opendev.org/707650
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=03d59e289369df4980bc1e7350e7f52a6f6aa828
Submitter: Zuul
Branch: stable/train

commit 03d59e289369df4980bc1e7350e7f52a6f6aa828
Author: Vladyslav Drok <email address hidden>
Date: Mon Jan 27 15:31:53 2020 +0100

    Make RBD imagebackend flatten method idempotent

    If glance and nova are both configured with RBD backend, but glance
    does not return location information from the API, nova will fail to
    clone the image from glance pool and will download it from the API.
    In this case, image will be already flat, and subsequent flatten call
    will fail.

    This commit makes flatten call idempotent, so that it ignores already
    flat images by catching ImageUnacceptable when requesting parent info
    from ceph.

    Closes-Bug: 1860990
    Change-Id: Ia6c184c31a980e4728b7309b2afaec4d9f494ac3
    (cherry picked from commit 65825ebfbd58920adac5e8594891eec8e9cec41f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/stein)

Reviewed: https://review.opendev.org/708615
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b3afe748335886403315c6a7bba6a8879a05d077
Submitter: Zuul
Branch: stable/stein

commit b3afe748335886403315c6a7bba6a8879a05d077
Author: Vladyslav Drok <email address hidden>
Date: Mon Jan 27 15:31:53 2020 +0100

    Make RBD imagebackend flatten method idempotent

    If glance and nova are both configured with RBD backend, but glance
    does not return location information from the API, nova will fail to
    clone the image from glance pool and will download it from the API.
    In this case, image will be already flat, and subsequent flatten call
    will fail.

    This commit makes flatten call idempotent, so that it ignores already
    flat images by catching ImageUnacceptable when requesting parent info
    from ceph.

    NOTE(lyarwood): Conflict as I361af845d6a733618ecd056aa7df973191184ae9 is
    not present in stable/stein.

    Conflicts:
       nova/virt/libvirt/imagebackend.py

    Closes-Bug: 1860990
    Change-Id: Ia6c184c31a980e4728b7309b2afaec4d9f494ac3
    (cherry picked from commit 65825ebfbd58920adac5e8594891eec8e9cec41f)
    (cherry picked from commit 03d59e289369df4980bc1e7350e7f52a6f6aa828)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.opendev.org/708616
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5d44052fedc9914aed4de4af3dcae4de3a03a856
Submitter: Zuul
Branch: stable/rocky

commit 5d44052fedc9914aed4de4af3dcae4de3a03a856
Author: Vladyslav Drok <email address hidden>
Date: Mon Jan 27 15:31:53 2020 +0100

    Make RBD imagebackend flatten method idempotent

    If glance and nova are both configured with RBD backend, but glance
    does not return location information from the API, nova will fail to
    clone the image from glance pool and will download it from the API.
    In this case, image will be already flat, and subsequent flatten call
    will fail.

    This commit makes flatten call idempotent, so that it ignores already
    flat images by catching ImageUnacceptable when requesting parent info
    from ceph.

    Closes-Bug: 1860990
    Change-Id: Ia6c184c31a980e4728b7309b2afaec4d9f494ac3
    (cherry picked from commit 65825ebfbd58920adac5e8594891eec8e9cec41f)
    (cherry picked from commit 03d59e289369df4980bc1e7350e7f52a6f6aa828)
    (cherry picked from commit dd3c17216cdf2814cbefc83371c712b3dd9d9147)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.opendev.org/708617
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bb358bdd42ce03da8040128bad1f79b2b436f25b
Submitter: Zuul
Branch: stable/queens

commit bb358bdd42ce03da8040128bad1f79b2b436f25b
Author: Vladyslav Drok <email address hidden>
Date: Mon Jan 27 15:31:53 2020 +0100

    Make RBD imagebackend flatten method idempotent

    If glance and nova are both configured with RBD backend, but glance
    does not return location information from the API, nova will fail to
    clone the image from glance pool and will download it from the API.
    In this case, image will be already flat, and subsequent flatten call
    will fail.

    This commit makes flatten call idempotent, so that it ignores already
    flat images by catching ImageUnacceptable when requesting parent info
    from ceph.

    Closes-Bug: 1860990
    Change-Id: Ia6c184c31a980e4728b7309b2afaec4d9f494ac3
    (cherry picked from commit 65825ebfbd58920adac5e8594891eec8e9cec41f)
    (cherry picked from commit 03d59e289369df4980bc1e7350e7f52a6f6aa828)
    (cherry picked from commit dd3c17216cdf2814cbefc83371c712b3dd9d9147)
    (cherry picked from commit 5d44052fedc9914aed4de4af3dcae4de3a03a856)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova queens-eol

This issue was fixed in the openstack/nova queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova rocky-eol

This issue was fixed in the openstack/nova rocky-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers