RBD: Upload volume to image with different format fails

[Rajat Dhasmana]

When uploading an RBD volume to image and the image having different format than 'raw', the operation fails.

COMMAND: cinder upload-to-image --disk-format qcow2 <vol-id> test-img-qcow2

ERROR TRACE:

Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager Traceback (most recent call last):
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 1778, in copy_volume_to_image
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager self.driver.copy_volume_to_image(context, volume,
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 2094, in copy_volume_to_image
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager volume_utils.upload_volume(context, image_service, image_meta, None,
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/volume_utils.py", line 1352, in upload_volume
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager image_utils.upload_volume(context, image_service, image_meta, volume_path,
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/image/image_utils.py", line 1127, in upload_volume
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager data = qemu_img_info(
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/image/image_utils.py", line 169, in qemu_img_info
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager format_name = cinder.privsep.format_inspector.get_format_if_safe(
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager return self.channel.remote_call(name, args, kwargs,
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager raise exc_type(*result[2])
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager TypeError: expected str, bytes or os.PathLike object, not NoneType
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR cinder.volume.manager

Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server [req-394633b7-958a-45b9-85a5-6df4f8f1a091 req-5901327f-2a96-48d5-9bd3-c7e3bcb96fab admin None] Exception during message handling: TypeError: expected str, bytes or os.PathLike object, not NoneType
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 172, in _process_incoming
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1797, in copy_volume_to_image
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server self.force_reraise()
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server raise self.value
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1778, in copy_volume_to_image
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server self.driver.copy_volume_to_image(context, volume,
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/rbd.py", line 2094, in copy_volume_to_image
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server volume_utils.upload_volume(context, image_service, image_meta, None,
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/volume_utils.py", line 1352, in upload_volume
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server image_utils.upload_volume(context, image_service, image_meta, volume_path,
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/image/image_utils.py", line 1127, in upload_volume
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server data = qemu_img_info(
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/image/image_utils.py", line 169, in qemu_img_info
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server format_name = cinder.privsep.format_inspector.get_format_if_safe(
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server return self.channel.remote_call(name, args, kwargs,
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server raise exc_type(*result[2])
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server TypeError: expected str, bytes or os.PathLike object, not NoneType
Dec 26 16:37:06 devstack-ceph cinder-volume[696260]: ERROR oslo_messaging.rpc.server

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/938265

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/938566

[Tobias Urdin (tobias-urdin)]

Hit this today, thanks for working on this Rajat

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2025.1
Review: https://review.opendev.org/c/openstack/cinder/+/945616

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/938265
Committed: https://opendev.org/openstack/cinder/commit/7d320764980a0e33a9abad14db41a4cdee71d57a
Submitter: "Zuul (22348)"
Branch: master

commit 7d320764980a0e33a9abad14db41a4cdee71d57a
Author: Rajat Dhasmana <email address hidden>
Date: Thu Dec 26 20:20:19 2024 +0530

    RBD: Fix upload volume with different format

    With change[1], we introduced an optimized path to upload an RBD
    volume to image. However, this does not work well when the image
    format is not 'raw' and conversion is required.

    Currently there are 2 places where we need handling of the
    RBDVolumeIOWrapper object:
    1. format inspector
    2. qemu-img commands

    While testing, I found out the following issues that needs to be
    addressed to fix the upload path with conversion:

    1. Passing RBDVolumeIOWrapper to format inspector
    We fail when calling privsep since RPC cannot serialize the
    RBDVolumeIOWrapper object and fails with the following error

    ERROR oslo_messaging.rpc.server TypeError: can not serialize 'RBDVolumeIOWrapper' object

    2. Handling in format inspector
    Either we need to pass the volume_fd var to several methods in
    format inspector code or introduce a condition to find out if
    we get volume path or RBDVolumeIOWrapper and open it if it's a path

    3. Handling in qemu-img commands
    We need to use rbd:{pool-name}/{image-name} format instead of file path when
    executing qemu-img commands like convert[2] otherwise we will fail with the
    below error.

    Stderr: "qemu-img: Could not open '<os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x781f747a6aa0>':
    Could not open '<os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x781f747a6aa0>': No such file or directory\n"

    Not to mention passing the volume_fd object to all the qemu related methods
    and handling cases when volume_path is none and volume_fd contains the
    RBD IO object.

    Even after fixing all the issues, it's still not sure that the optimized
    path offers better performance compared to the traditional workflow as
    qemu-img will use librbd to read the volume file and convert it to a local
    path.

    Due to the above reasons, we only use the optimized path when the image
    format is same as volume format (i.e. raw) and container format is not
    'compressed'.

    [1] https://review.opendev.org/c/openstack/cinder/+/928024
    [2] https://docs.ceph.com/en/latest/rbd/qemu-rbd/#running-qemu-with-rbd

    Closes-Bug: #2092534
    Change-Id: I32b505aa69c71b62e7e3a52d65d38165d34e97d8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/945616
Committed: https://opendev.org/openstack/cinder/commit/bb8ad77e60098c49f2937b13463e695103587e1c
Submitter: "Zuul (22348)"
Branch: stable/2025.1

commit bb8ad77e60098c49f2937b13463e695103587e1c
Author: Rajat Dhasmana <email address hidden>
Date: Thu Dec 26 20:20:19 2024 +0530

    RBD: Fix upload volume with different format

    With change[1], we introduced an optimized path to upload an RBD
    volume to image. However, this does not work well when the image
    format is not 'raw' and conversion is required.

    Currently there are 2 places where we need handling of the
    RBDVolumeIOWrapper object:
    1. format inspector
    2. qemu-img commands

    While testing, I found out the following issues that needs to be
    addressed to fix the upload path with conversion:

    1. Passing RBDVolumeIOWrapper to format inspector
    We fail when calling privsep since RPC cannot serialize the
    RBDVolumeIOWrapper object and fails with the following error

    ERROR oslo_messaging.rpc.server TypeError: can not serialize 'RBDVolumeIOWrapper' object

    2. Handling in format inspector
    Either we need to pass the volume_fd var to several methods in
    format inspector code or introduce a condition to find out if
    we get volume path or RBDVolumeIOWrapper and open it if it's a path

    3. Handling in qemu-img commands
    We need to use rbd:{pool-name}/{image-name} format instead of file path when
    executing qemu-img commands like convert[2] otherwise we will fail with the
    below error.

    Stderr: "qemu-img: Could not open '<os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x781f747a6aa0>':
    Could not open '<os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x781f747a6aa0>': No such file or directory\n"

    Not to mention passing the volume_fd object to all the qemu related methods
    and handling cases when volume_path is none and volume_fd contains the
    RBD IO object.

    Even after fixing all the issues, it's still not sure that the optimized
    path offers better performance compared to the traditional workflow as
    qemu-img will use librbd to read the volume file and convert it to a local
    path.

    Due to the above reasons, we only use the optimized path when the image
    format is same as volume format (i.e. raw) and container format is not
    'compressed'.

    [1] https://review.opendev.org/c/openstack/cinder/+/928024
    [2] https://docs.ceph.com/en/latest/rbd/qemu-rbd/#running-qemu-with-rbd

    Closes-Bug: #2092534
    Change-Id: I32b505aa69c71b62e7e3a52d65d38165d34e97d8
    (cherry picked from commit 7d320764980a0e33a9abad14db41a4cdee71d57a)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/cinder/+/945825

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 26.0.0.0rc2 Epoxy release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/cinder/+/946101

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/945825
Committed: https://opendev.org/openstack/cinder/commit/c579760ddaee4e65dc45f8f546814763af0da0ef
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit c579760ddaee4e65dc45f8f546814763af0da0ef
Author: Rajat Dhasmana <email address hidden>
Date: Thu Dec 26 20:20:19 2024 +0530

    RBD: Fix upload volume with different format

    With change[1], we introduced an optimized path to upload an RBD
    volume to image. However, this does not work well when the image
    format is not 'raw' and conversion is required.

    Currently there are 2 places where we need handling of the
    RBDVolumeIOWrapper object:
    1. format inspector
    2. qemu-img commands

    While testing, I found out the following issues that needs to be
    addressed to fix the upload path with conversion:

    1. Passing RBDVolumeIOWrapper to format inspector
    We fail when calling privsep since RPC cannot serialize the
    RBDVolumeIOWrapper object and fails with the following error

    ERROR oslo_messaging.rpc.server TypeError: can not serialize 'RBDVolumeIOWrapper' object

    2. Handling in format inspector
    Either we need to pass the volume_fd var to several methods in
    format inspector code or introduce a condition to find out if
    we get volume path or RBDVolumeIOWrapper and open it if it's a path

    3. Handling in qemu-img commands
    We need to use rbd:{pool-name}/{image-name} format instead of file path when
    executing qemu-img commands like convert[2] otherwise we will fail with the
    below error.

    Stderr: "qemu-img: Could not open '<os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x781f747a6aa0>':
    Could not open '<os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x781f747a6aa0>': No such file or directory\n"

    Not to mention passing the volume_fd object to all the qemu related methods
    and handling cases when volume_path is none and volume_fd contains the
    RBD IO object.

    Even after fixing all the issues, it's still not sure that the optimized
    path offers better performance compared to the traditional workflow as
    qemu-img will use librbd to read the volume file and convert it to a local
    path.

    Due to the above reasons, we only use the optimized path when the image
    format is same as volume format (i.e. raw) and container format is not
    'compressed'.

    [1] https://review.opendev.org/c/openstack/cinder/+/928024
    [2] https://docs.ceph.com/en/latest/rbd/qemu-rbd/#running-qemu-with-rbd

    Closes-Bug: #2092534
    Change-Id: I32b505aa69c71b62e7e3a52d65d38165d34e97d8
    (cherry picked from commit 7d320764980a0e33a9abad14db41a4cdee71d57a)
    (cherry picked from commit bb8ad77e60098c49f2937b13463e695103587e1c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/946101
Committed: https://opendev.org/openstack/cinder/commit/7b90004f54ccc9d6c7fc68d5f535366200b5c9ff
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 7b90004f54ccc9d6c7fc68d5f535366200b5c9ff
Author: Rajat Dhasmana <email address hidden>
Date: Thu Dec 26 20:20:19 2024 +0530

    RBD: Fix upload volume with different format

    With change[1], we introduced an optimized path to upload an RBD
    volume to image. However, this does not work well when the image
    format is not 'raw' and conversion is required.

    Currently there are 2 places where we need handling of the
    RBDVolumeIOWrapper object:
    1. format inspector
    2. qemu-img commands

    While testing, I found out the following issues that needs to be
    addressed to fix the upload path with conversion:

    1. Passing RBDVolumeIOWrapper to format inspector
    We fail when calling privsep since RPC cannot serialize the
    RBDVolumeIOWrapper object and fails with the following error

    ERROR oslo_messaging.rpc.server TypeError: can not serialize 'RBDVolumeIOWrapper' object

    2. Handling in format inspector
    Either we need to pass the volume_fd var to several methods in
    format inspector code or introduce a condition to find out if
    we get volume path or RBDVolumeIOWrapper and open it if it's a path

    3. Handling in qemu-img commands
    We need to use rbd:{pool-name}/{image-name} format instead of file path when
    executing qemu-img commands like convert[2] otherwise we will fail with the
    below error.

    Stderr: "qemu-img: Could not open '<os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x781f747a6aa0>':
    Could not open '<os_brick.initiator.linuxrbd.RBDVolumeIOWrapper object at 0x781f747a6aa0>': No such file or directory\n"

    Not to mention passing the volume_fd object to all the qemu related methods
    and handling cases when volume_path is none and volume_fd contains the
    RBD IO object.

    Even after fixing all the issues, it's still not sure that the optimized
    path offers better performance compared to the traditional workflow as
    qemu-img will use librbd to read the volume file and convert it to a local
    path.

    Due to the above reasons, we only use the optimized path when the image
    format is same as volume format (i.e. raw) and container format is not
    'compressed'.

    [1] https://review.opendev.org/c/openstack/cinder/+/928024
    [2] https://docs.ceph.com/en/latest/rbd/qemu-rbd/#running-qemu-with-rbd

    Closes-Bug: #2092534
    Change-Id: I32b505aa69c71b62e7e3a52d65d38165d34e97d8
    (cherry picked from commit 7d320764980a0e33a9abad14db41a4cdee71d57a)
    (cherry picked from commit bb8ad77e60098c49f2937b13463e695103587e1c)
    (cherry picked from commit c579760ddaee4e65dc45f8f546814763af0da0ef)