cinder upload volume operation doesn't work with multiple cinder stores

Bug #2054575 reported by Rajat Dhasmana
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Undecided
Unassigned

Bug Description

Env: Glance is configured to use cinder as it's backend with multiple stores enabled.

When we upload a volume from cinder to glance, we face the following error and image never gets created.

glanceclient.exc.HTTPBadRequest: HTTP 400 Bad Request: Invalid location

This is because glance expected a location URI in the format
cinder://<store-id>/<vol-id>
instead cinder sends the URI
cinder://<vol-id>

Cinder cannot provide store-id since it doesn't have info about the glance stores and in which store the volume will be uploaded as an image.
Glance can handle this case by updating the location received from cinder (startswith("cinder://") and update it in the new format.

Steps to reproduce:
1. Configure glance to use Cinder store in multi store configuration
2. set the following cinder.conf options in the backend section

[<backend>]
image_upload_use_cinder_backend = True
image_upload_use_internal_tenant = True

3. Upload a volume to glance and check logs to verify the optimized path is taken

Here are the reference logs from a successful run

Create image volume (by cloning)

Jun 19 17:34:25 Ubuntu cinder-volume[835237]: DEBUG cinder.volume.manager [req-6c9ddac2-50ab-40aa-9ac2-612dce659101 req-1b038166-69c5-40ee-8312-3a589abd7e94 admin None] Creating image volume entry: {'volume_glance_metadata': [], 'consistencygroup_id': None, 'provider_id': None, 'group_id': None, 'previous_status': 'available', 'service_uuid': 'b11ecadf-5a67-4cb0-9988-b0b0b2584f8b', 'provider_auth': 'CHAP LW968dSz8Gj8cekHNQqz SsdYNRA5PbE3PWCy', 'updated_at': datetime.datetime(2024, 6, 19, 12, 4, 26, tzinfo=datetime.timezone.utc), 'ec2_id': None, 'project_id': 'b87315e82e6746858f53ecc804551571', 'replication_driver_data': None, 'provider_location': '127.0.0.1:3260;10.0.2.15:3260,iqn.2010-10.org.openstack:volume-78ee1687-fb5b-420e-9706-8944c5076033 iqn.2010-10.org.openstack:volume-78ee1687-fb5b-420e-9706-8944c5076033 0', 'host': 'Ubuntu@lvmdriver-1#lvmdriver-1', 'deleted': False, 'deleted_at': None, 'bootable': False, 'encryption_key_id': None, 'shared_targets': False, 'volume_admin_metadata': [], 'replication_extended_status': None, 'admin_metadata': {}, 'provider_geometry': None, 'metadata': {}, 'size': 1, 'volume_type_id': '57f1a15e-76e0-4e44-abb3-7e30c2c1f523', 'terminated_at': None, 'volume_metadata': [], 'scheduled_at': datetime.datetime(2024, 6, 17, 12, 51, 52, tzinfo=datetime.timezone.utc), 'availability_zone': 'nova', 'launched_at': datetime.datetime(2024, 6, 17, 12, 51, 53, tzinfo=datetime.timezone.utc), 'replication_status': None, 'cluster_name': None, 'multiattach': False, 'display_name': 'image-b73d7905-dab1-4503-962a-698ed384807a', 'display_description': None, 'created_at': datetime.datetime(2024, 6, 17, 12, 51, 52, tzinfo=datetime.timezone.utc), 'user_id': 'b9c61e5079de45e8b9cd1df9392c2827', 'attach_status': 'detached', 'status': 'creating', 'source_volid': '78ee1687-fb5b-420e-9706-8944c5076033'}. {{(pid=835237) _clone_image_volume /opt/stack/cinder/cinder/volume/manager.py:1680}}

Register location to glance

Jun 19 17:34:27 Ubuntu cinder-volume[835237]: DEBUG cinder.volume.manager [req-6c9ddac2-50ab-40aa-9ac2-612dce659101 req-1b038166-69c5-40ee-8312-3a589abd7e94 admin None] Registered image volume location to glance image-id: b73d7905-dab1-4503-962a-698ed384807a. {{(pid=835237) copy_volume_to_image /opt/stack/cinder/cinder/volume/manager.py:1806}}

Changed in glance:
status: New → In Progress
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance/+/922316

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to glance (master)

Reviewed: https://review.opendev.org/c/openstack/glance/+/906359
Committed: https://opendev.org/openstack/glance/commit/1c523cf94e89470c008163a31385a6cff3658c93
Submitter: "Zuul (22348)"
Branch: master

commit 1c523cf94e89470c008163a31385a6cff3658c93
Author: Pranali Deore <email address hidden>
Date: Tue Jan 23 11:00:37 2024 +0000

    Make location URL compatible with cinder backend

    While adding location to an image, cinder sends location url as
    `cinder://volume_id` for single as well as multistore which is
    incompatible with glance multistore and store throws
    InvalidLoctation error.

    Modifying the location url to be compatible with multistore as
    `cinder://store_id/volume_id` to avoid Invalid Location error.

    Related-Bug: #2054575
    Change-Id: I5f791c58ae857f6c553276dd9808799c3db3aa4f

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance (master)

Change abandoned by "Rajat Dhasmana <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/glance/+/922316
Reason: gate is behaving strange, abandon/restoring to restore gate jobs to sanity

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.opendev.org/c/openstack/glance/+/922316
Committed: https://opendev.org/openstack/glance/commit/8318da1d5fad2b47fc5747c049b90fe9255c28ad
Submitter: "Zuul (22348)"
Branch: master

commit 8318da1d5fad2b47fc5747c049b90fe9255c28ad
Author: Rajat Dhasmana <email address hidden>
Date: Wed Jun 19 17:55:52 2024 +0530

    Fix: optimized upload volume in Cinder store

    When Glance is configured to use Cinder store and we upload
    volume to Glance in the optimized path, it fails with
    InvalidLocation error.
    This happens because Cinder is not aware about the store
    in which we will create the image and supplies the old
    format URL i.e. cinder://<vol-id> whereas Glance expects
    new location format i.e. cinder://<store-id>/<vol-id>.

    Glance has code to update the format from old location format
    to new location format but it isn't triggered in case of
    old location APIs.

    This patch adds the context to the update store ID request
    which calls the Cinder store to provide the updated location,
    hence fixing the optimized path for upload volume to image.

    Closes-Bug: #2054575
    Change-Id: Idd1cb8982b40b85a17821596f76dfa10207f6381

Changed in glance:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 29.0.0.0b2

This issue was fixed in the openstack/glance 29.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/glance/+/924674

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance (stable/2023.2)

Change abandoned by "Rajat Dhasmana <email address hidden>" on branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/glance/+/924674
Reason: needs to go to 2024.1 first

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (stable/2024.1)

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/glance/+/924949

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/2024.1)

Reviewed: https://review.opendev.org/c/openstack/glance/+/924949
Committed: https://opendev.org/openstack/glance/commit/53175262f0f8df303176f4e9cd44e36d442c6cea
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 53175262f0f8df303176f4e9cd44e36d442c6cea
Author: Rajat Dhasmana <email address hidden>
Date: Thu Jul 25 21:16:27 2024 +0530

    Fix LP#2054575

    This patch contains a squash of two commits:

    COMMIT 1:

    Make location URL compatible with cinder backend

    While adding location to an image, cinder sends location url as
    `cinder://volume_id` for single as well as multistore which is
    incompatible with glance multistore and store throws
    InvalidLoctation error.

    Modifying the location url to be compatible with multistore as
    `cinder://store_id/volume_id` to avoid Invalid Location error.

    Related-Bug: #2054575
    Change-Id: I5f791c58ae857f6c553276dd9808799c3db3aa4f

    COMMIT 2:

    Fix: optimized upload volume in Cinder store

    When Glance is configured to use Cinder store and we upload
    volume to Glance in the optimized path, it fails with
    InvalidLocation error.
    This happens because Cinder is not aware about the store
    in which we will create the image and supplies the old
    format URL i.e. cinder://<vol-id> whereas Glance expects
    new location format i.e. cinder://<store-id>/<vol-id>.

    Glance has code to update the format from old location format
    to new location format but it isn't triggered in case of
    old location APIs.

    This patch adds the context to the update store ID request
    which calls the Cinder store to provide the updated location,
    hence fixing the optimized path for upload volume to image.

    Closes-Bug: #2054575
    Change-Id: Idd1cb8982b40b85a17821596f76dfa10207f6381

    The commits are squashed together to make backport easier.

    Change-Id: I9ecdfe08b63c00446dc3e24195e3b8e59b82f55c

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/glance/+/926025

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/glance/+/926025
Committed: https://opendev.org/openstack/glance/commit/8540ffc6ee34976285209e9f3c7c728bf649e0ff
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 8540ffc6ee34976285209e9f3c7c728bf649e0ff
Author: Rajat Dhasmana <email address hidden>
Date: Thu Jul 25 21:16:27 2024 +0530

    Fix LP#2054575

    This patch contains a squash of two commits:

    COMMIT 1:

    Make location URL compatible with cinder backend

    While adding location to an image, cinder sends location url as
    `cinder://volume_id` for single as well as multistore which is
    incompatible with glance multistore and store throws
    InvalidLoctation error.

    Modifying the location url to be compatible with multistore as
    `cinder://store_id/volume_id` to avoid Invalid Location error.

    Related-Bug: #2054575
    Change-Id: I5f791c58ae857f6c553276dd9808799c3db3aa4f

    COMMIT 2:

    Fix: optimized upload volume in Cinder store

    When Glance is configured to use Cinder store and we upload
    volume to Glance in the optimized path, it fails with
    InvalidLocation error.
    This happens because Cinder is not aware about the store
    in which we will create the image and supplies the old
    format URL i.e. cinder://<vol-id> whereas Glance expects
    new location format i.e. cinder://<store-id>/<vol-id>.

    Glance has code to update the format from old location format
    to new location format but it isn't triggered in case of
    old location APIs.

    This patch adds the context to the update store ID request
    which calls the Cinder store to provide the updated location,
    hence fixing the optimized path for upload volume to image.

    Closes-Bug: #2054575
    Change-Id: Idd1cb8982b40b85a17821596f76dfa10207f6381

    The commits are squashed together to make backport easier.

    Change-Id: I9ecdfe08b63c00446dc3e24195e3b8e59b82f55c
    (cherry picked from commit 53175262f0f8df303176f4e9cd44e36d442c6cea)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/glance/+/926112

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/glance/+/926112
Committed: https://opendev.org/openstack/glance/commit/6a17e4e2bae0f3661ffed08b2d4360dfc8b3841d
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 6a17e4e2bae0f3661ffed08b2d4360dfc8b3841d
Author: Rajat Dhasmana <email address hidden>
Date: Thu Jul 25 21:16:27 2024 +0530

    Fix LP#2054575

    This patch contains a squash of two commits:

    COMMIT 1:

    Make location URL compatible with cinder backend

    While adding location to an image, cinder sends location url as
    `cinder://volume_id` for single as well as multistore which is
    incompatible with glance multistore and store throws
    InvalidLoctation error.

    Modifying the location url to be compatible with multistore as
    `cinder://store_id/volume_id` to avoid Invalid Location error.

    Related-Bug: #2054575
    Change-Id: I5f791c58ae857f6c553276dd9808799c3db3aa4f

    COMMIT 2:

    Fix: optimized upload volume in Cinder store

    When Glance is configured to use Cinder store and we upload
    volume to Glance in the optimized path, it fails with
    InvalidLocation error.
    This happens because Cinder is not aware about the store
    in which we will create the image and supplies the old
    format URL i.e. cinder://<vol-id> whereas Glance expects
    new location format i.e. cinder://<store-id>/<vol-id>.

    Glance has code to update the format from old location format
    to new location format but it isn't triggered in case of
    old location APIs.

    This patch adds the context to the update store ID request
    which calls the Cinder store to provide the updated location,
    hence fixing the optimized path for upload volume to image.

    Closes-Bug: #2054575
    Change-Id: Idd1cb8982b40b85a17821596f76dfa10207f6381

    The commits are squashed together to make backport easier.

    Change-Id: I9ecdfe08b63c00446dc3e24195e3b8e59b82f55c
    (cherry picked from commit 53175262f0f8df303176f4e9cd44e36d442c6cea)
    (cherry picked from commit 8540ffc6ee34976285209e9f3c7c728bf649e0ff)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 2023.1-eom

This issue was fixed in the openstack/glance 2023.1-eom release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.