Failed to set image property. Invalid input for field/attribute simplestreams_metadata. Value: ... is too long (HTTP 400)

[Diko Parvanov]

[Impact]

Cinder does not allow setting volume metadata properties with values longer that 255 characters. Glance does allow values of up to 65536 bytes and a volume created from an image with metadata larger than 255 characters will keep that longer value in Cinder (the Cinder database field is long enough to hold it). The problem is that Cinder would not let a longer value to be set after the volume has been created.
This is inconsistent and generates issues in backup services that expect to be able to set metadata values to the same value the volume had before.

[Test Plan]

Create a volume:

openstack volume create test_vol --size 1

Try to set a property with a long value:

openstack volume set test_vol --image-property longprop=12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890

This should return an error like:

Invalid input for field/attribute longprop. Value: 123456789012345...' is too long (HTTP 400) (Request-ID: req-69875caf-16e1-4afb-91c0-3e50ff694c07)

Update cinder to this SRU version and run the volume set command again. It should work and return nothing.

[Where problems could occur]

The patch just changes the validation for the metadata length. It includes several tests of border cases. Worst case could be that it would not allow setting a metadata value that was allowed before but as the check has been changed to increase the value size that is unlikely.

[Sofia Enriquez (lsofia-enriquez)]

Dear Diko Parvanov,

From the logs, I can see that you are using the cinder client and the error looks like this:

RESP BODY: {"badRequest": {"code": 400, "message": "Invalid input for field/attribute simplestreams_metadata. Value: {\"aliases\": \"20.04,default,f,focal,lts,ubuntu\", \"arch\": \"amd64\", \"ftype\": \"disk1.img\", \"label\": \"release\", \"md5\": \"8646f69a6814ed96e27de0a1be1120a4\", \"os\": \"ubuntu\", \"pubname\": \"ubuntu-focal-20.04-amd64-server-20201111\", \"release\": \"focal\", \"release_codename\": \"Focal Fossa\", \"release_title\": \"20.04 LTS\", \"sha256\": \"151e5e797dc0131c4a1a41b76dac34c551389be67e6b551ab18c36026c420e6a\", \"size\": \"546570752\", \"support_eol\": \"2025-04-23\", \"supported\": \"True\", \"version\": \"20.04\"}. '{\"aliases\": \"20.04,default,f,focal,lts,ubuntu\", \"arch\": \"amd64\", \"ftype\": \"disk1.img\", \"label\": \"release\", \"md5\": \"8646f69a6814ed96e27de0a1be1120a4\", \"os\": \"ubuntu\", \"pubname\": \"ubuntu-focal-20.04-amd64-server-20201111\", \"release\": \"focal\", \"release_codename\": \"Focal Fossa\", \"release_title\": \"20.04 LTS\", \"sha256\": \"151e5e797dc0131c4a1a41b76dac34c551389be67e6b551ab18c36026c420e6a\", \"size\": \"546570752\", \"support_eol\": \"2025-04-23\", \"supported\": \"True\", \"version\": \"20.04\"}' is too long"}}
POST call to volumev3 for http://172.22.8.3:8776/v3/74833ea5ad9e4d1db569803774547fc7/volumes/82449828-aa1e-468c-bfe8-ed79514fd9f7/action used request id req-5e6c7ce5-b5bc-445f-9b59-9946845e1a0b
Failed to set image property: Invalid input for field/attribute simplestreams_metadata. Value: {"aliases": "20.04,default,f,focal,lts,ubuntu", "arch": "amd64", "ftype": "disk1.img", "label": "release", "md5": "8646f69a6814ed96e27de0a1be1120a4", "os": "ubuntu", "pubname": "ubuntu-focal-20.04-amd64-server-20201111", "release": "focal", "release_codename": "Focal Fossa", "release_title": "20.04 LTS", "sha256": "151e5e797dc0131c4a1a41b76dac34c551389be67e6b551ab18c36026c420e6a", "size": "546570752", "support_eol": "2025-04-23", "supported": "True", "version": "20.04"}. '{"aliases": "20.04,default,f,focal,lts,ubuntu", "arch": "amd64", "ftype": "disk1.img", "label": "release", "md5": "8646f69a6814ed96e27de0a1be1120a4", "os": "ubuntu", "pubname": "ubuntu-focal-20.04-amd64-server-20201111", "release": "focal", "release_codename": "Focal Fossa", "release_title": "20.04 LTS", "sha256": "151e5e797dc0131c4a1a41b76dac34c551389be67e6b551ab18c36026c420e6a", "size": "546570752", "support_eol": "2025-04-23", "supported": "True", "version": "20.04"}' is too long (HTTP 400) (Request-ID: req-5e6c7ce5-b5bc-445f-9b59-9946845e1a0b)

Would you mind sharing the next information with us so the cinder team can reproduce the problem:

- Would you mind sharing an input sample to reproduce this without simplestream and check if this is a cinder problem or a simplestream feature problem?
- Steps to reproduce this. What command are you using to update the metadata?
- Cinder Version

Best Regards
Sofia

[Paul Goins (vultaire)]

I looked at sources a little, but I'm not a cinder expert, so anyone who reads this, please take this with a grain of salt.

It looks like this is likely hitting cinder/api/contrib/volume_image_metadata.py, VolumeImageMetadataController.create(). That function has a @validation.schema(volume_image_metadata.set_image_metadata) decorator, and, if I'm reading it right, it defines the schema for metadata essentially as:

{"metadata": {nested dict with keys as 1-255 character strings and values as 0-255 character strings}}

In other words, the metadata field can contain many key/value pairs, and it doesn't look like there is a hardcoded limit in the number of pairs, but each key and value have a max length of 255 characters.

This is based on looking at the cinder master branch.

Interestingly, the volume_glance_metadata table of the cinder DB defines the key as a varchar(255), but the value as a text field, so it seems the DB schema may be able to allow longer values in theory without modification - it's the API's validation schema which is rejecting this.

...Again, this is my interpretation as a non-expert.

[Paul Goins (vultaire)]

@Sofia: I'm from the same team as Diko; I'll try to add some context here.

As far as I can tell, this is simply a matter of a metadata field, simplestreams_metadata, being longer than supported by the API in question.

We leverage simplestreams to automatically synchronize images from external repositories (e.g. http://cloud-images.ubuntu.com/releases/) into glance. Looking at it's code, it's simply using the create/upload/update methods of glanceclient.Client.images to create/update images in glance.

I see images created by simplestreams which clearly have this field exceeding 255 characters. And I see we can create instances or volumes with this image passed in via the --image field, and the instances/volumes are created without error, and the simplestreams_metadata field appears to be copied over without a problem.

Honestly, I'm not exactly sure what action is taking place in the log files uploaded by Diko that is triggering this, since the above does work fine.

However, if I run "openstack volume set --image-property simplestreams_metadata='<blob with over 255 characters>'" - even if I try to set the exact same property which is already set - I do get the same error. So, that seems inconsistent that we can create servers/volumes against an image with a larger field like this, but we can't set the property directly.

I've reproduced this issue using cinder-api version 2:14.2.1-0ubuntu1~cloud0 from the bionic-updates/stein/main pocket of the ubuntu cloud archive, but based upon looking at the sources in the master branch, I believe this bug still exists in the trunk.

Hope this helps.

Best Regards,
Paul Goins

[Diko Parvanov (dparv)]

moving back to new as Paul as stated how to reproduce this

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/868485

[sean mooney (sean-k-mooney)]

as i noted on the patch the limit comes form nova not glance.

nova dropped support for custom image properties in 2015

when we converted images to be modelled as oslo versioned objects in
https://github.com/openstack/nova/commit/40c2a4d6b05db879f976fc7eaf7a1a5aa93b270c

as part of the request spec object spec
https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/request-spec-object-mitaka.html

nova stores image properties in the isntance_system_metadata db table with a max value length of 255

https://github.com/openstack/nova/blob/9dceb735f6926c71a8c2e2c6ef8b5e0a1363041d/nova/db/main/models.py#L945-L946

volume metadata may exceed 255 but each image metadata value may not exceed 255

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/868485
Committed: https://opendev.org/openstack/cinder/commit/0bd1bd699d51162557dd0791ac6e79cb3149db8c
Submitter: "Zuul (22348)"
Branch: master

commit 0bd1bd699d51162557dd0791ac6e79cb3149db8c
Author: Jorge Merlino <email address hidden>
Date: Fri Dec 23 09:46:43 2022 -0300

    Increase size of volume image metadata values

    Volume image metadata values were limited to 255 characters but Glance
    allows up to 65535 (considering it uses a TEXT field in MySQL). Cinder
    database also uses a TEXT field for those values so it made no sense to
    limit them to 255. The actual values could already be longer when they
    were copied from the image at volume creation time.

    Closes-Bug: #1988942
    Change-Id: Id200ae93384a452b34bdd20dd1f3fc656ec5650f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/cinder/+/903828

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/cinder/+/905723

[Bartosz Woronicz (mastier1)]

The another case is that Trilio creates the volume when restoring our glance simplestreams image with long metadata.
When you create volume from image it works, when directly creating a volume in cinder it fails.

We need that patch in Yoga backported.
Marking Field Critical as it blocks the deployment.

[Jorge Merlino (jorge-merlino)]

Patch for jammy

[Jorge Merlino (jorge-merlino)]

Patch for mantic

[Jorge Merlino (jorge-merlino)]

Patch for uca yoga

[Jorge Merlino (jorge-merlino)]

Patch for uca zed

[Jorge Merlino (jorge-merlino)]

Patch for uca antelope

[Jorge Merlino (jorge-merlino)]

Patch for uca bobcat

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 24.0.0.0rc1 release candidate.

[Łukasz Zemczak (sil2100)]

The SRU looks okay, but any reason for the excessive versioning? The queues have 2:20.3.1-0ubuntu1.1.22.04.1 and 2:23.0.0-0ubuntu1.1.22.10.1 - the 22.10.1 and 22.04.1 suffixes seem unnecessary? Why not 2:23.0.0-0ubuntu1.2 and 2:20.3.1-0ubuntu1.2?

[Jorge Merlino (jorge-merlino)]

Hi Łukasz,

Sorry but I'm missing something here, the patches I submitted are versions 2:23.0.0-0ubuntu1.2 for mantic and 2:20.3.1-0ubuntu1.2 for jammy.

[Robie Basak (racb)]

@dgadomski looks like you signed the uploads. Please could you take a look and re-upload with the correct versions?

[James Page (james-page)]

Please can the current set of cinder uploads for mantic and jammy be rejected; we're going to consolidate this SRU with another one that's in the pipeline (bug 1987663) - I'll deal with merging them and uploading for SRU team review.

[Mauricio Faria de Oliveira (mfo)]

Hi James; done!

[Mauricio Faria de Oliveira (mfo)]

Uploaded to Mantic, Jammy.

This upload has been coordinated with the Openstack Engineering team (~james-page) for bugs 1987663, 1988942, and 1994521.
Test PPA and build-time unit test summary in https://bugs.launchpad.net/ubuntu/+source/cinder/+bug/1987663/comments/16

[Mauricio Faria de Oliveira (mfo)]

Cloud Archive:

Caracal is addressed with Noble
Bobcat is addressed with Mantic
* Antelope is _not_ addressed with Lunar (no longer supported), needs patching in UCA [Triaged]
Zed is no longer supported (2024/04)
Yoga is addressed with Jammy

[Timo Aaltonen (tjaalton)]

Hello Diko, or anyone else affected,

Accepted cinder into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cinder/2:23.0.0-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Diko, or anyone else affected,

Accepted cinder into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cinder/2:20.3.1-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.