Not able to create image with cinder as storage backend

[Hemanth Nakkina]

On Ussuri, creation of image with cinder as glance storage backend fails.

Reproduction Steps:

1. Deploy cloud environment - focal ussuri
2. Change the following configuration in /etc/glance/glance-api.conf to setup Cinder as Storage backend to Glance.

[DEFAULT]
enabled_backends = local:file, cinder:cinder
[glance_store]
default_backend = cinder

Restart glance-api service
systemctl restart glance-api.service

3. Upload a cirros image to glance
openstack image create --container-format bare --disk-format qcow2 --file /home/ubuntu/images/cirros-0.4.0-x86_64-disk.img test

The above command throws an exception.

Exception in /var/log/glance/glance-api.log
2021-10-20 06:34:34.894 279293 ERROR glance_store._drivers.cinder [req-282b09b7-3db4-44f7-937c-b7c2fb71453b da9d04a5652c41b98126e1a2b1ce9601 1c133b846e4e4948873aa3af847c23df - 4d430c7d67e0416a95e38b10aad4fc5f 4d430c7d67e0416a95e38b10aad4fc5f] Exception while accessing to cinder volume 7553d4cd-d315-4f97-8abe-f375864fa84a.: TypeError: temporary_chown() got an unexpected keyword argument 'backend'
2021-10-20 06:34:34.894 279293 ERROR glance_store._drivers.cinder Traceback (most recent call last):
2021-10-20 06:34:34.894 279293 ERROR glance_store._drivers.cinder File "/usr/lib/python3/dist-packages/glance_store/_drivers/cinder.py", line 575, in _open_cinder_volume
2021-10-20 06:34:34.894 279293 ERROR glance_store._drivers.cinder with self.temporary_chown(
2021-10-20 06:34:34.894 279293 ERROR glance_store._drivers.cinder File "/usr/lib/python3.8/contextlib.py", line 240, in helper
2021-10-20 06:34:34.894 279293 ERROR glance_store._drivers.cinder return _GeneratorContextManager(func, args, kwds)
2021-10-20 06:34:34.894 279293 ERROR glance_store._drivers.cinder File "/usr/lib/python3.8/contextlib.py", line 83, in __init__
2021-10-20 06:34:34.894 279293 ERROR glance_store._drivers.cinder self.gen = func(*args, **kwds)
2021-10-20 06:34:34.894 279293 ERROR glance_store._drivers.cinder TypeError: temporary_chown() got an unexpected keyword argument 'backend'
2021-10-20 06:34:34.894 279293 ERROR glance_store._drivers.cinder

Package used:
python3-glance-store/focal-updates,now 2.0.0-0ubuntu2 all [installed,automatic]

Analysis:

The method signature and the calling method args seems different for function temporary_chown
https://opendev.org/openstack/glance_store/src/tag/2.0.0/glance_store/_drivers/cinder.py#L458
https://opendev.org/openstack/glance_store/src/tag/2.0.0/glance_store/_drivers/cinder.py#L573

This is fixed in upstream as part of https://bugs.launchpad.net/glance-store/+bug/1870289
The bug fix for LP#1870289 need to be SRU'ed to Ubuntu python3-glance-store packages.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[Impact]
Not able to upload an image to glance

[Test Case]
1. Deploy cloud with focal ussuri
2. Configure cinder as glance storage backend

Ensure the following configurations in glance-api.conf
[DEFAULT]
enabled_backends = local:file, cinder:cinder
[glance_store]
default_backend = cinder

Restart glance-api service
systemctl restart glance-api.service

3. Upload an image to glance
openstack image create --container-format bare --disk-format qcow2 --file /home/ubuntu/images/cirros-0.4.0-x86_64-disk.img test

Image creation should be successful.

4. Repeat step 3 with creation of 5 images concurrently.
   Run the command in step3 in 5 different consoles.

Verify if image creation is successful for all of them.
'openstack image list' should list all the 5 images

[Regression Potential]
The fix enhanced the locking mechanism to support concurrent image creation requests in addition to correcting the function signature that caused the initial problem. The test case is enhanced to verify concurrent
creation of images to avoid any regressions.
Also the patch is verified upstream via CI unit test cases and functional test cases.

[Hemanth Nakkina (hemanth-n)]

The fix for LP#1870289 are available in UCA Victoria and further.

[Hemanth Nakkina (hemanth-n)]

Hi SRU team,

Uploaded debdiff's for focal and UCA ussuri.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1948439_focal.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Corey Bryant (corey.bryant)]

Thanks Hemanth. A new version of python-glance-store has been uploaded to the focal unapproved queue.

https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=python-glance-store

[Brian Murray (brian-murray)]

The "Regression Potential" section of the bug description should be an analysis of where things might go wrong and how those issues would manifest. This should give us insight into further test cases or things to look out for when testing the updated version of the package.

[Hemanth Nakkina (hemanth-n)]

@brian-murray I have updated the Regression Potential and modified the test case to verify some more scenarios. Please take a look.

[Brian Murray (brian-murray)]

Hello Hemanth, or anyone else affected,

Accepted python-glance-store into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-glance-store/2.0.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Hemanth Nakkina (hemanth-n)]

Verified the test case on focal and its successful.

Package installed:
$ sudo dpkg -l | grep glance-store
ii python3-glance-store 2.0.0-0ubuntu3 all OpenStack Image Service store library - Python 3.x
$ sudo systemctl restart glance-api.service

Create a single image
$ openstack image create --container-format bare --disk-format qcow2 --file /home/ubuntu/images/cirros-0.4.0-x86_64-disk.img test
+------------------+------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------+
| container_format | bare |
| created_at | 2021-10-28T03:27:03Z |
| disk_format | qcow2 |
| file | /v2/images/00207cb6-a16d-4aad-80ad-a78fbe37a454/file |
| id | 00207cb6-a16d-4aad-80ad-a78fbe37a454 |
| min_disk | 0 |
| min_ram | 0 |
| name | test |
| owner | 5f8dd88e2e2c436cb44098f4f63d0fe8 |
| properties | os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/test', owner_specified.openstack.sha256='' |
| protected | False |
| schema | /v2/schemas/image |
| status | queued |
| tags | |
| updated_at | 2021-10-28T03:27:0...

[Corey Bryant (corey.bryant)]

Hello Hemanth, or anyone else affected,

Accepted python-glance-store into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Hemanth Nakkina (hemanth-n)]

Verified the test case on bionic ussuri and is successful.

$ sudo dpkg -l | grep glance-store
ii python3-glance-store 2.0.0-0ubuntu3~cloud0 all OpenStack Image Service store library - Python 3.x

$ openstack image create --container-format bare --disk-format qcow2 --file /home/ubuntu/images/cirros-0.4.0-x86_64-disk.img test
+------------------+------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------+
| container_format | bare |
| created_at | 2021-10-29T10:10:47Z |
| disk_format | qcow2 |
| file | /v2/images/36aaf632-2a38-4998-b1cf-41915c69a4a4/file |
| id | 36aaf632-2a38-4998-b1cf-41915c69a4a4 |
| min_disk | 0 |
| min_ram | 0 |
| name | test |
| owner | 4d2649fe53014a57ba6188c0fea0d48a |
| properties | os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/test', owner_specified.openstack.sha256='' |
| protected | False |
| schema | /v2/schemas/image |
| status | queued |
| tags | |
| updated_at | 2021-10-29T10:10:47Z ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-glance-store - 2.0.0-0ubuntu3

---------------
python-glance-store (2.0.0-0ubuntu3) focal; urgency=medium

  * d/p/0001-Add-lock-per-share-for-cinder-nfs-mount-umount.patch: Cherry-picked
    from upstream to fix image creation failure with cinder as storage backend
    (LP: #1948439).

 -- Hemanth Nakkina <email address hidden> Fri, 22 Oct 2021 15:12:31 +0530

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for python-glance-store has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-glance-store has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-glance-store - 2.0.0-0ubuntu3~cloud0
---------------

 python-glance-store (2.0.0-0ubuntu3~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-glance-store (2.0.0-0ubuntu3) focal; urgency=medium
 .
   * d/p/0001-Add-lock-per-share-for-cinder-nfs-mount-umount.patch: Cherry-picked
     from upstream to fix image creation failure with cinder as storage backend
     (LP: #1948439).