Simultaneous volume creation with the same image in multi-attach mode returns error.

[Mitya Eremeev]

environment:

Glance backend is Cinder.
Multi-attachment is enabled.

1. launch two commands one after another and don’t wait responses (in 2 terminal tabs)

$ openstack volume create --size 50 --image Ubuntu-18.04 volume_1

$ openstack volume create --size 50 --image Ubuntu-18.04 volume_2

2.see volumes statuses

$ openstack volume list
+--------------------------------------+----------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+----------+-----------+------+-------------+
| f1cbfc3c-263b-48fd-8ef0-5ad06d2e99ba | volume_2 | error | 50 | |
| 5bbb5dab-c001-4c19-a61d-f1e7f8be8170 | volume_1 | available | 50 | |
+--------------------------------------+----------+-----------+------+-------------+

3. see cinder logs

ERROR cinder.volume.api [req-41b9c765-827d-4c75-ac95-331c8fa0c840 cd15c27229234675ac78f3f83af1c60c 3a2aec7e5c5b46c2a366e471fa133066 - 3f2f1264e29e4cc9913aa2a63197866a 3f2f1264e29e4cc9913aa2a63197866a] Volume status must be available or in-use to reserve, but the status is attaching.

or similar error message, but the status in detaching.

expected behavior:
two volumes created successfully

actual behavior:
the second volume was not created because the first volume creation task reserved volume.

Bug in RHOSP: https://bugzilla.redhat.com/show_bug.cgi?id=1886013

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/805072

[Sofia Enriquez (lsofia-enriquez)]

As far as I can see, this is more a feature than a bug. We don't have anything in the documentation [1] saying that this is not possible, we probably need to add this to the known issues section.

[1] https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiattach.html

[Sofia Enriquez (lsofia-enriquez)]

I'm adding this to next week cinder meeting.

[Rajat Dhasmana (whoami-rajat)]

Hi,

Is the glance backend cinder in this case?
and which openstack release are you using?

[Mitya Eremeev (mitos)]

Hi,

yes, glance backend is cinder.
Openstack version - Victoria.

[Bence Romsics (bence-romsics)]

We just had a discussion in #openstack-cinder [1] about a problem that I believe shares the same root cause as this bug report. The logs I'd like to share were too big for pastebin, so I'm attaching them here.

[1] https://meetings.opendev.org/irclogs/%23openstack-cinder/%23openstack-cinder.2022-04-07.log.html#t2022-04-07T08:13:58

[Rajat Dhasmana (whoami-rajat)]

Hi Bence,

NOTE: This is more of a glance issue than cinder as the changes made are part of glance cinder store.

I did the same image save operation 10 times concurrently and was able to save all 10 images successfully.

stack@devstack-VirtualBox:~/devstack$ glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file one.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file two.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file three.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file four.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file five.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file six.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file seven.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file eight.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file nine.qcow2; glance image-download 18dc5568-1a47-44ff-a724-5ef8d44d9e08 --file ten.qcow2;

stack@devstack-VirtualBox:~/devstack$ ll *.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:20 eight.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:19 five.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:19 four.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:21 nine.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:17 one.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:20 seven.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:19 six.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:21 ten.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:18 three.qcow2
-rw-rw-r-- 1 stack stack 16300544 Apr 18 14:18 two.qcow2

Glance image:

stack@devstack-VirtualBox:~/devstack$ glance image-show 18dc5568-1a47-44ff-a724-5ef8d44d9e08
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | b874c39491a2377b8490f5f1e89761a4 |
| container_format | bare |
| created_at | 2022-04-18T08:14:24Z |
| direct_url | cinder://lvmdriver-1/8851784d-5bef-4379-8c37-dd178fff8e37 |
| disk_format | qcow2 |
| id | 18dc5568-1a47-44ff-a724-5ef8d44d9e08 |
| locations | [{"url": "cinder://lvmdriver-1/8851784d-5bef-4379-8c37-dd178fff8e37", |
| | "metadata": {"store": "lvmdriver-1"}}] |
| min_disk | 0 |
| min_ram | 0 |
| name | test-2-multiattach-lvm ...

[Rajat Dhasmana (whoami-rajat)]

Sorry about my last comment, i was performing the operation serially.

Performing the operation in parallel, I was able to reproduce the issue and the root cause is that the retry interval is very small (1 second). I've reported a similar bug in glance store launchpad page[1] and proposed a fix[2]. Thanks for reporting the issue.

[1] https://bugs.launchpad.net/glance-store/+bug/1969373
[2] https://review.opendev.org/c/openstack/glance_store/+/838326

[Gorka Eguileor (gorka)]

From cinder's perspective this is not a bug in our API.
We only accept a single attach/detach operation at any given time for a given volume. Callers supporting multi-attach need to be aware and add the appropriate retry mechanism.

There won't be any code changes in cinder for this bug, but leaving the bug open to add the appropriate documentation explicitly stating this.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Gorka Eguileor <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/805072
Reason: Thank you for working on this and for helping us to robustify the feature, but from cinder's perspective this is not a bug in our API.

We only accept a single attach/detach operation at any given time for a given volume. Callers supporting multi-attach need to be aware and add the appropriate retry mechanism.

Patch on the glance_store side will fix this issue: https://review.opendev.org/c/openstack/glance_store/+/838326