Simultaneous volume creation with the same image in multi-attach mode returns error

[Rajat Dhasmana]

Glance backend is Cinder.
Multi-attachment is enabled.

1. launch two commands one after another and don’t wait responses (in 2 terminal tabs)

$ openstack volume create --size 50 --image Ubuntu-18.04 volume_1

$ openstack volume create --size 50 --image Ubuntu-18.04 volume_2

2.see volumes statuses

$ openstack volume list
+--------------------------------------+----------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+----------+-----------+------+-------------+
| f1cbfc3c-263b-48fd-8ef0-5ad06d2e99ba | volume_2 | error | 50 | |
| 5bbb5dab-c001-4c19-a61d-f1e7f8be8170 | volume_1 | available | 50 | |
+--------------------------------------+----------+-----------+------+-------------+

3. see cinder logs

ERROR cinder.volume.api [req-41b9c765-827d-4c75-ac95-331c8fa0c840 cd15c27229234675ac78f3f83af1c60c 3a2aec7e5c5b46c2a366e471fa133066 - 3f2f1264e29e4cc9913aa2a63197866a 3f2f1264e29e4cc9913aa2a63197866a] Volume status must be available or in-use to reserve, but the status is attaching.

or similar error message, but the status in detaching.

expected behavior:
two volumes created successfully

actual behavior:
the second volume was not created because the first volume creation task reserved volume.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance_store/+/838326

[Bence Romsics (bence-romsics)]

Just for the record: there's some history (e.g. alternative reproduction steps) to this bug in another bug report:
https://bugs.launchpad.net/cinder/+bug/1940466

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/838326
Committed: https://opendev.org/openstack/glance_store/commit/ba4af147fb93d522f8cf396098891f4cbe7b30fb
Submitter: "Zuul (22348)"
Branch: master

commit ba4af147fb93d522f8cf396098891f4cbe7b30fb
Author: whoami-rajat <email address hidden>
Date: Mon Apr 18 21:43:14 2022 +0530

    Correct retry interval during attach volume

    When we try to simultaneously attach same volume multiple times
    (multiattach), there are multiple attachments created, suppose
    attachA and attachB. If attachA marks the volume "reserved" then
    attachB can't proceed until the volume is in "in-use" or "available"
    state. We retry until we reach any of these states for which we use
    the retrying library.

    The retrying library defaults to 1 second retry[1] (5 times) which causes
    the original failure as the volume takes time to transition between
    "reserved" -> "in-use" state. This patch corrects it by adding an
    exponential retry time and max exponential retry i.e.

    1: 2 ** 1 = 2 seconds
    2: 2 ** 2 = 4 seconds
    3: 2 ** 3 = 8 seconds
    4: 2 ** 4 = 16 seconds (but max wait time is 10 seconds)
    5: 2 ** 5 = 32 seconds (but max wait time is 10 seconds)

    [1] https://github.com/rholder/retrying/blob/master/retrying.py#L84

    Closes-Bug: #1969373

    Change-Id: I0094b044085d7f92b07ea86236de3b6efd7d67ea

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/glance_store/+/839203

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/glance_store/+/839204

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/839203
Committed: https://opendev.org/openstack/glance_store/commit/c6716cb8fbbdd75b3c5ce36f5fc9cede8b83bc0d
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit c6716cb8fbbdd75b3c5ce36f5fc9cede8b83bc0d
Author: whoami-rajat <email address hidden>
Date: Mon Apr 18 21:43:14 2022 +0530

    Correct retry interval during attach volume

    When we try to simultaneously attach same volume multiple times
    (multiattach), there are multiple attachments created, suppose
    attachA and attachB. If attachA marks the volume "reserved" then
    attachB can't proceed until the volume is in "in-use" or "available"
    state. We retry until we reach any of these states for which we use
    the retrying library.

    The retrying library defaults to 1 second retry[1] (5 times) which causes
    the original failure as the volume takes time to transition between
    "reserved" -> "in-use" state. This patch corrects it by adding an
    exponential retry time and max exponential retry i.e.

    1: 2 ** 1 = 2 seconds
    2: 2 ** 2 = 4 seconds
    3: 2 ** 3 = 8 seconds
    4: 2 ** 4 = 16 seconds (but max wait time is 10 seconds)
    5: 2 ** 5 = 32 seconds (but max wait time is 10 seconds)

    [1] https://github.com/rholder/retrying/blob/master/retrying.py#L84

    Closes-Bug: #1969373

    Change-Id: I0094b044085d7f92b07ea86236de3b6efd7d67ea
    (cherry picked from commit ba4af147fb93d522f8cf396098891f4cbe7b30fb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/839204
Committed: https://opendev.org/openstack/glance_store/commit/a4ba0c4fded13190f9161acedf6240f025718a77
Submitter: "Zuul (22348)"
Branch: stable/xena

commit a4ba0c4fded13190f9161acedf6240f025718a77
Author: whoami-rajat <email address hidden>
Date: Mon Apr 18 21:43:14 2022 +0530

    Correct retry interval during attach volume

    When we try to simultaneously attach same volume multiple times
    (multiattach), there are multiple attachments created, suppose
    attachA and attachB. If attachA marks the volume "reserved" then
    attachB can't proceed until the volume is in "in-use" or "available"
    state. We retry until we reach any of these states for which we use
    the retrying library.

    The retrying library defaults to 1 second retry[1] (5 times) which causes
    the original failure as the volume takes time to transition between
    "reserved" -> "in-use" state. This patch corrects it by adding an
    exponential retry time and max exponential retry i.e.

    1: 2 ** 1 = 2 seconds
    2: 2 ** 2 = 4 seconds
    3: 2 ** 3 = 8 seconds
    4: 2 ** 4 = 16 seconds (but max wait time is 10 seconds)
    5: 2 ** 5 = 32 seconds (but max wait time is 10 seconds)

    [1] https://github.com/rholder/retrying/blob/master/retrying.py#L84

    Closes-Bug: #1969373

    Change-Id: I0094b044085d7f92b07ea86236de3b6efd7d67ea
    (cherry picked from commit ba4af147fb93d522f8cf396098891f4cbe7b30fb)
    (cherry picked from commit c6716cb8fbbdd75b3c5ce36f5fc9cede8b83bc0d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 4.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 2.7.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 3.0.1 release.