Cinder: Multiattach volume locks doesn't work as expected

[Rajat Dhasmana]

The attachment state manager[1] uses threading locks which doesn't work well when multiple processes are working on the same image-volume resource.
With the glance uwsgi deployment model, each glance process is independent of the other and doesn't recognize the locks acquired/released in other processes since they are threading locks.
This could be problematic if two different processes think it's OK to disconnect volume from the host when the other process is writing using the same image-volume.

We can handle this by replacing the threading locks with process locks/file locks to be shared across all glance processes but a simpler and more straightforward way is to rely on Cinder DB information regarding attachments and decide if we want to disconnect the volume or not (>1 attachment on same host).

Nova handles this by counting number of attachments on same host[2] but it's not trivial for the glance case since we don't have information about the host of the attachment.
We can get this from volume info but cinder currently doesn't expose this information for non-admins[3].

Following is the reproducer:
I've a devstack master setup with 4 glance workers
stack 1126342 0.0 0.1 61476 17232 ? Ss 14:16 0:00 glance-apiuWSGI master
stack 1126345 0.1 0.9 587312 150444 ? Sl 14:16 0:02 glance-apiuWSGI worker 1
stack 1126346 0.1 0.9 513496 150124 ? Sl 14:16 0:02 glance-apiuWSGI worker 2
stack 1126347 0.1 0.9 587296 150404 ? Sl 14:16 0:02 glance-apiuWSGI worker 3
stack 1126348 0.1 0.9 595216 158088 ? Sl 14:16 0:02 glance-apiuWSGI worker 4

With the following diff[4], I was able to see the issue:

With the logs[5], you can see multiple attach and detach requests overlapping, not to mention that attach and detach acquire same lock so even those requests shouldn't overlap.
Here are two examples of two attach requests overlapping with different PIDs

1.
PID 1126347
PID 1126345

Sep 27 14:18:03 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-04c4e0ca-b4f3-4142-b7f5-72d089bd394c admin admin] ++++ LOCKING for attaching volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:190}}
Sep 27 14:18:03 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-fc1e7d6d-0782-4640-b52a-aa7dc3f2b4a4 admin admin] ++++ LOCKING for attaching volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:190}}

2.
PID 1126348
PID 1126346

Sep 27 14:18:03 rdhasman-glance-cinder <email address hidden>[1126348]: DEBUG glance_store.common.attachment_state_manager [None req-b7ddde39-715b-4dd6-9b98-95e53d003b36 admin admin] ++++ LOCKING for attaching volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126348) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:190}}
Sep 27 14:18:03 rdhasman-glance-cinder <email address hidden>[1126346]: DEBUG glance_store.common.attachment_state_manager [None req-b9a53b1d-7582-417e-ab01-86ce067a62a5 admin admin] ++++ LOCKING for attaching volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126346) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:190}}

[1] https://github.com/openstack/glance_store/blob/111b04f54573933bf193f819f9e216c97e68de96/glance_store/common/attachment_state_manager.py
[2] https://github.com/openstack/nova/blob/b99a882366251f88d145e27312b94692e0b2266f/nova/virt/libvirt/driver.py#L2051
[3] https://github.com/openstack/cinder/blob/8064195521f9f511f630ff63b5dde1e88ff53e18/cinder/api/v2/views/volumes.py#L131-L132
[4]
diff --git a/glance_store/common/attachment_state_manager.py b/glance_store/common/attachment_state_manager.py
index 04cad12..81ef166 100644
--- a/glance_store/common/attachment_state_manager.py
+++ b/glance_store/common/attachment_state_manager.py
@@ -187,6 +187,7 @@ class _AttachmentState(object):
         with self._get_locked(volume_id) as vol_attachment:

             try:
+ LOG.debug("++++ LOCKING for attaching volume %s", volume_id)
                 attachment = self.volume_api.attachment_create(
                     client, volume_id, mode=mode)
             except Exception:
@@ -197,6 +198,7 @@ class _AttachmentState(object):

             vol_attachment.add_attachment(attachment['id'], host)

+ LOG.debug("++++ UNLOCKING after attaching volume %s", volume_id)
         LOG.debug('_AttachmentState.attach for volume_id=%(volume_id)s '
                   'and attachment_id=%(attachment_id)s completed successfully',
                   {'volume_id': volume_id, 'attachment_id': attachment['id']})
@@ -222,6 +224,7 @@ class _AttachmentState(object):
                   {'volume_id': volume_id, 'attachment_id': attachment_id})
         with self._get_locked(volume_id) as vol_attachment:
             try:
+ LOG.debug("---- LOCKING for detaching volume %s", volume_id)
                 vol_attachment.remove_attachment(attachment_id, host)
             except KeyError:
                 LOG.warning(_LW("Request to remove attachment "
@@ -230,8 +233,10 @@ class _AttachmentState(object):
                             {'volume_id': volume_id, 'host': host})

             if not vol_attachment.in_use():
+ LOG.debug("**** INSIDE CHECK FOR disconnect volume %s", volume_id)
                 conn.disconnect_volume(device)
                 del self.volumes[volume_id]
+ LOG.debug("**** OUTSIDE CHECK FOR disconnect volume %s", volume_id)
             self.volume_api.attachment_delete(client, attachment_id)

             LOG.debug('_AttachmentState.detach for volume %(volume_id)s '
@@ -239,6 +244,7 @@ class _AttachmentState(object):
                       'successfully',
                       {'volume_id': volume_id,
                        'attachment_id': attachment_id})
+ LOG.debug("---- UNLOCKING for detaching volume %s", volume_id)

 __manager__ = _AttachmentStateManager(HOST)

[5]
Sep 27 14:18:03 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-04c4e0ca-b4f3-4142-b7f5-72d089bd394c admin admin] ++++ LOCKING for attaching volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:190}}
Sep 27 14:18:03 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-fc1e7d6d-0782-4640-b52a-aa7dc3f2b4a4 admin admin] ++++ LOCKING for attaching volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:190}}
Sep 27 14:18:03 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-fc1e7d6d-0782-4640-b52a-aa7dc3f2b4a4 admin admin] ++++ UNLOCKING after attachi
ng volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:201}}
Sep 27 14:18:03 rdhasman-glance-cinder <email address hidden>[1126348]: DEBUG glance_store.common.attachment_state_manager [None req-b7ddde39-715b-4dd6-9b98-95e53d003b36 admin admin] ++++ LOCKING for attaching v
olume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126348) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:190}}
Sep 27 14:18:03 rdhasman-glance-cinder <email address hidden>[1126346]: DEBUG glance_store.common.attachment_state_manager [None req-b9a53b1d-7582-417e-ab01-86ce067a62a5 admin admin] ++++ LOCKING for attaching v
olume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126346) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:190}}
Sep 27 14:18:08 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-fc1e7d6d-0782-4640-b52a-aa7dc3f2b4a4 admin admin] ---- LOCKING for detaching v
olume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:227}}
Sep 27 14:18:08 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-fc1e7d6d-0782-4640-b52a-aa7dc3f2b4a4 admin admin] **** INSIDE CHECK FOR discon
nect volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:236}}
Sep 27 14:18:09 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-fc1e7d6d-0782-4640-b52a-aa7dc3f2b4a4 admin admin] **** OUTSIDE CHECK FOR disco
nnect volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:239}}
Sep 27 14:18:09 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-04c4e0ca-b4f3-4142-b7f5-72d089bd394c admin admin] ++++ UNLOCKING after attachi
ng volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:201}}
Sep 27 14:18:11 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-fc1e7d6d-0782-4640-b52a-aa7dc3f2b4a4 admin admin] ---- UNLOCKING for detaching
 volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:247}}
Sep 27 14:18:11 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-404adb88-8eda-4937-b2f9-ac1128d0b52e admin admin] ++++ LOCKING for attaching v
olume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:190}}
Sep 27 14:18:16 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-04c4e0ca-b4f3-4142-b7f5-72d089bd394c admin admin] ---- LOCKING for detaching v
olume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:227}}
Sep 27 14:18:16 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-04c4e0ca-b4f3-4142-b7f5-72d089bd394c admin admin] **** INSIDE CHECK FOR discon
nect volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:236}}
Sep 27 14:18:16 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-04c4e0ca-b4f3-4142-b7f5-72d089bd394c admin admin] **** OUTSIDE CHECK FOR disco
nnect volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:239}}
Sep 27 14:18:17 rdhasman-glance-cinder <email address hidden>[1126348]: DEBUG glance_store.common.attachment_state_manager [None req-b7ddde39-715b-4dd6-9b98-95e53d003b36 admin admin] ++++ UNLOCKING after attachi
ng volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126348) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:201}}
Sep 27 14:18:19 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-04c4e0ca-b4f3-4142-b7f5-72d089bd394c admin admin] ---- UNLOCKING for detaching
 volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:247}}
Sep 27 14:18:24 rdhasman-glance-cinder <email address hidden>[1126348]: DEBUG glance_store.common.attachment_state_manager [None req-b7ddde39-715b-4dd6-9b98-95e53d003b36 admin admin] ---- LOCKING for detaching v
olume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126348) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:227}}
Sep 27 14:18:24 rdhasman-glance-cinder <email address hidden>[1126348]: DEBUG glance_store.common.attachment_state_manager [None req-b7ddde39-715b-4dd6-9b98-95e53d003b36 admin admin] **** INSIDE CHECK FOR discon
nect volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126348) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:236}}
Sep 27 14:18:24 rdhasman-glance-cinder <email address hidden>[1126348]: DEBUG glance_store.common.attachment_state_manager [None req-b7ddde39-715b-4dd6-9b98-95e53d003b36 admin admin] **** OUTSIDE CHECK FOR disco
nnect volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126348) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:239}}
Sep 27 14:18:25 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-404adb88-8eda-4937-b2f9-ac1128d0b52e admin admin] ++++ UNLOCKING after attachi
ng volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:201}}
Sep 27 14:18:27 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-b9a53b1d-7582-417e-ab01-86ce067a62a5 admin admin] ++++ LOCKING for attaching v
olume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:190}}
Sep 27 14:18:27 rdhasman-glance-cinder <email address hidden>[1126348]: DEBUG glance_store.common.attachment_state_manager [None req-b7ddde39-715b-4dd6-9b98-95e53d003b36 admin admin] ---- UNLOCKING for detaching
 volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126348) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:247}}
Sep 27 14:18:33 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-b9a53b1d-7582-417e-ab01-86ce067a62a5 admin admin] ++++ UNLOCKING after attachi
ng volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) attach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:201}}
Sep 27 14:18:34 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-404adb88-8eda-4937-b2f9-ac1128d0b52e admin admin] ---- LOCKING for detaching v
olume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:227}}
Sep 27 14:18:34 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-404adb88-8eda-4937-b2f9-ac1128d0b52e admin admin] **** INSIDE CHECK FOR discon
nect volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:236}}
Sep 27 14:18:34 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-404adb88-8eda-4937-b2f9-ac1128d0b52e admin admin] **** OUTSIDE CHECK FOR disco
nnect volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:239}}
Sep 27 14:18:37 rdhasman-glance-cinder <email address hidden>[1126345]: DEBUG glance_store.common.attachment_state_manager [None req-404adb88-8eda-4937-b2f9-ac1128d0b52e admin admin] ---- UNLOCKING for detaching
 volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126345) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:247}}
Sep 27 14:18:48 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-b9a53b1d-7582-417e-ab01-86ce067a62a5 admin admin] ---- LOCKING for detaching v
olume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:227}}
Sep 27 14:18:48 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-b9a53b1d-7582-417e-ab01-86ce067a62a5 admin admin] **** INSIDE CHECK FOR discon
nect volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:236}}
Sep 27 14:18:49 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-b9a53b1d-7582-417e-ab01-86ce067a62a5 admin admin] **** OUTSIDE CHECK FOR disco
nnect volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:239}}
Sep 27 14:18:51 rdhasman-glance-cinder <email address hidden>[1126347]: DEBUG glance_store.common.attachment_state_manager [None req-b9a53b1d-7582-417e-ab01-86ce067a62a5 admin admin] ---- UNLOCKING for detaching
 volume 62db8429-94b9-4f32-aff8-042f030b40c5 {{(pid=1126347) detach /opt/stack/data/venv/lib/python3.10/site-packages/glance_store/common/attachment_state_manager.py:247}}

[Rajat Dhasmana (whoami-rajat)]

After applying changes in topic[1], I was able to see the locks working correctly.
Created 10 volumes from a cirros raw image.

1. The lock was created

$ ls -l /opt/stack/data/glance/locks/
-rw-r--r-- 1 stack stack 0 Sep 27 21:10 glance_store-62db8429-94b9-4f32-aff8-042f030b40c5

2. The locking of _disconnect_and_detach_with_lock was synchronized and execution didn't overlap[2]

3. disconnect_volume was called for all of the attachments[3]

4. attachment_delete was called for all the attachments[4]

All volumes were created successfully and the image-volume was eventually in ``available`` state[5]

[1] https://review.opendev.org/q/topic:%22fix-multiattach-handling%22
[2]

Sep 27 21:10:51 rdhasman-glance-cinder <email address hidden>[1362280]: DEBUG oslo_concurrency.lockutils [None req-24670f91-39e6-4b8d-b1fb-779aa27b978b admin admin] Lock "62db8429-94b9-4f32-aff8-042f030b40c5" ac
quired by "glance_store._drivers.cinder.store.Store._disconnect_and_detach.<locals>._disconnect_and_detach_with_lock" :: waited 0.001s {{(pid=1362280) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:410}} Sep 27 21:10:54 rdhasman-glance-cinder <email address hidden>[1362280]: DEBUG oslo_concurrency.lockutils [None req-24670f91-39e6-4b8d-b1fb-779aa27b978b admin admin] Lock "62db8429-94b9-4f32-aff8-042f030b40c5" "r
eleased" by "glance_store._drivers.cinder.store.Store._disconnect_and_detach.<locals>._disconnect_and_detach_with_lock" :: held 2.742s {{(pid=1362280) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:424}}

Sep 27 21:10:59 rdhasman-glance-cinder <email address hidden>[1362277]: DEBUG oslo_concurrency.lockutils [None req-1d1a75cf-7a29-4212-9318-0a0399d8b35d admin admin] Lock "62db8429-94b9-4f32-aff8-042f030b40c5" ac
quired by "glance_store._drivers.cinder.store.Store._disconnect_and_detach.<locals>._disconnect_and_detach_with_lock" :: waited 0.001s {{(pid=1362277) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:410}} Sep 27 21:11:02 rdhasman-glance-cinder <email address hidden>[1362277]: DEBUG oslo_concurrency.lockutils [None req-1d1a75cf-7a29-4212-9318-0a0399d8b35d admin admin] Lock "62db8429-94b9-4f32-aff8-042f030b40c5" "r
eleased" by "glance_store._drivers.cinder.store.Store._disconnect_and_detach.<locals>._disconnect_and_detach_with_lock" :: held 2.758s {{(pid=1362277) inner /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py:424}} ...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/962408
Committed: https://opendev.org/openstack/glance_store/commit/915dffabc19493e0116cd2bf79c8b4f4dd47c9a3
Submitter: "Zuul (22348)"
Branch: master

commit 915dffabc19493e0116cd2bf79c8b4f4dd47c9a3
Author: Rajat Dhasmana <email address hidden>
Date: Fri Sep 26 21:28:25 2025 +0000

    Cinder: Handle multiattach volumes in multi worker env

    Since glance completed the support for uWSGI[1], devstack[2]
    along with other deployment tools have started to default
    glance in uWSGI mode.
    The problem we currently face is that the attachment state
    manager (written with threading locks) is not capable of
    handling multiple uWSGI processes accessing the same image-volume
    with multiple attach/detach operations which are not coordinated.

    This patch aims to address the problem with a different approach
    keeping the following things in mind:

    1. Attachment create doesn't require locking since we already handle
    it with Cinder attachment states ('reserved', 'attaching', 'in-use')
    and the retry mechanism while creating attachment[3]

    2. While disconnecting, we need to only perform it if there are
    no other attachments for same volume

    3. Delete attachment should always be performed to remove
    attachment records from Cinder DB to make decision for step 2.
    NOTE that Cinder volume driver handles case for multiattach
    volumes and avoids unmapping if LUN is attached > 1 times.

    With the following in mind, the new approach is:

    1. Stop persisting attachment state in memory and rely on
    Cinder DB since that is more reliable and fault tolerant.

    2. Removing locking from attachment create call

    3. Use external/file locks for disconnect and detach operations

    4. While disconnecting, acquire file lock and count number
    of attachments for a specific volume and don't disconnect if
    attachments > 1

    5. For non-multiattach volumes, always disconnect

    6. Delete the volume attachment inside file lock to ensure
    the attachment counting is accurate.

    [1] https://review.opendev.org/c/openstack/glance/+/742065
    [2] https://review.opendev.org/c/openstack/devstack/+/932201
    [3] https://review.opendev.org/c/openstack/glance_store/+/786410

    Closes-Bug: #1965679
    Closes-Bug: #2125840

    Depends-On: https://review.opendev.org/c/openstack/cinder/+/962409

    Signed-off-by: Rajat Dhasmana <email address hidden>
    Change-Id: I32fce5aeebe3319f167affa954c681598f81ad74

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2025.2
Review: https://review.opendev.org/c/openstack/glance_store/+/966034

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2025.1
Review: https://review.opendev.org/c/openstack/glance_store/+/966035

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance_store 5.3.0 Gazpacho release.