Nova not detaching Cinder volume is volume is available

[Gorka Eguileor]

It is known that Cinder and Nova can sometime be out of sync, where one of them thinks that a volume is attached and the other doesn't.

With the Cinder attachment API an operator can now fix the issue where Nova says the volume is detached and Cinder says it's not. The operator just needs to run the "cinder attachment-delete" command on the specific attachment.

The opposite situation, Cinder says it's available and Nova says it's attached, cannot be currently fixed without modifying the Nova database and making manual os-brick calls or the appropriate CLI calls to detach the volume.

Ideally Nova should be able to call os-brick using the BDM information to locally detach the volume (using the force option which can lose data) and then not call cinder to do the detach since it already says the volume is not mapped.

One way to reproduce this is:

- Create a VM
- Create a volume
- Attach the volume
- Delete the attachment in cinder with "cinder attachment-delete <attach_id>"
- Trying to detach the volume in nova

The error we'll see is something like:

ERROR (BadRequest): Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-ec02147a-6b5b-40d2-991c-3d49207f5c9b) (HTTP 400) (Request-ID: req-d8ab82c5-cb32-446e-a8e9-fd8e30be0995)

[Uggla (rene-ribaud)]

stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 attachment-list
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| ID | Volume ID | Status | Server ID |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| 66824067-5b10-49fc-b3fa-84aa56ff0d55 | e6ed1a54-a57e-438d-8e0c-0289367a2a97 | attached | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
| 7001e8eb-b12d-4e5c-bfa5-2aeda7fbbb9b | 87b8bfd0-1470-4ebf-a13c-50df99ffed80 | attached | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+

stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 list
+--------------------------------------+--------+-------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------------+------+-------------+----------+--------------------------------------+
| 87b8bfd0-1470-4ebf-a13c-50df99ffed80 | in-use | | 10 | lvmdriver-1 | true | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
| e6ed1a54-a57e-438d-8e0c-0289367a2a97 | in-use | demo-volume | 1 | lvmdriver-1 | false | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
+--------------------------------------+--------+-------------+------+-------------+----------+--------------------------------------+

stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 attachment-delete 66824067-5b10-49fc-b3fa-84aa56ff0d55
stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 attachment-list
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| ID | Volume ID | Status | Server ID |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+
| 7001e8eb-b12d-4e5c-bfa5-2aeda7fbbb9b | 87b8bfd0-1470-4ebf-a13c-50df99ffed80 | attached | d0c23f72-50cc-4fcb-ba6d-c37a6d139e96 |
+--------------------------------------+--------------------------------------+----------+--------------------------------------+

stack@openstack:~/devstack$ cinder --os-volume-api-version 3.44 list
+--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------------+------+-------------+----------+--------------------------------------+
| 87b8bfd0-1470-4ebf-a13c-50df99ffed80 | in-use | | 10 | lvmdriver-1 | true | d0c23f72-50cc-4fcb-ba6d-c37...

[Uggla (rene-ribaud)]

Above lof done 20220913 using Zed.