volume_attachement entries are not getting deleted from DB

[Hemant Sonawane]

Hello
I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually to make it work.

For e.g I shelved some instance via openstack horizon to reproduce the same scenario.

Here are the detailed attachment entries you can see for the shelved instance.

+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·

| id | volume_id | attached_host | instance_uuid | attach_status | connector │·

                                                                                                                                                                                            | │·

+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·

| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | nfv1compute43 | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | attached | {"platform": "x86_64", "os_type": "linux│·

", "ip": "10.42.168.87", "host": "nfv1compute43", "multipath": false, "do_local_attach": false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint": "/dev/vda"} | │·

| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved | NULL │·

                                                                                                                                                                                            | │·

+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------│·

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ │·

2 rows in set (0.00 sec)

If I would like to unshelve this instance it wont work as it has a duplicate entry in cinder db for the attachment. So i have to delete it manually from db or via cli.

root@master01:/home/hemant# cinder --os-volume-api-version 3.27 attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a │·

| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | attached | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | │·

| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | reserved | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |

cinder --os-volume-api-version 3.27 attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f

I could only gather logs from nova-compute

nova-compute logs:

cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume

This is the only choice I have if I would like to unshelve vm. But this is not a good approach for production env to delete duplicate volume attachments entries every time for every vm.

Is there any way to fix this issue ?

[Sofia Enriquez (lsofia-enriquez)]

Hi Hermant, hope this message finds you well.

From the output of the attachment it looks like the volume is attached to two different server IDs. Is the volume multi-attach to several instances?

Would you mind sharing the c-vol logs to see if there's any error on the cinder side?

Thanks
Regards,
Sofia

[Hemant Sonawane (hemson95)]

Hello Sofia,

Thank you again. The volume is not multi-attach it is just attached to one server. We have no idea how its creating another entry for the volume-attachment after the shelving server and keep the volume in attached state. Interesting part we do not see anything in cinder logs. The logs I shared in bug report are from nova. Do you think it could be related to nova ?

I have also found something similar but the bug it seems in nova so not really sure

https://bugzilla.redhat.com/show_bug.cgi?id=1941054#c4

Thanks and Regards,
Hemant

[Hemant Sonawane (hemson95)]

Hello Sofia,

Here are some more detailed logs from nova-compute

2022-11-25 13:16:28.119 17272 ERROR nova.volume.cinder [req-aee88125-574b-4efd-85d2-cf8a6baceaf6 d23ef0fae9914621835c5f8bb2bca36e 00d3d6fa37cc4612b786e8d0253553bb - default default] Update attachment failed for attachment 683daabf-78f2-4f4c-9048-fb04ee8e0ca9. Error: Una
ble to update attachment.(Invalid volume: duplicate connectors detected on volume b942a2d5-1f93-410b-a7d2-4c975eff5fc3). (HTTP 500) (Request-ID: req-2d181126-10f9-41c1-92bb-777455cc092e) Code: 500: cinderclient.exceptions.ClientException: Unable to update attachment.(In
valid volume: duplicate connectors detected on volume b942a2d5-1f93-410b-a7d2-4c975eff5fc3). (HTTP 500) (Request-ID: req-2d181126-10f9-41c1-92bb-777455cc092e)
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [req-aee88125-574b-4efd-85d2-cf8a6baceaf6 d23ef0fae9914621835c5f8bb2bca36e 00d3d6fa37cc4612b786e8d0253553bb - default default] [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] Instance failed block device setup: c
inderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume b942a2d5-1f93-410b-a7d2-4c975eff5fc3). (HTTP 500) (Request-ID: req-2d181126-10f9-41c1-92bb-777455cc092e)
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] Traceback (most recent call last):
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 1976, in _prep_block_device
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] driver_block_device.attach_block_devices(
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 874, in attach_block_devices
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] _log_and_attach(device)
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 871, in _log_and_attach
2022-11-25 13:16:28.164 17272 ERROR nova.compute.manager [instance: 9266a2d7-9721-4994-a6b5-6b3290862dc6] bdm.attach(*attach_args, **attach_kwargs) ...

[Sofia Enriquez (lsofia-enriquez)]

I wasn't able to reproduce this problem on master branch + LVM driver. The entries on the volume_attachment table are updating correctly. Looks like the problem is fixed in master.

Steps I've tried:
1. Create a cirros server
2. Attach a LVM volume
3. Shelve the server
4. Unshelve the server

Let me know if this assumption is not correct but I think this involves Nova scope and not Cinder. Because of that I'm moving this bug to Nova for future assistance.

Thanks,
Cheers,
Sofia

[Uggla (rene-ribaud)]

Hello Hement,

Can you please check if the instance is shelve or shelve_offloaded.
Note: There is the shelved_offload_time and shelved_poll_interval parameter (https://docs.openstack.org/nova/latest/configuration/config.html) that define the time an instance is kept shelve then moved to to shelve_offloaded.

Can you confirm there is a device in reserved state created as soon as you shelve the instance ?

[Hemant Sonawane (hemson95)]

Hello Uggla,

I can confirm that instance is shelve_offloaded. But once I shelve the instance I see 2 entries in db one with reserved status and one with attached status. Take a look at following db entries of vm volume attachments.

1) db entry I delete every time of shelve_offloaded vm:

2023-03-09 11:48:28 2023-03-09 11:48:31 NULL 0 682f21cf-8dd0-4782-9c81-01dc099f32db 80ae4cb8-e93d-49fa-bfc3-6b5e5c282a54 xxxxxxx 2aca9b66-a3aa-4460-a17d-0a31c6c10656 /dev/vda 2023-03-09 11:48:30 NULL rw attached "{""name"": ""volumes/80ae4cb8-e93d-49fa-bfc3-6b5e5c282a54"", ""hosts"": [""xxxxx"", ""xxxxxx"", ""xxxx""], ""ports"": [""6789"", ""6789"", ""6789""], ""cluster_name"": ""ceph"", ""auth_enabled"": true, ""auth_username"": ""xxxxx"", ""secret_type"": ""xxx"", ""secret_uuid"": ""xxxxx"", ""volume_id"": ""xxxxxx"", ""discard"": true, ""qos_specs"": null, ""access_mode"": ""rw"", ""encrypted"": false, ""cacheable"": false, ""driver_volume_type"": ""rbd"", ""attachment_id"": ""682f21cf-8dd0-4782-9c81-01dc099f32db""}" "{""platform"": ""x86_64"", ""os_type"": ""linux"", ""ip"": ""10.44.168.88"", ""host"": ""xxxxxx"", ""multipath"": false, ""do_local_attach"": false, ""system uuid"": ""xxxxx"", ""mountpoint"": ""/dev/vda""}"

2) db entry which being used after unshelving vm:

2023-03-09 11:50:31 2023-03-09 11:50:31 NULL 0 9b998cd9-accb-4963-a79f-627552951a4e 80ae4cb8-e93d-49fa-bfc3-6b5e5c282a54 NULL 2aca9b66-a3aa-4460-a17d-0a31c6c10656 NULL NULL NULL null reserved {} NULL

cli :
root# cinder --os-volume-api-version 3.27 attachment-list --all | grep 2aca9b66-a3aa-4460-a17d-0a31c6c10656

| 682f21cf-8dd0-4782-9c81-01dc099f32db | 80ae4cb8-e93d-49fa-bfc3-6b5e5c282a54 | attached | 2aca9b66-a3aa-4460-a17d-0a31c6c10656 |
| 9b998cd9-accb-4963-a79f-627552951a4e | 80ae4cb8-e93d-49fa-bfc3-6b5e5c282a54 | reserved | 2aca9b66-a3aa-4460-a17d-0a31c6c10656 |

In this case I have to delete the entry from db or via cli which has "attach_status = attached" and then unshelving the vm works fine which uses the 2nd entry and attach volume device to instance. This happens everytime when we do shelve, unshelve, delete and resize operations.