Cinder volume isn't available after instance soft-deleted timer expired while volume is still attached

[Udi Shkalim]

Description of problem:
There is a feature in nova that allow you to restore a SOFT-DELETED instance (nova restore) when an instance is terminated there is a certain amount of time(defined in nova.conf - reclaim_instance_interval) for one to restore the instance including volume and floating IP attachments. Once this timer expire the instance goes to DELETED status and should release all its resources including the attached volume.
In this case the volume remain attached to an instance in DELETED state and not usable for a non admin user.

Version-Release number of selected component (if applicable):
# rpm -qa | grep -i cinder
openstack-cinder-2015.1.0-2.el7ost.noarch
python-cinder-2015.1.0-2.el7ost.noarch
python-cinderclient-1.1.1-1.el7ost.noarch

# rpm -qa | grep -i nova
openstack-nova-cert-2015.1.0-4.el7ost.noarch
openstack-nova-compute-2015.1.0-4.el7ost.noarch
openstack-nova-common-2015.1.0-4.el7ost.noarch
python-nova-2015.1.0-4.el7ost.noarch
openstack-nova-conductor-2015.1.0-4.el7ost.noarch
openstack-nova-scheduler-2015.1.0-4.el7ost.noarch
openstack-nova-console-2015.1.0-4.el7ost.noarch
python-novaclient-2.23.0-1.el7ost.noarch
openstack-nova-api-2015.1.0-4.el7ost.noarch
openstack-nova-novncproxy-2015.1.0-4.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Edit /etc/nova/nova.conf and change reclaim_instance_interval=60
2. Restart nova service
3. Create a volume and attach it to an instance
4. Delete instance - make sure its in "SOFT-DELETE" state
5. Wait for the timer to expire and make sure the instance is in "DELETED" state
6. Volume still shown as attached in CLI and in Horizon Its shown as attached to "None"

Actual results:
Volume is not usable

Expected results:
Volume should be released and usable

Additional info:
Attaching cinder and nova log dir

[Amador Pahim (apahim)]

I think all images attached to an instance should be available right after the delete command, even if the instance goes to soft_deleted state. Then, if restored, all volumes should be manually re-attached. Not sure if this is the best approach anyway. I'm sending a patch for review/discussion.

[Kashyap Chamarthy (kashyapc)]

[Looks like a Nova-related issue, since it is not requesting a volume detach from Cinder -- so moved this bug to Nova component.]

[Matt Riedemann (mriedem)]

This was reported against kilo, which is now end of life upstream. Can you reproduce with something newer like mitaka or better yet master?

[Alexandra (aallakhverdieva)]

Reproducible on mitaka-9.0
http://paste.openstack.org/show/543309/

[Alexandra (aallakhverdieva)]

diagnostic snapshot - https://drive.google.com/open?id=0BxmbtGZe1aPtVXJ6RDNsamV0bG8

[Anusha Unnam (anusha-unnam)]

I was able to reproduce this bug in current master. Steps I followed,

1.Edit /etc/nova/nova.conf and change reclaim_instance_interval=60

2.Restart nova service

3.Create volume:
cinder create --display_name myvol 2

4.cinder list
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+
| a481f9d0-9567-4b84-a529-55766fc275a3 | available | myvol | 2 | lvmdriver-1 | false | |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+

5.nova list
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------+
| 08151afa-bb0c-47d0-bb1c-e276e3c98cfb | test | ACTIVE | - | Running | private=10.0.0.2 |
+--------------------------------------+------+--------+------------+-------------+------------------+

6. nova volume-attach 08151afa-bb0c-47d0-bb1c-e276e3c98cfb a481f9d0-9567-4b84-a529-55766fc275a3

7. cinder list
+--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+
| a481f9d0-9567-4b84-a529-55766fc275a3 | in-use | myvol | 2 | lvmdriver-1 | false | 08151afa-bb0c-47d0-bb1c-e276e3c98cfb |
+--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+

8. nova delete test
Request to delete server test has been accepted.

9. waited 60secs(reclaim_instance_interval)

10. nova list
nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

11. cinder list
+--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+
| a481f9d0-9567-4b84-a529-55766fc275a3 | in-use | myvol | 2 | lvmdriver-1 | false | 08151afa-bb0c-47d0-bb1c-e276e3c98cfb

12. And also When i tried to attach myvol to another instance I got this error,

nova volume-attach 26147eb0-08d6-4e57-840e-95b8b92e4774 a481f9d0-9567-4b84-a529-55766fc275a3
ERROR (ClientException): Unexpected API Error. Please rep...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/384799

[Sean Dague (sdague)]

Automatically discovered version kilo in description. If this is incorrect, please update the description to include 'nova version: ...'

[Drew Freiberger (afreiberger)]

I have reproduced this issue in xenial/Ocata cloud. nova version: 15.0.7-0ubuntu1~cloud0.

From my testing and research, this appears to be a working as designed issue.

Note the paragraph "The second caveat" in this document:
https://raymii.org/s/articles/Openstack_Soft_Delete_-_recover_deleted_instances.html

What I found as a work-around is to follow the mysql db update noted in this document:
https://raymii.org/s/articles/Fix_inconsistent_Openstack_volumes_and_instances_from_Cinder_and_Nova_via_the_database.html#Set_a_volume_as_detached_in_Cinder

I also found that if I run:
openstack volume set --state=available <volume-uuid>
I can then attach the volume to another instance with:
openstack server add volume <server-uuid> <volume-uuid>

The new server gets the attachment and shows the attachment in openstack server show <uuid>, but the cinder database volume_attachment table is not updated to reflect the second attachment of the volume:

mysql> select * from volume_attachment where volume_id = 'aa651b54-33aa-4e3d-9d25-f30671535c1c';
+---------------------+------------+------------+---------+--------------------------------------+--------------------------------------+---------------+--------------------------------------+------------+---------------------+-------------+-------------+---------------+
| created_at | updated_at | deleted_at | deleted | id | volume_id | attached_host | instance_uuid | mountpoint | attach_time | detach_time | attach_mode | attach_status |
+---------------------+------------+------------+---------+--------------------------------------+--------------------------------------+---------------+--------------------------------------+------------+---------------------+-------------+-------------+---------------+
| 2017-12-04 17:34:25 | NULL | NULL | 0 | c51d3234-509a-4ed7-908b-4a8460e90164 | aa651b54-33aa-4e3d-9d25-f30671535c1c | NULL | 73e22c0b-0ee5-4f91-8400-95344463a60b | /dev/vdb | 2017-12-04 17:34:26 | NULL | rw | attached |
+---------------------+------------+------------+---------+--------------------------------------+--------------------------------------+---------------+--------------------------------------+------------+---------------------+-------------+-------------+---------------+
1 row in set (0.00 sec)

Above shows my soft_deleted instance's uuid. I can update this attachment via DB by changing this attachment's instance_uuid to the new instance's uuid, and now cinder shows proper state and functionality of detachment/delete, etc.

mysql> update volume_attachment set instance_uuid = '2ab262be-fc23-4e3b-a3fe-4fadb4097a22' where volume_id = 'aa651b54-33aa-4e3d-9d25-f30671535c1c';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0

I think nova-conductor/scheduler need updates to call cinder detach upon soft_delete reclaim_delete_interval passing.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/384799
Reason: This was fixed in I3c35bba43fee81baebe8261f546c1424ce3a3383 in Queens.