Volume keep status 'attaching' even there is no any instance

[Matt Ye]

openstack queen

[root@controller ~]# openstack --version
openstack 3.14.1
[root@controller ~]# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
[root@controller ~]# cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+--------------------------------------+
| d27c3220-a764-4d6f-8f0f-eca051129788 | attaching | test1 | 5 | - | false | |
| da48687b-05c7-4b37-9e9b-4b1b1cef2c57 | in-use | VolumeFromISO | 5 | - | true | 1bcfd809-52c5-4600-a4b0-a1981dc844c9 |
+--------------------------------------+-----------+---------------+------+-------------+----------+--------------------------------------+
[root@controller ~]# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
[root@controller ~]#

reproduction steps:
1. create clean volume named 'test1';
2. create instance named 'vm1';
3. start vm1;
4. attach test1 to vm1;
5. delete vm1;

results:
1. vm1 deletion keep 'deleting', waiting for a long time, cannot be finished;
2. then I reboot all the controller, compute and cinder nodes;
3. after rebooting, check vm1 is not existed anymore.
4. check volume, it's status keep 'attaching' as above mentioned;

[Matt Ye (mattye)]

More info:
1. then all the volumes cannot be deleted, error messages will be give as below:

Unable to retrieve attachement information;
You are not allowed to delete volume....

2. Error in nova-scheduler.log
2018-07-02 06:50:27.407 1074 ERROR oslo.messaging._drivers.impl_rabbit [req-634c6562-bc91-4a17-89ef-2f82721e9fd6 - - - - -] [1b263e57-b569-4ac2-9c24-21c19b07e296] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-07-02 06:50:33.482 1074 INFO oslo.messaging._drivers.impl_rabbit [req-634c6562-bc91-4a17-89ef-2f82721e9fd6 - - - - -] [1b263e57-b569-4ac2-9c24-21c19b07e296] Reconnected to AMQP server on controller:5672 via [amqp] client with port 54388.

3. cinder/scheduler.log
2018-07-02 06:50:17.991 1068 ERROR oslo.messaging._drivers.impl_rabbit [req-b1a1397c-df71-49d7-9ef3-3e27c5d4eca8 - - - - -] [95e8936e-c097-4007-98d6-743af446c2d7] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-07-02 06:50:20.034 1068 ERROR oslo.messaging._drivers.impl_rabbit [req-b1a1397c-df71-49d7-9ef3-3e27c5d4eca8 - - - - -] [95e8936e-c097-4007-98d6-743af446c2d7] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
2018-07-02 06:50:24.099 1068 ERROR oslo.messaging._drivers.impl_rabbit [req-b1a1397c-df71-49d7-9ef3-3e27c5d4eca8 - - - - -] [95e8936e-c097-4007-98d6-743af446c2d7] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED