Volume Type deletion error (image_volume_cache_enabled = True) - stable/xena

[Ai Hamano]

VolumeType cannot be deleted when "image_volume_cache_enabled = True" is set in "cinder.conf".
In the case of "image_volume_cache_enabled = True", the volume cache is created when the Volume is first created.
The volume cache is created by the VolumeType used in the original volume, so I cannot delete the VolumeType until the volume cache is deleted.
I can remove the Volume Type by manually deleting the volume cache before deleting the Volume Type.
But it doesn't seem to make sense for cache, and it's difficult to implement when using Stack.
Is there a way to control Volume and Volume Type by Stack in the case of “image_volume_cache_enabled = True”?

The procedure for causing the error is as follows.
1. create Stack
  1-1. create VolumeType
  1-2. create Volume
2. delete Stack
  2-1. delete Volume
  2-2. delete VolumeType <-- ERROR

* Error message
```
Resource DELETE failed: BadRequest: resources.multi: Target volume type is still in use. (HTTP 400) (Request-ID: req-ee7a653a-b6cd-45d7-9f5e-c43dcb011af6)
```
* Operation log
0. initial state
```
$ openstack volume list

$ openstack volume type list
+--------------------------------------+-------------+-----------+
| ID | Name | Is Public |
+--------------------------------------+-------------+-----------+
| afcc773f-109c-45f5-af11-776bb95c2493 | lvmdriver-1 | True |
| 0ac7b1cc-586e-412b-a30e-117f64e273dd | __DEFAULT__ | True |
+--------------------------------------+-------------+-----------+

mysql> select * from volumes;
Empty set (0.00 sec)
mysql> select * from volume_types;
+---------------------+---------------------+------------+---------+--------------------------------------+-------------+--------------+-----------+---------------------+
| created_at | updated_at | deleted_at | deleted | id | name | qos_specs_id | is_public | description |
+---------------------+---------------------+------------+---------+--------------------------------------+-------------+--------------+-----------+---------------------+
| 2021-11-09 02:01:54 | 2021-11-09 02:01:54 | NULL | 0 | 0ac7b1cc-586e-412b-a30e-117f64e273dd | __DEFAULT__ | NULL | 1 | Default Volume Type |
| 2021-11-09 02:07:28 | NULL | NULL | 0 | afcc773f-109c-45f5-af11-776bb95c2493 | lvmdriver-1 | NULL | 1 | NULL |
+---------------------+---------------------+------------+---------+--------------------------------------+-------------+--------------+-----------+---------------------+
2 rows in set (0.00 sec)
```

1. create Stack
```
$ openstack stack create --template stack-for-volume.yaml test-stack
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| id | 4908b2ab-2d49-43b9-b435-268b99587ed2 |
| stack_name | test-stack |
| description | Cinder test |
| creation_time | 2021-12-08T07:22:15Z |
| updated_time | None |
| stack_status | CREATE_IN_PROGRESS |
| stack_status_reason | Stack CREATE started |
+---------------------+--------------------------------------+
$ openstack stack list
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| ID | Stack Name | Project | Stack Status | Creation Time | Updated Time |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
| 4908b2ab-2d49-43b9-b435-268b99587ed2 | test-stack | 16ca0ec90a274db5b2cadb22ff7763cb | CREATE_COMPLETE | 2021-12-08T07:22:15Z | None |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+--------------+
$ openstack volume list
+--------------------------------------+-------------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-------------+-----------+------+-------------+
| 6b4b1bc1-3f2f-4701-8dbf-12cdc08aa487 | test-volume | available | 8 | |
+--------------------------------------+-------------+-----------+------+-------------+
$ openstack volume type list
+--------------------------------------+-----------------+-----------+
| ID | Name | Is Public |
+--------------------------------------+-----------------+-----------+
| bd5c3bc5-92ef-4ae0-9109-c3457bdc9c60 | test-volumetype | True |
| afcc773f-109c-45f5-af11-776bb95c2493 | lvmdriver-1 | True |
| 0ac7b1cc-586e-412b-a30e-117f64e273dd | __DEFAULT__ | True |
+--------------------------------------+-----------------+-----------+

mysql> select display_name,deleted,id,user_id,project_id,status,volume_type_id from volumes;
+--------------------------------------------+---------+--------------------------------------+----------------------------------+----------------------------------+-----------+--------------------------------------+
| display_name | deleted | id | user_id | project_id | status | volume_type_id |
+--------------------------------------------+---------+--------------------------------------+----------------------------------+----------------------------------+-----------+--------------------------------------+
| image-64090ba9-d3b0-4bbe-8ec1-f30d90b7ab56 | 0 | 5768d777-39a0-4ef8-96b9-848b073bf857 | 867ea1a8313b47e09cf2c5904f89363e | 8e8baeadc93548c18df078a2c0ecb034 | available | bd5c3bc5-92ef-4ae0-9109-c3457bdc9c60 |
| test-volume | 0 | 6b4b1bc1-3f2f-4701-8dbf-12cdc08aa487 | 867ea1a8313b47e09cf2c5904f89363e | 16ca0ec90a274db5b2cadb22ff7763cb | available | bd5c3bc5-92ef-4ae0-9109-c3457bdc9c60 |
+--------------------------------------------+---------+--------------------------------------+----------------------------------+----------------------------------+-----------+--------------------------------------+
2 rows in set (0.00 sec)
mysql> select * from volume_types;
+---------------------+---------------------+------------+---------+--------------------------------------+-----------------+--------------+-----------+---------------------+
| created_at | updated_at | deleted_at | deleted | id | name | qos_specs_id | is_public | description |
+---------------------+---------------------+------------+---------+--------------------------------------+-----------------+--------------+-----------+---------------------+
| 2021-11-09 02:01:54 | 2021-11-09 02:01:54 | NULL | 0 | 0ac7b1cc-586e-412b-a30e-117f64e273dd | __DEFAULT__ | NULL | 1 | Default Volume Type |
| 2021-11-09 02:07:28 | NULL | NULL | 0 | afcc773f-109c-45f5-af11-776bb95c2493 | lvmdriver-1 | NULL | 1 | NULL |
| 2021-12-08 07:22:15 | NULL | NULL | 0 | bd5c3bc5-92ef-4ae0-9109-c3457bdc9c60 | test-volumetype | NULL | 1 | NULL |
+---------------------+---------------------+------------+---------+--------------------------------------+-----------------+--------------+-----------+---------------------+
3 rows in set (0.00 sec)
```

2. delete Stack
```
$ openstack stack delete 4908b2ab-2d49-43b9-b435-268b99587ed2
Are you sure you want to delete this stack(s) [y/N]? y
$ openstack stack list
+--------------------------------------+------------+----------------------------------+---------------+----------------------+----------------------+
| ID | Stack Name | Project | Stack Status | Creation Time | Updated Time |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+----------------------+
| 4908b2ab-2d49-43b9-b435-268b99587ed2 | test-stack | 16ca0ec90a274db5b2cadb22ff7763cb | DELETE_FAILED | 2021-12-08T07:22:15Z | 2021-12-08T07:32:00Z |
+--------------------------------------+------------+----------------------------------+---------------+----------------------+----------------------+
$ openstack stack show 4908b2ab-2d49-43b9-b435-268b99587ed2
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | 4908b2ab-2d49-43b9-b435-268b99587ed2 |
| stack_name | test-stack |
| description | Cinder test |
| creation_time | 2021-12-08T07:22:15Z |
| updated_time | 2021-12-08T07:32:00Z |
| stack_status | DELETE_FAILED |
| stack_status_reason | Resource DELETE failed: BadRequest: resources.multi: Target volume type is still in use. (HTTP 400) (Request-ID: req-ee7a653a-b6cd-45d7-9f5e-c43dcb011af6) |
| parameters | OS::project_id: 16ca0ec90a274db5b2cadb22ff7763cb |
| | OS::stack_id: 4908b2ab-2d49-43b9-b435-268b99587ed2 |
| | OS::stack_name: test-stack |
| | |
| outputs | [] |
| | |
| links | - href: http://127.0.0.1/heat-api/v1/16ca0ec90a274db5b2cadb22ff7763cb/stacks/test-stack/4908b2ab-2d49-43b9-b435-268b99587ed2 |
| | rel: self |
| | |
| deletion_time | None |
| notification_topics | [] |
| capabilities | [] |
| disable_rollback | True |
| timeout_mins | None |
| stack_owner | nfv_user |
| parent | None |
| stack_user_project_id | 1358762f92d34531804412e1f289996c |
| tags | [] |
| | |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+
$ openstack volume list

$ openstack volume type list
+--------------------------------------+-----------------+-----------+
| ID | Name | Is Public |
+--------------------------------------+-----------------+-----------+
| bd5c3bc5-92ef-4ae0-9109-c3457bdc9c60 | test-volumetype | True |
| afcc773f-109c-45f5-af11-776bb95c2493 | lvmdriver-1 | True |
| 0ac7b1cc-586e-412b-a30e-117f64e273dd | __DEFAULT__ | True |
+--------------------------------------+-----------------+-----------+

mysql> select display_name,deleted,id,user_id,project_id,status,volume_type_id from volumes;
+--------------------------------------------+---------+--------------------------------------+----------------------------------+----------------------------------+-----------+--------------------------------------+
| display_name | deleted | id | user_id | project_id | status | volume_type_id |
+--------------------------------------------+---------+--------------------------------------+----------------------------------+----------------------------------+-----------+--------------------------------------+
| image-64090ba9-d3b0-4bbe-8ec1-f30d90b7ab56 | 0 | 5768d777-39a0-4ef8-96b9-848b073bf857 | 867ea1a8313b47e09cf2c5904f89363e | 8e8baeadc93548c18df078a2c0ecb034 | available | bd5c3bc5-92ef-4ae0-9109-c3457bdc9c60 |
| test-volume | 1 | 6b4b1bc1-3f2f-4701-8dbf-12cdc08aa487 | 867ea1a8313b47e09cf2c5904f89363e | 16ca0ec90a274db5b2cadb22ff7763cb | deleted | bd5c3bc5-92ef-4ae0-9109-c3457bdc9c60 |
+--------------------------------------------+---------+--------------------------------------+----------------------------------+----------------------------------+-----------+--------------------------------------+
2 rows in set (0.00 sec)
mysql> select * from volume_types;
+---------------------+---------------------+------------+---------+--------------------------------------+-----------------+--------------+-----------+---------------------+
| created_at | updated_at | deleted_at | deleted | id | name | qos_specs_id | is_public | description |
+---------------------+---------------------+------------+---------+--------------------------------------+-----------------+--------------+-----------+---------------------+
| 2021-11-09 02:01:54 | 2021-11-09 02:01:54 | NULL | 0 | 0ac7b1cc-586e-412b-a30e-117f64e273dd | __DEFAULT__ | NULL | 1 | Default Volume Type |
| 2021-11-09 02:07:28 | NULL | NULL | 0 | afcc773f-109c-45f5-af11-776bb95c2493 | lvmdriver-1 | NULL | 1 | NULL |
| 2021-12-08 07:22:15 | NULL | NULL | 0 | bd5c3bc5-92ef-4ae0-9109-c3457bdc9c60 | test-volumetype | NULL | 1 | NULL |
+---------------------+---------------------+------------+---------+--------------------------------------+-----------------+--------------+-----------+---------------------+
3 rows in set (0.00 sec)
```

* stack-for-volume.yaml(HOT)
```
description: Cinder test

heat_template_version: '2015-10-15'

resources:
  VirtualStorage:
    type: OS::Cinder::Volume
    properties:
      name: test-volume
      image: cirros-0.5.2-x86_64-disk
      size: 8
      volume_type: { get_resource: multi }
  multi:
    type: OS::Cinder::VolumeType
    properties:
      name: test-volumetype
```

* cinder.conf
```
...
[DEFAULT]
cinder_internal_tenant_user_id = a33498d744fe4a60a951dc059889a679
cinder_internal_tenant_project_id = 8e8baeadc93548c18df078a2c0ecb034
...
[lvmdriver-1]
image_volume_cache_enabled = True
...
```

I also confirmed the following:
https://bugs.launchpad.net/cinder/+bug/1823880

I thought it might be possible to solve it by creating and using Volume Type for volume cache instead of using the original Volume Type when creating volume cache.
I'm not familiar with cinder, so I'm not sure if it really works...

Please let me know if there is any good solution.