Activity log for bug #1995204

Date Who What changed Old value New value Message
2022-10-30 11:58:05 zhaoleilc bug added bug
2022-10-30 11:59:05 zhaoleilc description Description =========== Three cinder-volume services was started in different hosts and they all refer to the same rbd backend by setting same cinder backend configuration. In order to achieve Active-Active configurations the cluster configuration option is defined in the [DEFAULT] section of cinder.conf for each cinder-volume service. After these cinder-volume services was started, three records could be seen in volume service list. However, just one record could be seen for cinder get-pools conmmand. When 10 volumes of 10GB of that backend was created successfully, three different values of allocation_capacity_gb i.e. 30, 40 and 30 could be inspected by executing cinder get-pools conmmand many times. Therefore, each cinder-volume service has its own allocation_capacity_gb and allocation_capacity_gb does not have Active-Active realization. Steps to reproduce ================== 1. Spawn three cinder-volume services which all refer to the same rbd backend and belong to one cluster whose name is clus1 e.g. $ openstack volume service list +------------------+-----------------------+-------------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+-----------------------+-------------+---------+-------+----------------------------+ | cinder-volume | host1@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:29:07.000000 | | cinder-volume | host2@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:29:07.000000 | | cinder-volume | host3@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:27:51.000000 $ cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail +-----------------------------+---------------------------------------------------------+ | Property | Value | +-----------------------------+---------------------------------------------------------+ | allocated_capacity_gb | 0 | | backend_state | up | | driver_version | 1.2.0 | | filter_function | None | | free_capacity_gb | 5912.85 | | goodness_function | None | | location_info | ceph:/etc/ceph/ceph_a4842e0c-3207-4.....:cinder.volumes | | max_over_subscription_ratio | 20 | | multiattach | True | | name | clus1@rbd-clus-1#rbd-clust-1 | | replication_enabled | False | | reserved_percentage | 0 | | storage_protocol | ceph | | thin_provisioning_support | True | | timestamp | 2022-10-30T11:35:29.157787 | | total_capacity_gb | 6310.3 | | vendor_name | Open Source | | volume_backend_name | rbd-clust-1 | +-----------------------------+---------------------------------------------------------+ 2. Create 10 volumes of 10GB of that cinder backend e.g. $ for i in {1..10};do openstack volume create --type rbd-clust-1 --size 10 clus-v$i;done $ openstack volume list +--------------------------------------+----------+-----------+------+-------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------+-----------+------+-------------+ | 20ac2305-6017-4f96-b05e-126b1dbe6b75 | clus-v10 | available | 10 | | | a902574e-8e57-4a4c-ac9c-daaa335464bc | clus-v9 | available | 10 | | | 5a3e85f3-fe09-45e0-980e-be6ab3279b22 | clus-v8 | available | 10 | | | 1c0d720f-d407-4089-8c4f-574c74874c86 | clus-v7 | available | 10 | | | d7c08ba6-88cf-409b-9bcf-a3a5901bd67e | clus-v6 | available | 10 | | | 5d01544e-1c21-45d0-964f-1532f894b8a8 | clus-v5 | available | 10 | | | b510820d-79ba-4f64-b8b6-f443b9d8ae11 | clus-v4 | available | 10 | | | 139c1922-cb2f-4ca5-a594-196067cf649e | clus-v3 | available | 10 | | | 888e3fc7-d8d2-47e5-be40-ce910e27713e | clus-v2 | available | 10 | | | 82b29c4c-805d-4491-9d03-70183359ff85 | clus-v1 | available | 10 | | +--------------------------------------+----------+-----------+------+-------------+ 3. Inspect the allocated_capacity_gb value by executing cinder get-pools conmmand many times. Expected result =============== The allocated_capacity_gb value returns 100 no matter how many times cinder get-pools conmmand is executed. Actual result ============= Different allocated_capacity_gb values are returned and in aggregate they are accurate. [stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail +-----------------------------+-----------------------+ | Property | Value | +-----------------------------+-----------------------+ | allocated_capacity_gb | 30 | ... [stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail +-----------------------------+-----------------------+ | Property | Value | +-----------------------------+-----------------------+ | allocated_capacity_gb | 40 | ... [stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail +-----------------------------+-----------------------+ | Property | Value | +-----------------------------+-----------------------+ | allocated_capacity_gb | 30 | ... Environment =========== Victoria branch Of Openstack Description =========== Three cinder-volume services was started in different hosts and they all refer to the same rbd backend by setting same cinder backend configuration. In order to achieve Active-Active configurations the cluster configuration option is defined in the [DEFAULT] section of cinder.conf for each cinder-volume service. After these cinder-volume services was started, three records could be seen in volume service list. However, just one record could be seen for cinder get-pools conmmand. When 10 volumes of 10GB of that backend was created successfully, three different values of allocation_capacity_gb i.e. 30, 40 and 30 could be inspected by executing cinder get-pools conmmand many times. Therefore, each cinder-volume service has its own allocation_capacity_gb and allocation_capacity_gb does not have Active-Active realization. Steps to reproduce ================== 1. Spawn three cinder-volume services which all refer to the same rbd backend and belong to one cluster whose name is clus1 e.g. $ openstack volume service list +------------------+-----------------------+-------------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+-----------------------+-------------+---------+-------+----------------------------+ | cinder-volume | host1@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:29:07.000000 | | cinder-volume | host2@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:29:07.000000 | | cinder-volume | host3@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:27:51.000000 $ cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail +-----------------------------+---------------------------------------------------------+ | Property | Value | +-----------------------------+---------------------------------------------------------+ | allocated_capacity_gb | 0 | | backend_state | up | | driver_version | 1.2.0 | | filter_function | None | | free_capacity_gb | 5912.85 | | goodness_function | None | | location_info | ceph:/etc/ceph/ceph_a4842e0c-3207-4.....:cinder.volumes | | max_over_subscription_ratio | 20 | | multiattach | True | | name | clus1@rbd-clus-1#rbd-clust-1 | | replication_enabled | False | | reserved_percentage | 0 | | storage_protocol | ceph | | thin_provisioning_support | True | | timestamp | 2022-10-30T11:35:29.157787 | | total_capacity_gb | 6310.3 | | vendor_name | Open Source | | volume_backend_name | rbd-clust-1 | +-----------------------------+---------------------------------------------------------+ 2. Create 10 volumes of 10GB of that cinder backend e.g. $ for i in {1..10};do openstack volume create --type rbd-clust-1 --size 10 clus-v$i;done $ openstack volume list +--------------------------------------+----------+-----------+------+-------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------+-----------+------+-------------+ | 20ac2305-6017-4f96-b05e-126b1dbe6b75 | clus-v10 | available | 10 | | | a902574e-8e57-4a4c-ac9c-daaa335464bc | clus-v9 | available | 10 | | | 5a3e85f3-fe09-45e0-980e-be6ab3279b22 | clus-v8 | available | 10 | | | 1c0d720f-d407-4089-8c4f-574c74874c86 | clus-v7 | available | 10 | | | d7c08ba6-88cf-409b-9bcf-a3a5901bd67e | clus-v6 | available | 10 | | | 5d01544e-1c21-45d0-964f-1532f894b8a8 | clus-v5 | available | 10 | | | b510820d-79ba-4f64-b8b6-f443b9d8ae11 | clus-v4 | available | 10 | | | 139c1922-cb2f-4ca5-a594-196067cf649e | clus-v3 | available | 10 | | | 888e3fc7-d8d2-47e5-be40-ce910e27713e | clus-v2 | available | 10 | | | 82b29c4c-805d-4491-9d03-70183359ff85 | clus-v1 | available | 10 | | +--------------------------------------+----------+-----------+------+-------------+ 3. Inspect the allocated_capacity_gb value by executing cinder get-pools conmmand many times. Expected result =============== The allocated_capacity_gb value returns 100 no matter how many times cinder get-pools conmmand is executed. Actual result ============= Different allocated_capacity_gb values are returned and in aggregate they are accurate. [stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail +-----------------------------+-----------------------+ | Property | Value | +-----------------------------+-----------------------+ | allocated_capacity_gb | 30 | ... [stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail +-----------------------------+-----------------------+ | Property | Value | +-----------------------------+-----------------------+ | allocated_capacity_gb | 40 | ... [stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail +-----------------------------+-----------------------+ | Property | Value | +-----------------------------+-----------------------+ | allocated_capacity_gb | 30 | ... Environment =========== Victoria branch Of Openstack
2022-11-04 09:12:05 OpenStack Infra cinder: status New In Progress
2022-11-09 10:31:21 Sofia Enriquez nominated for series cinder/victoria
2022-11-09 10:31:21 Sofia Enriquez bug task added cinder/victoria
2022-11-09 10:53:18 Sofia Enriquez cinder: importance Undecided Low
2022-11-09 10:53:21 Sofia Enriquez cinder/victoria: importance Undecided Low
2022-11-09 10:54:31 Sofia Enriquez tags capacity get-pools
2022-11-09 11:25:36 Sofia Enriquez tags capacity get-pools active-active capacity