Fail to unify allocation_capacity_gb values among multiple Active-Active Cinder-Volume services

[zhaoleilc]

Description
===========
Three cinder-volume services was started in different hosts and
they all refer to the same rbd backend by setting same cinder backend
configuration. In order to achieve Active-Active configurations
the cluster configuration option is defined in the [DEFAULT] section
of cinder.conf for each cinder-volume service.

After these cinder-volume services was started, three records could be
seen in volume service list. However, just one record could be seen for
cinder get-pools conmmand.

When 10 volumes of 10GB of that backend was created successfully, three
different values of allocation_capacity_gb i.e. 30, 40 and 30 could be
inspected by executing cinder get-pools conmmand many times.

Therefore, each cinder-volume service has its own allocation_capacity_gb
and allocation_capacity_gb does not have Active-Active realization.

Steps to reproduce
==================
1. Spawn three cinder-volume services which all refer to the same rbd
backend and belong to one cluster whose name is clus1 e.g.

$ openstack volume service list
+------------------+-----------------------+-------------+---------+-------+----------------------------+
| Binary | Host | Zone | Status | State | Updated At |
+------------------+-----------------------+-------------+---------+-------+----------------------------+
| cinder-volume | host1@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:29:07.000000 |
| cinder-volume | host2@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:29:07.000000 |
| cinder-volume | host3@rbd-clus-1 | az-stackdev | enabled | up | 2022-10-30T11:27:51.000000

$ cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail
+-----------------------------+---------------------------------------------------------+
| Property | Value |
+-----------------------------+---------------------------------------------------------+
| allocated_capacity_gb | 0 |
| backend_state | up |
| driver_version | 1.2.0 |
| filter_function | None |
| free_capacity_gb | 5912.85 |
| goodness_function | None |
| location_info | ceph:/etc/ceph/ceph_a4842e0c-3207-4.....:cinder.volumes |
| max_over_subscription_ratio | 20 |
| multiattach | True |
| name | clus1@rbd-clus-1#rbd-clust-1 |
| replication_enabled | False |
| reserved_percentage | 0 |
| storage_protocol | ceph |
| thin_provisioning_support | True |
| timestamp | 2022-10-30T11:35:29.157787 |
| total_capacity_gb | 6310.3 |
| vendor_name | Open Source |
| volume_backend_name | rbd-clust-1 |
+-----------------------------+---------------------------------------------------------+

2. Create 10 volumes of 10GB of that cinder backend e.g.
$ for i in {1..10};do openstack volume create --type rbd-clust-1 --size 10 clus-v$i;done
$ openstack volume list
+--------------------------------------+----------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+----------+-----------+------+-------------+
| 20ac2305-6017-4f96-b05e-126b1dbe6b75 | clus-v10 | available | 10 | |
| a902574e-8e57-4a4c-ac9c-daaa335464bc | clus-v9 | available | 10 | |
| 5a3e85f3-fe09-45e0-980e-be6ab3279b22 | clus-v8 | available | 10 | |
| 1c0d720f-d407-4089-8c4f-574c74874c86 | clus-v7 | available | 10 | |
| d7c08ba6-88cf-409b-9bcf-a3a5901bd67e | clus-v6 | available | 10 | |
| 5d01544e-1c21-45d0-964f-1532f894b8a8 | clus-v5 | available | 10 | |
| b510820d-79ba-4f64-b8b6-f443b9d8ae11 | clus-v4 | available | 10 | |
| 139c1922-cb2f-4ca5-a594-196067cf649e | clus-v3 | available | 10 | |
| 888e3fc7-d8d2-47e5-be40-ce910e27713e | clus-v2 | available | 10 | |
| 82b29c4c-805d-4491-9d03-70183359ff85 | clus-v1 | available | 10 | |
+--------------------------------------+----------+-----------+------+-------------+

3. Inspect the allocated_capacity_gb value by executing cinder get-pools conmmand many times.

Expected result
===============
The allocated_capacity_gb value returns 100 no matter how many times cinder get-pools conmmand is executed.

Actual result
=============
Different allocated_capacity_gb values are returned and in aggregate they are accurate.

[stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail
+-----------------------------+-----------------------+
| Property | Value |
+-----------------------------+-----------------------+
| allocated_capacity_gb | 30 |
...
[stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail
+-----------------------------+-----------------------+
| Property | Value |
+-----------------------------+-----------------------+
| allocated_capacity_gb | 40 |
...
[stackdev-hci]root@hci-001:~# cinder get-pools --filters name=clus1@rbd-clus-1#rbd-clust-1 --detail
+-----------------------------+-----------------------+
| Property | Value |
+-----------------------------+-----------------------+
| allocated_capacity_gb | 30 |
...

Environment
===========
Victoria branch Of Openstack

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/863600

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/863769

[Sofia Enriquez (lsofia-enriquez)]

Hello,
Some backends like DELL EMC, Hitachi or IBM have a custom way of calculating pool stats. Do you mind sharing the backend you are using in the cluster?
Cheers,
Sofia

[zhaoleilc (zhaoleilc)]

Hi,
All backends I used in the cluster are rbd backends and they
refer to the same ceph cluster.
In addition, Although various backends may have a custom way
of calculating pool stats, the allocation_capacity_gb value is
calculated by cinder-volume service and it will override the
allocation_capacity_gb value in backends.

Cheers,
Zhaoleilc

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/863769
Reason: stable/victoria branch of openstack/cinder is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.