[victoria] max_over_subscription_ratio set auto mode rbd backend will have a negative value

[weisongf]

opesntack verison:V

cinder get-pools --detail
+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+-----------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| allocated_capacity_gb | -6 |
| backend_state | up |
| driver_version | 1.2.0 |
| filter_function | None |
| free_capacity_gb | 13356.85 |
| goodness_function | None |
| location_info | ceph:/etc/ceph/ceph_6e994228-5211-472e-a2f1-0bcb42cc9455.conf:6e994228-5211-472e-a2f1-0bcb42cc9455:cinder_6e994228-5211-472e-a2f1-0bcb42cc9455:cinder.volumes |
| max_over_subscription_ratio | auto |
| multiattach | True |
| name | rbd:volumes@rbd-sas-1#rbd-sas-1 |
| replication_enabled | False |
| reserved_percentage | 0 |
| storage_protocol | ceph |
| thin_provisioning_support | True |
| timestamp | 2021-08-04T03:50:59.139241 |
| total_capacity_gb | 13357.03 |
| vendor_name | Open Source |
| volume_backend_name | rbd-sas-1 |
+-----------------------------+-------------------------------

2021-08-04 02:47:35.099 1 INFO cinder.utils [None req-0c5f86f2-6ef6-48a9-9406-4c241ab615b7 58f9c640ef7e4133a97f3e50f2b46ca9 fd135d512f024dcb8903438ba6784b08 - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is -4.08474576271061
2021-08-04 02:53:34.565 1 INFO cinder.utils [None req-31ea4180-02d4-4196-81fa-9135bb8ab1a4 58f9c640ef7e4133a97f3e50f2b46ca9 fd135d512f024dcb8903438ba6784b08 - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is -4.128205128204809
2021-08-04 02:54:54.393 1 INFO cinder.utils [None req-59ac9940-9a48-4d3b-b42d-f264d4b8319a e381084f1a28456fb2ced16610b0572c 8cb235dcd5c14e6499162c78a7c82adc - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is -4.128205128204809
2021-08-04 02:54:54.395 1 WARNING cinder.scheduler.filters.capacity_filter [None req-59ac9940-9a48-4d3b-b42d-f264d4b8319a e381084f1a28456fb2ced16610b0572c 8cb235dcd5c14e6499162c78a7c82adc - - -] Filtering out host rbd:volumes@rbd-sas-1#rbd-sas-1 with an invalid maximum over subscription ratio of -4.13. The ratio should be a minimum of 1.0.
2021-08-04 02:54:54.396 1 INFO cinder.scheduler.base_filter [None req-59ac9940-9a48-4d3b-b42d-f264d4b8319a e381084f1a28456fb2ced16610b0572c 8cb235dcd5c14e6499162c78a7c82adc - - -] Filtering removed all hosts for the request with volume ID '0e6100d0-54b4-4cb4-92b2-46fb00f60f1e'. Filter results: AvailabilityZoneFilter: (start: 1, end: 1), CapacityFilter: (start: 1, end: 0), CapabilitiesFilter: (start: 0, end: 0), InstanceLocalityFilter: (start: 0, end: 0)
2021-08-04 02:54:54.397 1 WARNING cinder.scheduler.filter_scheduler [None req-59ac9940-9a48-4d3b-b42d-f264d4b8319a e381084f1a28456fb2ced16610b0572c 8cb235dcd5c14e6499162c78a7c82adc - - -] No weighed backend found for volume with properties: {'id': 'ee8a8cce-e602-454a-8962-bbb7ac5a63a8', 'name': 'EBS_ft_std', 'description': None, 'is_public': True, 'projects': [], 'extra_specs': {'volume_backend_name': 'rbd-sas-1'}, 'qos_specs_id': 'e5213293-b774-4ad2-9e2f-4c5427317845', 'created_at': '2021-08-03T11:20:23.000000', 'updated_at': '2021-08-03T11:20:23.000000', 'deleted_at': None, 'deleted': False}
2021-08-04 02:54:54.398 1 INFO cinder.message.api [None req-59ac9940-9a48-4d3b-b42d-f264d4b8319a e381084f1a28456fb2ced16610b0572c 8cb235dcd5c14e6499162c78a7c82adc - - -] Creating message record for request_id = req-59ac9940-9a48-4d3b-b42d-f264d4b8319a
2021-08-04 02:54:54.438 1 ERROR cinder.scheduler.flows.create_volume [None req-59ac9940-9a48-4d3b-b42d-f264d4b8319a e381084f1a28456fb2ced16610b0572c 8cb235dcd5c14e6499162c78a7c82adc - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: cinder.exception.NoValidBackend: No valid backend was found. No weighed backends available
2021-08-04 03:02:35.091 1 INFO cinder.utils [None req-f3be6c8c-4784-4410-a2e2-1e66bcf188d7 58f9c640ef7e4133a97f3e50f2b46ca9 fd135d512f024dcb8903438ba6784b08 - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is -4.128205128204809
2021-08-04 03:05:34.490 1 INFO cinder.utils [None req-0d29db30-c7eb-4b1b-8b31-5987040d5ea0 58f9c640ef7e4133a97f3e50f2b46ca9 fd135d512f024dcb8903438ba6784b08 - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is -4.128205128204809
2021-08-04 03:20:35.260 1 INFO cinder.utils [None req-507d9057-a114-412e-91c9-8447cd2d1113 58f9c640ef7e4133a97f3e50f2b46ca9 fd135d512f024dcb8903438ba6784b08 - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is -4.08474576271061
2021-08-04 03:23:35.080 1 INFO cinder.utils [None req-97b1f4b2-ece4-4980-9de1-68804753ed58 58f9c640ef7e4133a97f3e50f2b46ca9 fd135d512f024dcb8903438ba6784b08 - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is -4.08474576271061
2021-08-04 03:32:34.077 1 INFO cinder.utils [None req-f9c8aec9-10c3-4405-a464-5651a3529b15 58f9c640ef7e4133a97f3e50f2b46ca9 fd135d512f024dcb8903438ba6784b08 - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is -4.128205128204809

Return to normal after restarting the cinder-volume service.

2021-08-04 03:51:46.274 1 INFO cinder.utils [None req-a5cd85d9-daea-4d73-b041-688433c44eb0 e381084f1a28456fb2ced16610b0572c 8cb235dcd5c14e6499162c78a7c82adc - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is -4.08474576271061
2021-08-04 03:53:34.897 1 INFO cinder.utils [None req-82cd1453-8aaa-4d08-abe8-ac11901f786e 58f9c640ef7e4133a97f3e50f2b46ca9 fd135d512f024dcb8903438ba6784b08 - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is 85.74576271184351
2021-08-04 03:54:48.736 1 INFO cinder.utils [None req-542bfcc8-ab7f-425a-9475-2ddd2f10f80c e381084f1a28456fb2ced16610b0572c 8cb235dcd5c14e6499162c78a7c82adc - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is -4.128205128204809
2021-08-04 03:56:47.582 1 INFO cinder.utils [None req-fa22ecb5-ee08-440e-94ea-f8ccd6b31b76 e381084f1a28456fb2ced16610b0572c 8cb235dcd5c14e6499162c78a7c82adc - - -] Auto max_over_subscription_ratio for pool rbd-sas-1 is 86.59322033896194

[Sofia Enriquez (lsofia-enriquez)]

Greetings weisongf,
just to add more details to the case, what are the consequences you are facing because of this?

As the reporter is using the Victoria release I think It would be nice to reproduce this in master and check if it's a current bug.

- max_over_subscription_ratio = 20.0 (string value)

Representation of the over subscription ratio when thin provisioning is enabled. Default ratio is 20.0, meaning provisioned capacity can be 20 times of the total physical capacity. If the ratio is 10.5, it means provisioned capacity can be 10.5 times of the total physical capacity. A ratio of 1.0 means provisioned capacity cannot exceed the total physical capacity. If the ratio is auto, Cinder will automatically calculate the ratio based on the provisioned capacity and the used space. If not set to auto, the ratio has to be a minimum of 1.0.

Cheers
Sofia

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/configuration_reference/cinder

[weisongf (songwei-8)]

If max_over_subscription_ratio is negative, cinder-Scheduler cannot select a backend and volumes cannot be created.

 Filter results: AvailabilityZoneFilter: (start: 1, end: 1), CapacityFilter: (start: 1, end: 0), CapabilitiesFilter: (start: 0, end: 0), InstanceLocalityFilter: (start: 0, end: 0)

[Sofia Enriquez (lsofia-enriquez)]

Greetings,
I'm not able to reproduce this in master. I set max_over_subscription_ratio = auto and create several volumes but all of them reach the available state.

Do you mind sharing the steps you are doing to facilitate this? -were the volumes in-use or cloned volumes or volumes created from snapshot-

Thanks in advance,
Sofia

[Brian Rosmaita (brian-rosmaita)]

If it turns out that this needs to be worked on, Gorka wrote a detailed spec about how the calculations are supposed to be done in cinder. Worth reading because he points out how the Cinder definition of some of the terms are different from they way they're sometimes used in industry:

https://specs.openstack.org/openstack/cinder-specs/specs/queens/provisioning-improvements.html