Placement allows to overconsume resources beyond configured allocaiton ratio

Bug #2039560 reported by Pavlo Shchelokovskyy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
placement
In Progress
Undecided
Unassigned

Bug Description

There's still some small time window where two concurrent requests that try to consume the same last piece of resource can both succeed, and the total allocated resources are more than (total-reserved)*allocation_ratio set on this resource provider.

The following example is 100% reliably reproduced on current master in DevStack:

- create some custom resource class (say CUSTOM_CONCURRENCY_TEST)
- create a resource provider with inventory of 1 unit of that resource with allocation ratio 1.0
- make 2 concurrent requests trying to allocate one unit of this custom resource
- both requests succeed, and resource provider has 2 allocations against it even though its total capacity is 1.
+-------------------------+------------------+----------+----------+----------+-----------+-------+------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |
+-------------------------+------------------+----------+----------+----------+-----------+-------+------+
| CUSTOM_CONCURRENCY_TEST | 1.0 | 1 | 1 | 0 | 1 | 1 | 2 |
+-------------------------+------------------+----------+----------+----------+-----------+-------+------+

see full repro script at https://paste.openstack.org/show/btiWgpf6THddMoe7u1Gm/ (uses threads for concurrent requests).

While this is a synthetic test, we've seen actual examples in real deployments.

Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

What I am getting so far, is that the following happens:

two requests are being processed in parallel,
first one succeeds, the second one correctly detects the concurrent update, goes into retry here in the 'replace_all' function
https://opendev.org/openstack/placement/src/branch/master/placement/objects/allocation.py#L507

so calls _set_allocations(...) again, which in turn again calls the _check_capacity_exceeded(...), and there, the main big DB query which it does still does not see the new usages that should be already there, usages are still None as they were in the first attempt.

log snippet here https://paste.openstack.org/show/bhSOAeaeOPi7uourSvyW/
added logging of 'records' returned by the query, all the three times it is called - 1 for request1, 2 for first attempt of request2, 3 for the second attempt of request2 - the result is the same, usage is not changed and is still None, thus the capacity check passes

could that be some query caching on mysql or sqla side?..

Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

all in all, it looks like the same old identity map and transaction isolation level of 'repeatable reads'.
Since retry is happening in the same session/transaction, it simply can not read new data during retry, making the has capacity check useless.

Solution would be to re-write code to make retries happen above the place they are currently done, so that failed due to concurrent update session/transaction is closed, and new session/transaction is used during retry.

Will push a patch shortly.

Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

alternatively, we should make that query that checks capacity in a separate session, similar to how it is already done in replace_all to refresh resource providers on retry: https://opendev.org/openstack/placement/src/branch/master/placement/objects/allocation.py#L514-L524

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to placement (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/placement/+/898807

Changed in placement:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/placement/+/898808

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.