There's still some small time window where two concurrent requests that try to consume the same last piece of resource can both succeed, and the total allocated resources are more than (total-reserved)*allocation_ratio set on this resource provider.
The following example is 100% reliably reproduced on current master in DevStack:
- create some custom resource class (say CUSTOM_CONCURRENCY_TEST)
- create a resource provider with inventory of 1 unit of that resource with allocation ratio 1.0
- make 2 concurrent requests trying to allocate one unit of this custom resource
- both requests succeed, and resource provider has 2 allocations against it even though its total capacity is 1.
+-------------------------+------------------+----------+----------+----------+-----------+-------+------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |
+-------------------------+------------------+----------+----------+----------+-----------+-------+------+
| CUSTOM_CONCURRENCY_TEST | 1.0 | 1 | 1 | 0 | 1 | 1 | 2 |
+-------------------------+------------------+----------+----------+----------+-----------+-------+------+
see full repro script at https://paste.openstack.org/show/btiWgpf6THddMoe7u1Gm/ (uses threads for concurrent requests).
While this is a synthetic test, we've seen actual examples in real deployments.
What I am getting so far, is that the following happens:
two requests are being processed in parallel,
first one succeeds, the second one correctly detects the concurrent update, goes into retry here in the 'replace_all' function
https:/
so calls _set_allocation
log snippet here https:/
added logging of 'records' returned by the query, all the three times it is called - 1 for request1, 2 for first attempt of request2, 3 for the second attempt of request2 - the result is the same, usage is not changed and is still None, thus the capacity check passes
could that be some query caching on mysql or sqla side?..