[stable only] Handle quota usage during create/delete races
There is a race during an instance delete while booting where, if the
build request was found, there is still a chance we can fail to lookup
the instance record and if so, we need to go ahead and commit the quota
decrement reservations. We're currently skipping the quotas.commit() if
we *don't* find the instance record. I believe this is a regression as of
commit 361b88383130be8da9d474d02a9f62136239d506.
There are two reasons why we may fail to find the instance record after
we successfully found the build request:
a) Conductor didn't create the instance record yet.
b) Conductor deleted the instance record after finding the build
request was deleted by us (the API)
In either case, conductor doesn't do anything to decrement quota usage
so we need to do it in the compute/api.
There is a second race where we find the build request and succeed in
looking up the instance record but then fail to delete the instance
record (instance.destroy() raises InstanceNotFound). This can happen if
c) Conductor deleted the instance record after finding the build
request was deleted by us but after we looked up the instance
d) If we (the API) are racing with another delete request
If c) happens, we need to quotas.commit(). If d) happens, we need to
quotas.rollback(). Since we can't know which case it was when we get
InstanceNotFound, we'll do a quotas.rollback() to get rid of the
reservation record and force a refresh of quota usages to make sure
we end with correct usage in case rollback was not the right choice.
Reviewed: https:/ /review. openstack. org/582413 /git.openstack. org/cgit/ openstack/ nova/commit/ ?id=456b3d68bc4 91a6965316e6a4f 3fc11dd509237b
Committed: https:/
Submitter: Zuul
Branch: stable/ocata
commit 456b3d68bc491a6 965316e6a4f3fc1 1dd509237b
Author: melanie witt <email address hidden>
Date: Thu Jul 12 22:42:19 2018 +0000
[stable only] Handle quota usage during create/delete races
There is a race during an instance delete while booting where, if the da9d474d02a9f62 136239d506.
build request was found, there is still a chance we can fail to lookup
the instance record and if so, we need to go ahead and commit the quota
decrement reservations. We're currently skipping the quotas.commit() if
we *don't* find the instance record. I believe this is a regression as of
commit 361b88383130be8
There are two reasons why we may fail to find the instance record after
we successfully found the build request:
a) Conductor didn't create the instance record yet.
b) Conductor deleted the instance record after finding the build
request was deleted by us (the API)
In either case, conductor doesn't do anything to decrement quota usage
so we need to do it in the compute/api.
There is a second race where we find the build request and succeed in
looking up the instance record but then fail to delete the instance
record (instance.destroy() raises InstanceNotFound). This can happen if
c) Conductor deleted the instance record after finding the build
request was deleted by us but after we looked up the instance
d) If we (the API) are racing with another delete request
If c) happens, we need to quotas.commit(). If d) happens, we need to rollback( ). Since we can't know which case it was when we get Found, we'll do a quotas.rollback() to get rid of the
quotas.
InstanceNot
reservation record and force a refresh of quota usages to make sure
we end with correct usage in case rollback was not the right choice.
Closes-Bug: #1783613
Change-Id: I4d492dcad193ff 0b1202dcae97954 cc65b75c109