RFE: Enforcer should optionally handle resource cleanup

[Lance Bragstad]

The limit.Enforcer() object is currently implemented as a context manager, where the operation creating the resources in the request are handled within that context. For example:

  with limit.Enforcer(claim, callback=callback):
      self._create_foobar()

This pattern was chosen to make things easier for developers to verify resource usage before and after the creation of the thing in the request, thus preventing race conditions. The typical flow for not verifying usage is the following:

  1.) project has a limit of 10 instances and usage of 9 instances
  2.) client A and client B both request a single instance at the same time
  3.) both verification checks pass upon entering the context manager and perform the initial check
  4.) client B's request is process quicker than client A's, but both instances are created
  5.) the project is over limit since usage is 11

Conversely, if checking verification after resource creation, the flow would be:

  1.) project has a limit of 10 instances and usage of 9 instances
  2.) client A and client B both request a single instance at the same time
  3.) both verification checks pass upon entering the context manager and perform the initial check
  4.) both instances are created
  5.) client B's request exits the context manager first and perform the verification that the usage <= limit
  6.) client A's requests exits the context manager second and is aware that the limit already met
  7.) the instance created by client A is deleted and a 403 is returned

The second case adds additional protection for making sure limits are enforced despite race conditions.

The current implementation of limit.Enforcer is missing step #7 from the second flow. The Enforcer object needs to be aware of what to do to clean up resources and which resources to clean up. Note that resource cleanup should only be necessary if verify=True.

An example of what this might look like as someone using oslo.limit would be [0]. This was discussed when walking through limit usage with nova [1]

[0] https://gist.github.com/lbragstad/69d28dca8adfa689c00b272d6db8bde7#file-cleanup-in-library-py-L10-L13
[1] http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2018-09-05.log.html