Exception raised when cleaning up resources if aggregate doesn't exist

[Sam Morrison]

Get the following when out of quota

Jul 27 11:37:39 workeru2-3 blazar-manager[4018848]: 2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server [req-464454f4-8d9f-4928-9a2d-c92729f91052 - cee95453198645238a40d9db5a294453 - - -] Exception during message handling: blazar>
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/blazar/plugins/instances/instance_plugin.py", line 439, in reserve_resource
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server flavor, group, pool = self._create_resources(instance_reservation)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/blazar/plugins/instances/instance_plugin.py", line 303, in _create_resources
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server reserved_group = user_client.nova.server_groups.create(
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/novaclient/api_versions.py", line 393, in substitution
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server return methods[-1].func(obj, *args, **kwargs)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/novaclient/v2/server_groups.py", line 101, in create
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server return self._create('/os-server-groups', body, 'server_group')
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/novaclient/base.py", line 363, in _create
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 392, in post
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/novaclient/client.py", line 78, in request
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body, url, method)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server novaclient.exceptions.Forbidden: Quota exceeded, too many server groups. (HTTP 403) (Request-ID: req-d9051871-d172-48b1-b419->
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/blazar/utils/service.py", line 81, in run_method
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server return method(**kwargs)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/blazar/manager/service.py", line 332, in create_lease
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server self._create_reservation(reservation)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/blazar/manager/service.py", line 612, in _create_reservation
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server resource_id = self.plugins[resource_type].reserve_resource(
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/blazar/plugins/instances/instance_plugin.py", line 443, in reserve_resource
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server self.cleanup_resources(instance_reservation)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/blazar/plugins/instances/instance_plugin.py", line 340, in cleanup_resources
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server check_and_delete_resource(nova.ReservationPool(), reservation_id)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/blazar/plugins/instances/instance_plugin.py", line 331, in check_and_delete_resource
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server client.delete(id)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/blazar/utils/openstack/nova.py", line 280, in delete
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server agg = self.get_aggregate_from_name_or_id(pool)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/blazar/utils/openstack/nova.py", line 234, in get_aggregate_from_name_or_id
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server raise manager_exceptions.AggregateNotFound(pool=aggregate_obj)
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server blazar.manager.exceptions.AggregateNotFound: Aggregate 'e2e73025-93c4-4809-b0c4-f9d2a19a53fa' not found!
                                                    2021-07-27 11:37:39.988 4018848 ERROR oslo_messaging.rpc.server

[Sam Morrison (sorrison)]

https://review.opendev.org/c/openstack/blazar/+/802652

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/blazar/+/802652
Committed: https://opendev.org/openstack/blazar/commit/339e91342d67d17b84b600d107a7f996564df3c6
Submitter: "Zuul (22348)"
Branch: master

commit 339e91342d67d17b84b600d107a7f996564df3c6
Author: Sam Morrison <email address hidden>
Date: Wed Jul 28 13:35:47 2021 +1000

    Handle AggregateNotFound when deleting aggregate

    If an error occurs during lease create and an aggregate isn't created,
    an exception will be raised in cleanup_resources when trying to delete
    the missing aggregate. Handle this gracefully and continue.

    Closes-Bug: #1938241

    Change-Id: I866bc7c369fb390f8c9204e36c06fadea17f6dd5

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/blazar 8.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/blazar/+/835055

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/blazar/+/835055
Committed: https://opendev.org/openstack/blazar/commit/6549ab0f9e6997fe28b51c6dffd03f171fda6647
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 6549ab0f9e6997fe28b51c6dffd03f171fda6647
Author: Sam Morrison <email address hidden>
Date: Wed Jul 28 13:35:47 2021 +1000

    Handle AggregateNotFound when deleting aggregate

    If an error occurs during lease create and an aggregate isn't created,
    an exception will be raised in cleanup_resources when trying to delete
    the missing aggregate. Handle this gracefully and continue.

    Closes-Bug: #1938241

    Change-Id: I866bc7c369fb390f8c9204e36c06fadea17f6dd5
    (cherry picked from commit 339e91342d67d17b84b600d107a7f996564df3c6)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/blazar 7.0.1 release.