When Blazar fails to remove a host from an aggregate, it immediately stops executing the end_lease vent, for example:
ERROR blazar.manager.service [-] Error occurred while handling event. ERROR blazar.manager.service Traceback (most recent call last): ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/blazar/manager/service.py", line 242, in _exec_event ERROR blazar.manager.service event_fn(lease_id=event['lease_id'], event_id=event['id']) ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/blazar/status.py", line 225, in wrapper ERROR blazar.manager.service {'status': cls.ERROR}) ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR blazar.manager.service self.force_reraise() ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR blazar.manager.service six.reraise(self.type_, self.value, self.tb) ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/blazar/status.py", line 209, in wrapper ERROR blazar.manager.service result = func(*args, **kwargs) ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/blazar/manager/service.py", line 623, in end_lease ERROR blazar.manager.service status.reservation.DELETED) ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/blazar/manager/service.py", line 644, in _basic_action ERROR blazar.manager.service reservation['resource_id'] ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/blazar/plugins/oshosts/host_plugin.py", line 282, in on_end ERROR blazar.manager.service pool.delete(host_reservation['aggregate_id']) ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/blazar/utils/openstack/nova.py", line 304, in delete ERROR blazar.manager.service self.nova.aggregates.remove_host(agg.id, host) ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/novaclient/v2/aggregates.py", line 90, in remove_host ERROR blazar.manager.service body, "aggregate") ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 361, in _create ERROR blazar.manager.service resp, body = self.api.client.post(url, body=body) ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 223, in post ERROR blazar.manager.service return self.request(url, 'POST', **kwargs) ERROR blazar.manager.service File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 80, in request ERROR blazar.manager.service raise exceptions.from_response(resp, body, url, method) ERROR blazar.manager.service NotFound: Cannot remove host e5fcdcf3-8355-4705-9a0b-371e75ea1732 in aggregate 9657 (HTTP 404) (Request-ID: req-401c46d4-9159-4880-a4ca-0d8d898c924e)
This causes issues with future leases using the same hosts, as they are still in the uncleaned aggregate instead of the freepool.
Blazar should continue removing hosts from the aggregate even when it gets a failure, while logging a warning that not all hosts have been removed.
When Blazar fails to remove a host from an aggregate, it immediately stops executing the end_lease vent, for example:
ERROR blazar. manager. service [-] Error occurred while handling event. manager. service Traceback (most recent call last): manager. service File "/usr/lib/ python2. 7/site- packages/ blazar/ manager/ service. py", line 242, in _exec_event manager. service event_fn( lease_id= event[' lease_id' ], event_id= event[' id']) manager. service File "/usr/lib/ python2. 7/site- packages/ blazar/ status. py", line 225, in wrapper manager. service {'status': cls.ERROR}) manager. service File "/usr/lib/ python2. 7/site- packages/ oslo_utils/ excutils. py", line 220, in __exit__ manager. service self.force_ reraise( ) manager. service File "/usr/lib/ python2. 7/site- packages/ oslo_utils/ excutils. py", line 196, in force_reraise manager. service six.reraise( self.type_ , self.value, self.tb) manager. service File "/usr/lib/ python2. 7/site- packages/ blazar/ status. py", line 209, in wrapper manager. service result = func(*args, **kwargs) manager. service File "/usr/lib/ python2. 7/site- packages/ blazar/ manager/ service. py", line 623, in end_lease manager. service status. reservation. DELETED) manager. service File "/usr/lib/ python2. 7/site- packages/ blazar/ manager/ service. py", line 644, in _basic_action manager. service reservation[ 'resource_ id'] manager. service File "/usr/lib/ python2. 7/site- packages/ blazar/ plugins/ oshosts/ host_plugin. py", line 282, in on_end manager. service pool.delete( host_reservatio n['aggregate_ id']) manager. service File "/usr/lib/ python2. 7/site- packages/ blazar/ utils/openstack /nova.py" , line 304, in delete manager. service self.nova. aggregates. remove_ host(agg. id, host) manager. service File "/usr/lib/ python2. 7/site- packages/ novaclient/ v2/aggregates. py", line 90, in remove_host manager. service body, "aggregate") manager. service File "/usr/lib/ python2. 7/site- packages/ novaclient/ base.py" , line 361, in _create manager. service resp, body = self.api. client. post(url, body=body) manager. service File "/usr/lib/ python2. 7/site- packages/ keystoneauth1/ adapter. py", line 223, in post manager. service return self.request(url, 'POST', **kwargs) manager. service File "/usr/lib/ python2. 7/site- packages/ novaclient/ client. py", line 80, in request manager. service raise exceptions. from_response( resp, body, url, method) manager. service NotFound: Cannot remove host e5fcdcf3- 8355-4705- 9a0b-371e75ea17 32 in aggregate 9657 (HTTP 404) (Request-ID: req-401c46d4- 9159-4880- a4ca-0d8d898c92 4e)
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
ERROR blazar.
This causes issues with future leases using the same hosts, as they are still in the uncleaned aggregate instead of the freepool.
Blazar should continue removing hosts from the aggregate even when it gets a failure, while logging a warning that not all hosts have been removed.