Removing a hypervisor doesn't delete it entirely

[Xav Paice]

When removing a host (because it got rebuilt, for example), we use:

openstack compute service delete <someid>

I expected that to remove the hostname cleanly from the database (or at least mark it as deleted) so that the hostname can be re-used. This isn't the case, the host remained in the nova_api database in the resource_providers table, and therefore could not be re-used.

Starting nova-compute on the host in this state resulted in:

2019-03-18 22:48:26.023 62597 ERROR nova.scheduler.client.report [req-445f587d-74e5-4a96-a5b5-4717f9095fb6 - - - - -] [req-1f1e781e-2ed8-4f6a-ac9e-93ecc450cec5] Failed to create resource provider record in placement API for UUID c6c4d923-1d0c-4f12-8505-d5af60c28ade. Got 409: {"errors": [{"status": 409, "request_id": "req-1f1e781e-2ed8-4f6a-ac9e-93ecc450cec5", "detail": "There was a conflict when trying to complete your request.\n\n Conflicting resource provider name: host22.maas already exists. ", "title": "Conflict"}]}.
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager [req-445f587d-74e5-4a96-a5b5-4717f9095fb6 - - - - -] Error updating resources for node host22.maas.: ResourceProviderCreationFailed: Failed to create resource provider host22.maas
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager Traceback (most recent call last):
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 7345, in update_available_resource_for_node
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 689, in update_available_resource
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager self._update_available_resource(context, resources)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 277, in inner
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager return f(*args, **kwargs)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 713, in _update_available_resource
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager self._init_compute_node(context, resources)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 572, in _init_compute_node
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager self._update(context, cn)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 887, in _update
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager inv_data,
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 68, in set_inventory_for_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid,
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 1104, in set_inventory_for_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 665, in _ensure_resource_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 64, in wrapper
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager return f(self, *a, **k)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/report.py", line 612, in _create_resource_provider
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager raise exception.ResourceProviderCreationFailed(name=name)
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager ResourceProviderCreationFailed: Failed to create resource provider host22.maas
2019-03-18 22:48:26.024 62597 ERROR nova.compute.manager

I was unable to clear the database entry:
mysql> delete from resource_providers where name='host22.maas';
ERROR 1451 (23000): Cannot delete or update a parent row: a foreign key constraint fails (`nova_api`.`resource_providers`, CONSTRAINT `resource_providers_ibfk_1` FOREIGN KEY (`root_provider_id`) REFERENCES `resource_providers` (`id`))

I did, in the end, rename the host info in the table which allowed me to continue:
update resource_providers set name='host22.maas.deleted' where id='94';

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Midolesov (martinmidolesov)]

Same issue here. Are they any patches or currently developing some?