ResourceProviderUpdateFailed when total available resource is 0

Bug #1901120 reported by melanie witt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
melanie witt
Ussuri
Medium
melanie witt
Victoria
Medium
Lars Erik Pedersen

Bug Description

This got reported downstream [1] where if a compute node has no available (zero) inventory for a resource class, the libvirt driver attempts to PUT the 'total' inventory of the resource as 0 and the placement API does not allow it. To represent a 'total' of 0 in placement for a resource, one must omit that resource from the PUT request instead of trying to set it to zero.

The following traceback is emitted when an attempt to update 'total' inventory to 0 occurs:

2020-10-20 11:32:45.484 7 ERROR nova.compute.manager nova.exception.ResourceProviderUpdateFailed: Failed to update resource provider via URL /resource_providers/5812a7ab-f8e7-4c57-aba8-f283fedfd04b/inventories: {"errors": [{"status": 400, "title": "Bad Request", "detail": "The server could not comply with the request since it is either malformed or otherwise incorrect.\n\n JSON does not validate: 0 is less than the minimum of 1 Failed validating 'minimum' in schema['properties']['inventories']['patternProperties']['^[A-Z0-9_]+$']['properties']['total']: {'maximum': 2147483647, 'minimum': 1, 'type': 'integer'} On instance['inventories']['DISK_GB']['total']: 0 ", "code": "placement.undefined_code", "request_id": "req-7fc88053-8e43-4c18-9810-73ad05c3a993"}]}
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager Traceback (most recent call last):
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8740, in _update_available_resource_for_node
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager startup=startup)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 887, in update_available_resource
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager return f(*args, **kwargs)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 972, in _update_available_resource
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager self._update(context, cn, startup=startup)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1237, in _update
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager raise value
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1215, in _update_to_placement
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager allocations=allocs)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 1421, in update_from_provider_tree
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager self.set_traits_for_provider(context, pd.uuid, pd.traits)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager self.gen.throw(type, value, traceback)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 1341, in catch_all
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager raise exception.ResourceProviderSyncFailed()
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager nova.exception.ResourceProviderSyncFailed: Failed to synchronize the placement service with resource provider information supplied by the compute host.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1889723

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/759348

Changed in nova:
status: New → In Progress
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

The fix merged to master: https://review.opendev.org/c/openstack/nova/+/759348
stable/victoria backport has been proposed: https://review.opendev.org/c/openstack/nova/+/766177

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
melanie witt (melwitt) wrote :

Patch is at https://review.opendev.org/c/openstack/nova/+/766825 for review after stable/victoria change merges.

Revision history for this message
Lars Erik Pedersen (pedersen-larserik) wrote :

Any progress on this? :-)

Revision history for this message
melanie witt (melwitt) wrote :
Revision history for this message
melanie witt (melwitt) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 22.1.0

This issue was fixed in the openstack/nova 22.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 21.1.2

This issue was fixed in the openstack/nova 21.1.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 23.0.0.0rc1

This issue was fixed in the openstack/nova 23.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers