ResourceProviderUpdateFailed when total available resource is 0

Bug #1901120 reported by melanie witt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
melanie witt
Ussuri
Fix Released
Medium
melanie witt
Victoria
Fix Released
Medium
Lars Erik Pedersen

Bug Description

This got reported downstream [1] where if a compute node has no available (zero) inventory for a resource class, the libvirt driver attempts to PUT the 'total' inventory of the resource as 0 and the placement API does not allow it. To represent a 'total' of 0 in placement for a resource, one must omit that resource from the PUT request instead of trying to set it to zero.

The following traceback is emitted when an attempt to update 'total' inventory to 0 occurs:

2020-10-20 11:32:45.484 7 ERROR nova.compute.manager nova.exception.ResourceProviderUpdateFailed: Failed to update resource provider via URL /resource_providers/5812a7ab-f8e7-4c57-aba8-f283fedfd04b/inventories: {"errors": [{"status": 400, "title": "Bad Request", "detail": "The server could not comply with the request since it is either malformed or otherwise incorrect.\n\n JSON does not validate: 0 is less than the minimum of 1 Failed validating 'minimum' in schema['properties']['inventories']['patternProperties']['^[A-Z0-9_]+$']['properties']['total']: {'maximum': 2147483647, 'minimum': 1, 'type': 'integer'} On instance['inventories']['DISK_GB']['total']: 0 ", "code": "placement.undefined_code", "request_id": "req-7fc88053-8e43-4c18-9810-73ad05c3a993"}]}
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager During handling of the above exception, another exception occurred:
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager Traceback (most recent call last):
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 8740, in _update_available_resource_for_node
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager startup=startup)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 887, in update_available_resource
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager return f(*args, **kwargs)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 972, in _update_available_resource
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager self._update(context, cn, startup=startup)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1237, in _update
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager return attempt.get(self._wrap_exception)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2])
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager raise value
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1215, in _update_to_placement
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager allocations=allocs)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 1421, in update_from_provider_tree
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager self.set_traits_for_provider(context, pd.uuid, pd.traits)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager self.gen.throw(type, value, traceback)
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 1341, in catch_all
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager raise exception.ResourceProviderSyncFailed()
2020-10-20 11:32:45.484 7 ERROR nova.compute.manager nova.exception.ResourceProviderSyncFailed: Failed to synchronize the placement service with resource provider information supplied by the compute host.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1889723

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/759348

Changed in nova:
status: New → In Progress
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

The fix merged to master: https://review.opendev.org/c/openstack/nova/+/759348
stable/victoria backport has been proposed: https://review.opendev.org/c/openstack/nova/+/766177

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
melanie witt (melwitt) wrote :

Patch is at https://review.opendev.org/c/openstack/nova/+/766825 for review after stable/victoria change merges.

Revision history for this message
Lars Erik Pedersen (pedersen-larserik) wrote :

Any progress on this? :-)

Revision history for this message
melanie witt (melwitt) wrote :
Revision history for this message
melanie witt (melwitt) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 22.1.0

This issue was fixed in the openstack/nova 22.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 21.1.2

This issue was fixed in the openstack/nova 21.1.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 23.0.0.0rc1

This issue was fixed in the openstack/nova 23.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.