PUT /allocations/{consumer_id} fails with a 500 if "resources: {}"

Bug #1714072 reported by Matt Riedemann
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Chris Dent
Pike
Fix Committed
Medium
Matt Riedemann

Bug Description

I hit this while writing a functional recreate test for bug 1713786 where the destination node during an evacuate doesn't have it's allocations created by the scheduler. When the source node comes up after the evacuation, it tries to remove the allocations on the source node, which is the only one because of bug 1713786, but that results in sending a request like this:

2017-08-30 14:45:13,495 INFO [nova.scheduler.client.report] Sending updated allocation [{'resource_provider': {'uuid': '7ab9dab7-65c6-4961-9403-c8fc50dedb6b'}, 'resources': {}}] for instance dc8a686c-ad92-48f3-8594-d00c6e671a1c after removing resources for 7ab9dab7-65c6-4961-9403-c8fc50dedb6b.

And you get this stacktrace in the Placement API:

2017-08-30 14:45:13,502 ERROR [nova.api.openstack.placement.handler] Uncaught exception
    Traceback (most recent call last):
      File "nova/api/openstack/placement/handler.py", line 217, in __call__
        return dispatch(environ, start_response, self._map)
      File "nova/api/openstack/placement/handler.py", line 144, in dispatch
        return handler(environ, start_response)
      File "/home/user/git/nova/.tox/functional/local/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
        resp = self.call_func(req, *args, **self.kwargs)
      File "nova/api/openstack/placement/wsgi_wrapper.py", line 29, in call_func
        super(PlacementWsgify, self).call_func(req, *args, **kwargs)
      File "/home/user/git/nova/.tox/functional/local/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
        return self.func(req, *args, **kwargs)
      File "nova/api/openstack/placement/microversion.py", line 268, in decorated_func
        return _find_method(f, version)(req, *args, **kwargs)
      File "nova/api/openstack/placement/util.py", line 138, in decorated_function
        return f(req)
      File "nova/api/openstack/placement/handlers/allocation.py", line 286, in set_allocations
        return _set_allocations(req, ALLOCATION_SCHEMA_V1_8)
      File "nova/api/openstack/placement/handlers/allocation.py", line 252, in _set_allocations
        allocations.create_all()
      File "nova/objects/resource_provider.py", line 1877, in create_all
        self._set_allocations(self._context, self.objects)
      File "/home/user/git/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_db/api.py", line 150, in wrapper
        ectxt.value = e.inner_exc
      File "/home/user/git/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
        self.force_reraise()
      File "/home/user/git/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
        six.reraise(self.type_, self.value, self.tb)
      File "/home/user/git/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_db/api.py", line 138, in wrapper
        return f(*args, **kwargs)
      File "/home/user/git/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 979, in wrapper
        return fn(*args, **kwargs)
      File "nova/objects/resource_provider.py", line 1835, in _set_allocations
        consumer_id = allocs[0].consumer_id
    IndexError: list index out of range

The schema validation on PUT /allocations requires a minimum of one provider in the request, but it doesn't validate that there is at least one resource for that provider:

https://github.com/openstack/nova/blob/da4083d7bc0a0c1272df35ecb12c4c2fd2102e21/nova/api/openstack/placement/handlers/allocation.py#L52-L61

Revision history for this message
Chris Dent (cdent) wrote :

what should the outcome be here:

* 400 you didn't follow the schema
* 204 we erased allocations for that resource provider

Revision history for this message
Matt Riedemann (mriedem) wrote :

I think 400. Since we said you have to provide at least one resource provider:

https://review.openstack.org/#/c/490195/

It would be weird to say, OK you can set one provider but no resources. And if you don't want allocations for a given consumer on any provider, then DELETE /allocations/{consumer_id} is what you need.

--

Also, I think we could use minProperties: 1 to fix this for the 'resources' key:

http://json-schema.org/latest/json-schema-validation.html#rfc.section.6.16

Chris Dent (cdent)
Changed in nova:
assignee: nobody → Chris Dent (cdent)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/499269

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/499270

Changed in nova:
assignee: Chris Dent (cdent) → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/499269
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ad2d3fc0f806f46382fa3c0a95c2a5d003d40736
Submitter: Jenkins
Branch: master

commit ad2d3fc0f806f46382fa3c0a95c2a5d003d40736
Author: Chris Dent <email address hidden>
Date: Wed Aug 30 20:19:31 2017 +0100

    [placement] Add test for empty resources in allocation

    If the resources object in an allocation request is an empty dict then
    there will be a 500 deep in the belly of the AllocationList object
    handling when it tries get a information. The patch demonstrates the bug
    with a new gabbi test that verfies the 500. The next patch will fix it.

    Change-Id: I5ab417e39044af63fb9acb57c48c087ecf1e6f04
    Related-Bug: #1714072

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/499270
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=10f8a9aa127cfaecab368e26c3b896e203e301bc
Submitter: Jenkins
Branch: master

commit 10f8a9aa127cfaecab368e26c3b896e203e301bc
Author: Chris Dent <email address hidden>
Date: Wed Aug 30 20:30:19 2017 +0100

    [placement] Require at least one resource class in allocation

    If an allocation was made with an empty resources object, a 500 response
    code would result. This change adjusts the schema to use minProperties
    of 1 to say there must be at least one resource class and value pair in
    the allocation. If there is not a 400 response is returned.

    As this is fixing 500 response to a useful error, no microversion is
    required. A previous gabbi file which demonstrated the problem has been
    updated to demonstrate that the problem has been fixed.

    Change-Id: I7d9c64c77586564fb3bdbe92c693bd2a1bc06f24
    Closes-Bug: #1714072

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/499615

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/499616

Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Chris Dent (cdent)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/499615
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=334905a753c660c5f02ec6c03c427ebbbaf4b89a
Submitter: Jenkins
Branch: stable/pike

commit 334905a753c660c5f02ec6c03c427ebbbaf4b89a
Author: Chris Dent <email address hidden>
Date: Wed Aug 30 20:19:31 2017 +0100

    [placement] Add test for empty resources in allocation

    If the resources object in an allocation request is an empty dict then
    there will be a 500 deep in the belly of the AllocationList object
    handling when it tries get a information. The patch demonstrates the bug
    with a new gabbi test that verfies the 500. The next patch will fix it.

    Change-Id: I5ab417e39044af63fb9acb57c48c087ecf1e6f04
    Related-Bug: #1714072
    (cherry picked from commit ad2d3fc0f806f46382fa3c0a95c2a5d003d40736)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/499616
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2b82aa3a9978ecb31632fae6b400d7fe9137c01b
Submitter: Jenkins
Branch: stable/pike

commit 2b82aa3a9978ecb31632fae6b400d7fe9137c01b
Author: Chris Dent <email address hidden>
Date: Wed Aug 30 20:30:19 2017 +0100

    [placement] Require at least one resource class in allocation

    If an allocation was made with an empty resources object, a 500 response
    code would result. This change adjusts the schema to use minProperties
    of 1 to say there must be at least one resource class and value pair in
    the allocation. If there is not a 400 response is returned.

    As this is fixing 500 response to a useful error, no microversion is
    required. A previous gabbi file which demonstrated the problem has been
    updated to demonstrate that the problem has been fixed.

    Change-Id: I7d9c64c77586564fb3bdbe92c693bd2a1bc06f24
    Closes-Bug: #1714072
    (cherry picked from commit 10f8a9aa127cfaecab368e26c3b896e203e301bc)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.1

This issue was fixed in the openstack/nova 16.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b1

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.