placement reshaper doesn't clear all inventories for a resource provider

[Chris Dent]

The /reshaper API is willing to accept an empty dictionary for the inventories attribute of a resource provider. This is intended to mean "clear all the inventory".

However, the backend transformer code is not prepared to handle this:

      File "nova/api/openstack/placement/handlers/reshaper.py", line 103, in reshape
        rp_obj.reshape(context, inventory_by_rp, allocation_objects)
      File "/mnt/share/cdentsrc/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 993, in wrapper
        return fn(*args, **kwargs)
      File "nova/api/openstack/placement/objects/resource_provider.py", line 4087, in reshape
        rp = new_inv_list[0].resource_provider
      File "/mnt/share/cdentsrc/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 829, in __getitem__
        return self.objects[index]
    IndexError: list index out of range

This happening because 'new_inv_list' can be empty at

    for rp_uuid, new_inv_list in inventories.items():
        LOG.debug("reshaping: *interim* inventory replacement for provider %s",
                  rp_uuid)
        rp = new_inv_list[0].resource_provider

If the length of new_inv_list is zero we need to do nothing for this iteration though the loop.

Then a few lines later at

    for rp_uuid, new_inv_list in inventories.items():
        LOG.debug("reshaping: *final* inventory replacement for provider %s",
                  rp_uuid)
        # TODO(efried): If we wanted this to be more efficient, we could keep
        # track of providers for which all inventories are being deleted in the
        # above loop and just do those and skip the rest, since they're already
        # in their final form.
        new_inv_list[0].resource_provider.set_inventory(new_inv_list)

We have the same IndexError problem and need to behave differently.

A thing we might do is instead of using the resource_provider object on the (maybe not there) inventory objects, is create a new object: we have the rp_uuid.

[Eric Fried (efried)]

We still need to be careful not to duplicate rp objects, because of the generation conflict thing we already saw. I think we should whip through the old/new inventory objects (and maybe the allocation objects too) and build up a dict of rp_uuid: rp_obj to use in those two loops so we don't ever have to assume [0] exists.

I'll work through something this morning.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/585033

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585033
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d13e5a5011af75cfb6141a6b5d6aaf0f1a042aeb
Submitter: Zuul
Branch: master

commit d13e5a5011af75cfb6141a6b5d6aaf0f1a042aeb
Author: Eric Fried <email address hidden>
Date: Mon Jul 23 12:10:12 2018 -0500

    reshaper: Look up provider if not in inventories

    Per the referenced bug, we weren't accounting for the scenario where a
    reshape operation was removing *all* inventories for a provider (which
    could be fairly common). With this fix, we do a three-stage lookup of
    the provider object: If it's not in the inventories, we look in the
    allocations; if it's not in the allocations, we look it up in the
    database.

    Change-Id: I594bb64f87c61b7ffd39c19e0fd42c4c087a3a11
    Closes-Bug: #1783130

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.