Unable to transition to Ironic Node Resource Classes in Pike

[John Garbutt]

So the scenario is:

* upgraded to pike
* have ironic with a multiple flavor
* attempting to transition to resource class based scheduling, now pike is installed

In Pike we ask people to:

* Update Ironic Node with a Resource Class
* Update flavors to request the new Resource Class (and not request VCPU, RAM, DISK), using the docs: https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html#scheduling-based-on-resource-classes

Consider this case:

* some old instances are running from before the updates
* some new instances are created after the updates

In placement:

* all inventory is correct, new resource class and legacy resource classes are both present
* old instance allocations: only request

In nova db:

* old instances and new instances correctly request the new resource class in their flavor
* new instances also include the anti-request for VCPU, DISK and RAM

Now this is the flow that shows the problem:

* get list of candidate allocations
* this includes nodes that already have instances on (they only claim part of the inventory, but the new instance is only requesting the bit of the inventory the old instance isn't using)
* boom, scheduling new instances fails after you hit the retry count, unless you got lucky and found a free slot by accident

Possible reason for this:

* Pike no longer updated instance allocations, if we updated the allocations of old instances to request the new custom resource class allocations, we would fix the above issue.

Possible work around:

* in the new flavor, keep requesting VCPU, RAM and CPU resources for pike, fix that up in queens?

[Matt Riedemann (mriedem)]

From IRC, my way of describing the problem:

(9:24:05 AM) mriedem: do you mean the custom resource class allocatoin isn't showing up for the instance?
(9:24:37 AM) johnthetubaguy: I mean old instances don't have allocations for the new resource, which breaks the scheduling of new flavors that use the resource class
(9:24:50 AM) mriedem: so instance A had vcpu/ram/disk allocations before the node.resource_class was set, then you set node.resource_class = baremetal and now you expect to see a 'baremetal' allocation for instance A in placement
(9:25:17 AM) johnthetubaguy: that is what I expected, yes
(9:25:24 AM) johnthetubaguy: (clearly incorrectly)
(9:25:31 AM) mriedem: that's probably because the RT isn't reporting allocations anymore once all of your computes are pike

(9:27:48 AM) mriedem: johnthetubaguy: ok so the scheduling issue is a new instance request can try to claim the 'baremetal' resource on node A even though instance A is already using it
(9:27:52 AM) mriedem: which causes a scheduler failure yes?
(9:28:00 AM) johnthetubaguy: mriedem: yes

We made a similar sounding fix in Pike https://review.openstack.org/#/c/492964/ but that was for reporting inventory with the new node.resource_class, not for updating instance allocations for a migrated flavor/instance.

[Matt Riedemann (mriedem)]

One thought is that the ironic driver itself could update the allocations once a flavor is migrated and has a new custom resource class. That gets tricky for anyone that has already upgraded to pike and some of their instances / nodes have migrated - the nova code would have to check the existing allocations and see if they contain the custom resource class, and we don't want to do that check on every flavor migration run (which happens when nodes are refreshed).

[John Garbutt (johngarbutt)]

Best way forward seems to be fixing up the allocations in here:
https://github.com/openstack/nova/blob/e11a8687aef527eee9f7c733db6cccd5b902afbb/nova/virt/ironic/driver.py#L523

[Matt Riedemann (mriedem)]

To deal with the issue in comment #2 above, before we get here:

https://github.com/openstack/nova/blob/e11a8687aef527eee9f7c733db6cccd5b902afbb/nova/virt/ironic/driver.py#L523

We'd have to check if the instance has an allocation in placement for the normalized_rc and if not, add it. That is at most redundant once per instance per compute service restart because of this:

https://github.com/openstack/nova/blob/e11a8687aef527eee9f7c733db6cccd5b902afbb/nova/virt/ironic/driver.py#L559

So we would only process that once.

[Matt Riedemann (mriedem)]

(10:00:03 AM) mriedem: johnthetubaguy: one issue might be a chicken and egg with the custom RC inventory being available when we PUT /allocations/{consumer_uuid}
(10:00:20 AM) mriedem: because the inventory gets updated via the RT periodic
(10:01:01 AM) mriedem: so if i just set a custom rc on my ironic node, the periodic runs, it detects the new rc and will update inventory, but while in the driver we're migrating the instance because it's node has an rc now, if we try updating the allocation for that rc before the node RP inventory has that rc, it will fail with a 409
(10:01:07 AM) johnthetubaguy: mriedem: dang, yeah
(10:01:08 AM) mriedem: "409 Conflict if there is no available inventory in any of the resource providers for any specified resource classes or inventories are updated by another thread while attempting the operation."

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/513085

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/513198

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513085
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5c2b8675e3e13e32b23681153f226de93bb99628
Submitter: Zuul
Branch: master

commit 5c2b8675e3e13e32b23681153f226de93bb99628
Author: John Garbutt <email address hidden>
Date: Wed Oct 18 17:05:43 2017 +0100

    Keep updating allocations for Ironic

    When ironic updates the instance.flavor to require the new custom
    resource class, we really need the allocations to get updated. Easiest
    way to do that is to make the resource tracker keep updating allocations
    for the ironic virt driver. This can be dropped once the transition to
    custom resource classes is complete.

    If we were not to claim the extra resources, placement will pick nodes
    that already have instances running on them when you boot an instance
    with a flavor that only requests the custom resource class. This should
    be what all ironic flavors do, before the upgrade to queens is
    performed.

    Closes-Bug: #1724589

    Change-Id: Ibbf65a8d817d359786abcdffa6358089ed1107f6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513198
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=842641c9b0b8f60d0e19b38ad1180078e9c4330c
Submitter: Zuul
Branch: stable/pike

commit 842641c9b0b8f60d0e19b38ad1180078e9c4330c
Author: John Garbutt <email address hidden>
Date: Wed Oct 18 17:05:43 2017 +0100

    Keep updating allocations for Ironic

    When ironic updates the instance.flavor to require the new custom
    resource class, we really need the allocations to get updated. Easiest
    way to do that is to make the resource tracker keep updating allocations
    for the ironic virt driver. This can be dropped once the transition to
    custom resource classes is complete.

    If we were not to claim the extra resources, placement will pick nodes
    that already have instances running on them when you boot an instance
    with a flavor that only requests the custom resource class. This should
    be what all ironic flavors do, before the upgrade to queens is
    performed.

    Closes-Bug: #1724589

    Change-Id: Ibbf65a8d817d359786abcdffa6358089ed1107f6
    (cherry picked from commit 5c2b8675e3e13e32b23681153f226de93bb99628)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.0.2 release.