Changing node's properties in Ironic after node is deployed will count as available resources in Nova

[Lucas Alvares Gomes]

If you increase the properties of a node which was already deployed the different will go to nova as available resources. For e.g, a node with properties/memory_mb=512 was deployed, and n-cpu is showing:

2014-04-02 10:37:26.514 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 0
2014-04-02 10:37:26.514 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 0
2014-04-02 10:37:26.514 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 0

Now if we update that that to properties/memory_mb=1024, the difference will be shown in nova as available resources:

2014-04-02 10:40:48.266 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 512
2014-04-02 10:40:48.266 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 0
2014-04-02 10:40:48.266 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 0

LOGs: http://paste.openstack.org/show/74806/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/84687

[Adam Gandelman (gandelman-a)]

The issue here is that the ironic nova driver attempts to calculate total node resources AND the used resources. However, calculating the the amount of resources available in the presence of an instance is handled by nova's resource_tracker and that overrides the *_used keys we set in the driver:

https://git.openstack.org/cgit/openstack/nova/tree/nova/compute/resource_tracker.py#n596

If the properties of an Ironic node change while it is associated with an instance, I think the best we can do is lie about available resources until the instance has been deleted and the resource tracker hits the node when it is unused.

[Ling Gao (linggao)]

Discussed this on IRC with a few folks, the suggestion is not allow property changes for a node that is deployed. We should enforce that in Ironic API. If this sounds like a solution, I can go ahead add the enforcement part in the API.

[aeva black (tenbrae)]

It is conceivable that an operator may perform physical maintenance on a node, while an instance is deployed, that do in fact change its properties -- and I think we should continue to allow that to be reflected in Ironic's API.

If I understand correctly, the real problem here isn't whether nova sees additional resources -- it's whether nova will attempt to schedule a second instance on that node.

I think the solution is to adjust the scheduler filters to prevent that situation, rather than add constraints in Ironic's API that do not reflect reality. We could also change the calculation of available resources to be based on the flavor, rather than the node, which would avoid this problem.

[Jim Rollenhagen (jim-rollenhagen)]

Perhaps we should add a scheduler filter that checks node.instance_uuid; if it is None, it may be scheduled, otherwise it may not.

[Adam Gandelman (gandelman-a)]

Doing this in a baremetal-specific scheduler filter seems like the right approach. Attempting to handle this in resource tracking on either end (nova or ironic) will end up in one side forcing fake calculation to emulate expected behavior and skewing accounting somewhere else.

[Jim Rollenhagen (jim-rollenhagen)]

This seems to be fixed already, as I suggested in June: https://github.com/openstack/ironic/blob/2412946e25fcfc7c7776582a7b858c1b10340624/ironic/nova/virt/ironic/driver.py#L216-220

[Lucas Alvares Gomes (lucasagomes)]

Will re-test looking into it

[Lucas Alvares Gomes (lucasagomes)]

Tested it today, this bug is not fixed, see logs: http://paste.openstack.org/show/102399/

Now this would involve changing things in the Nova driver, which I don't think that we need to do in J3. So I'm bumping it to ironic-next

[Zhenguo Niu (niu-zglinux)]

hi Nha Pham (phqnha), are you still working on this, or maybe I can take over it, there has been no real activity for a long time.

[Nha Pham (phqnha)]

Hi, feel free to take it. I am so sorry for my inactivity.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/204088

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/204088
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b99fb0a51c658301188bbc729d1437a9c8b75d00
Submitter: Jenkins
Branch: master

commit b99fb0a51c658301188bbc729d1437a9c8b75d00
Author: Zhenguo Niu <email address hidden>
Date: Tue Jul 21 21:16:22 2015 +0800

    [Ironic]Not count available resources of deployed ironic node

    Changing node's properties in Ironic after node is deployed will
    count as available resources in Nova, as nova's resource_tracker
    will override the *_used keys we set in the driver.

    This patch changes the caculation of available resources based on
    the instance flavor instead of node to avoid the issue.

    Change-Id: I1a6e4501122c4781e103fd83a146e0914c7671c9
    Closes-Bug: #1301279

[Jim Rollenhagen (jim-rollenhagen)]

I don't believe there's anything to do here in Ironic, correct me if I'm wrong.