Changing node's properties in Ironic after node is deployed will count as available resources in Nova

Bug #1301279 reported by Lucas Alvares Gomes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Invalid
Medium
Zhenguo Niu
OpenStack Compute (nova)
Fix Released
Medium
Zhenguo Niu

Bug Description

If you increase the properties of a node which was already deployed the different will go to nova as available resources. For e.g, a node with properties/memory_mb=512 was deployed, and n-cpu is showing:

2014-04-02 10:37:26.514 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 0
2014-04-02 10:37:26.514 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 0
2014-04-02 10:37:26.514 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 0

Now if we update that that to properties/memory_mb=1024, the difference will be shown in nova as available resources:

2014-04-02 10:40:48.266 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 512
2014-04-02 10:40:48.266 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 0
2014-04-02 10:40:48.266 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 0

LOGs: http://paste.openstack.org/show/74806/

Changed in ironic:
importance: Undecided → High
assignee: nobody → Lucas Alvares Gomes (lucasagomes)
milestone: none → juno-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/84687

Changed in ironic:
status: New → In Progress
Revision history for this message
Adam Gandelman (gandelman-a) wrote : Re: Changing nodes properties after it's deployed will count as available resources in Nova

The issue here is that the ironic nova driver attempts to calculate total node resources AND the used resources. However, calculating the the amount of resources available in the presence of an instance is handled by nova's resource_tracker and that overrides the *_used keys we set in the driver:

https://git.openstack.org/cgit/openstack/nova/tree/nova/compute/resource_tracker.py#n596

If the properties of an Ironic node change while it is associated with an instance, I think the best we can do is lie about available resources until the instance has been deleted and the resource tracker hits the node when it is unused.

Changed in ironic:
assignee: Lucas Alvares Gomes (lucasagomes) → nobody
Dmitry Tantsur (divius)
Changed in ironic:
status: In Progress → Triaged
Revision history for this message
Ling Gao (linggao) wrote :

Discussed this on IRC with a few folks, the suggestion is not allow property changes for a node that is deployed. We should enforce that in Ironic API. If this sounds like a solution, I can go ahead add the enforcement part in the API.

Dmitry Tantsur (divius)
tags: added: driver
aeva black (tenbrae)
tags: added: nova-driver
removed: driver
Revision history for this message
aeva black (tenbrae) wrote :

It is conceivable that an operator may perform physical maintenance on a node, while an instance is deployed, that do in fact change its properties -- and I think we should continue to allow that to be reflected in Ironic's API.

If I understand correctly, the real problem here isn't whether nova sees additional resources -- it's whether nova will attempt to schedule a second instance on that node.

I think the solution is to adjust the scheduler filters to prevent that situation, rather than add constraints in Ironic's API that do not reflect reality. We could also change the calculation of available resources to be based on the flavor, rather than the node, which would avoid this problem.

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

Perhaps we should add a scheduler filter that checks node.instance_uuid; if it is None, it may be scheduled, otherwise it may not.

aeva black (tenbrae)
Changed in ironic:
milestone: juno-1 → none
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Doing this in a baremetal-specific scheduler filter seems like the right approach. Attempting to handle this in resource tracking on either end (nova or ironic) will end up in one side forcing fake calculation to emulate expected behavior and skewing accounting somewhere else.

aeva black (tenbrae)
Changed in ironic:
milestone: none → juno-2
aeva black (tenbrae)
Changed in ironic:
milestone: juno-2 → juno-3
Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :
Revision history for this message
Lucas Alvares Gomes (lucasagomes) wrote :

Will re-test looking into it

Changed in ironic:
assignee: nobody → Lucas Alvares Gomes (lucasagomes)
Revision history for this message
Lucas Alvares Gomes (lucasagomes) wrote :

Tested it today, this bug is not fixed, see logs: http://paste.openstack.org/show/102399/

Now this would involve changing things in the Nova driver, which I don't think that we need to do in J3. So I'm bumping it to ironic-next

Changed in ironic:
milestone: juno-3 → next
Dmitry Tantsur (divius)
summary: - Changing nodes properties after it's deployed will count as available
- resources in Nova
+ Changing node's properties in Ironic after node is deployed will count
+ as available resources in Nova
tags: added: ironic
Sean Dague (sdague)
Changed in nova:
status: New → Confirmed
importance: Undecided → Medium
Changed in ironic:
assignee: Lucas Alvares Gomes (lucasagomes) → nobody
aeva black (tenbrae)
Changed in ironic:
milestone: next → none
Nha Pham (phqnha)
Changed in ironic:
assignee: nobody → Nha Pham (phqnha)
Changed in nova:
assignee: nobody → Nha Pham (phqnha)
Changed in ironic:
milestone: none → kilo-rc1
status: Triaged → Confirmed
aeva black (tenbrae)
Changed in ironic:
importance: High → Medium
Changed in ironic:
milestone: kilo-rc1 → liberty-1
Revision history for this message
Zhenguo Niu (niu-zglinux) wrote :

hi Nha Pham (phqnha), are you still working on this, or maybe I can take over it, there has been no real activity for a long time.

Revision history for this message
Nha Pham (phqnha) wrote :

Hi, feel free to take it. I am so sorry for my inactivity.

Changed in ironic:
assignee: Nha Pham (phqnha) → nobody
Changed in nova:
assignee: Nha Pham (phqnha) → nobody
Changed in ironic:
assignee: nobody → Zhenguo Niu (niu-zglinux)
Changed in nova:
assignee: nobody → Zhenguo Niu (niu-zglinux)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/204088

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/204088
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b99fb0a51c658301188bbc729d1437a9c8b75d00
Submitter: Jenkins
Branch: master

commit b99fb0a51c658301188bbc729d1437a9c8b75d00
Author: Zhenguo Niu <email address hidden>
Date: Tue Jul 21 21:16:22 2015 +0800

    [Ironic]Not count available resources of deployed ironic node

    Changing node's properties in Ironic after node is deployed will
    count as available resources in Nova, as nova's resource_tracker
    will override the *_used keys we set in the driver.

    This patch changes the caculation of available resources based on
    the instance flavor instead of node to avoid the issue.

    Change-Id: I1a6e4501122c4781e103fd83a146e0914c7671c9
    Closes-Bug: #1301279

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

I don't believe there's anything to do here in Ironic, correct me if I'm wrong.

Changed in ironic:
status: Confirmed → Invalid
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-3 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers