OpenStack Compute (nova)

Existing Ironic instances report negative available RAM for the node after upgrade

Bug #1502177 reported by Jim Rollenhagen on 2015-10-02

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Jim Rollenhagen	OpenStack Compute (nova) 12.0.0 "liberty"

Bug Description

Ironic nodes that have an existing instance will report negative available RAM after upgrading beyond this commit: https://github.com/openstack/nova/commit/b99fb0a51c658301188bbc729d1437a9c8b75d00

The node attached to the instance will not have instance_info[memory_mb], etc, set on the node object in Ironic. This code causes the driver to report memory_mb_used=memory_mb=0 if this info is unset. The resource tracker notices that an instance is on that node and sets memory_mb_used to X (the size of the instance). After which the node reports (-X) available memory.

This can wreak havoc on tools that look at total available memory. These could range from capacity reporting tools to scheduler/cells filters. If more than half of the capacity has instances, the total memory available will be negative, and could cause things to not schedule properly or generate alerts.

Tags:

John Garbutt (johngarbutt) on 2015-10-02

tags:	added: kilo-rc-potential
tags:	added: liberty-rc-potential removed: kilo-rc-potential

OpenStack Infra (hudson-openstack) on 2015-10-02

Changed in nova:
assignee:	nobody → Jim Rollenhagen (jim-rollenhagen)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-05: Fix merged to nova (master)

Reviewed: https://review.openstack.org/230487
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=047da6498dbb3af71bcb9e6d0e2c38aa23b06615
Submitter: Jenkins
Branch: master

commit 047da6498dbb3af71bcb9e6d0e2c38aa23b06615
Author: Jim Rollenhagen <email address hidden>
Date: Thu Oct 1 21:12:53 2015 -0700

Ironic: Fix bad capacity reporting if instance_info is unset

    If node.instance_info is unset for a node that has an instance on it
    (for example, when upgrading Nova from before the patch that set these
    values on spawn), the Ironic driver reports 0 for both $resource and
    $resource_used. The resource tracker will correct $resource_used, and
    resources available will be reported as a negative number.

Correct this by reporting the original value if instance_info is unset.

Closes-Bug: #1502177
Change-Id: I13c5e5430fd305cd0ee2f24cd95304660ccf11eb

Changed in nova:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-05: Fix proposed to nova (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/231179

Jim Rollenhagen (jim-rollenhagen) on 2015-10-05

Changed in nova:
importance:	Undecided → High

John Garbutt (johngarbutt) on 2015-10-06

Changed in nova:
milestone:	none → liberty-rc2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-06: Fix merged to nova (stable/liberty)

Reviewed: https://review.openstack.org/231179
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8af97531e3fbd2f24c091f14c3a20cacf538cff2
Submitter: Jenkins
Branch: stable/liberty

commit 8af97531e3fbd2f24c091f14c3a20cacf538cff2
Author: Jim Rollenhagen <email address hidden>
Date: Thu Oct 1 21:12:53 2015 -0700

Ironic: Fix bad capacity reporting if instance_info is unset

Correct this by reporting the original value if instance_info is unset.

    Closes-Bug: #1502177
    Change-Id: I13c5e5430fd305cd0ee2f24cd95304660ccf11eb
    (cherry picked from commit 047da6498dbb3af71bcb9e6d0e2c38aa23b06615)

tags:

added: in-stable-liberty

Thierry Carrez (ttx) on 2015-10-06

Changed in nova:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-10-15

Changed in nova:
milestone:	liberty-rc2 → 12.0.0

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-15: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/235181

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-23: Fix merged to nova (master)

Download full text (10.6 KiB)

Reviewed: https://review.openstack.org/235181
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8a3b9d6408627ce9a293d1e62d003757af655a1
Submitter: Jenkins
Branch: master

commit 6df6ad3ff32f2b1fe2978df1032002548ad8eb66
Author: Davanum Srinivas <email address hidden>
Date: Wed Oct 7 08:11:35 2015 -0700

Omnibus stable/liberty fix

    There are currently 3 different blocking issues in stable/liberty due
    to library releases: webob 1.5, oslo.db 3.0.0, and
    oslo.versionedobjects 0.11.0. This is a squashed fix for all of them
    as none can land without the others.

Issue #1 - oslo.db

Add testresources used by oslo.db fixture

If we use oslo.db fixtures, we'll need the package or
the next version of oslo.db release will break us.

(Cherry-picked from 4bcc26487837b7ece7797f88622dea1b6d09bd94)

Closes-Bug: #1503501

Issue #2 - oslo.versionedobjects

Drop unused obj_to_primitive() override

    This was a band-aid override until o.vo gained the obj_relationships fix
    that this method overrides. That has been in place since o.vo 0.8.0, which
    means this is long since no longer necessary (and is actually blocking our
    ability to absorb bug fixes to this code in o.vo). Further, we no longer
    use this directly because we're doing backports based on version manifests,
    which means we no longer consult child_versions _or_ obj_relationships.

(cherry picked from commit 142f1d9cc4ace90956c665c40b1f78795f9f7e29)

Issue #3 - webob

Default ConvertedException code to 500

    webob 1.5.0 released on 10/11 has change f6c749011 which
    strictly enforces status codes in exceptions, and 0 is not
    a valid status code so tests fail.

Change the default to 500 to match the default in the parent
class in webob.

Closes-Bug: #1505153
(cherry picked from commit 10438c0fc34bd088e018e1a5e8ec57b396528792)

Change-Id: I1e06e77308a7dd23209124f0807d61fb52470188

commit 606204354b5ed96852240020769c81acda9f9fc8
Author: Matt Riedemann <email address hidden>
Date: Mon Oct 5 20:32:58 2015 +0000

Revert "[libvirt] Move cleanup of imported files to imagebackend"

This reverts commit 9ba70756de326ffaa8be43acfde12cad04ed0af2

    The change introduced an UnboundLocalError if we fail to
    create the config_drive_image variable. Also, the original
    change didn't have any unit tests and came late in the
    liberty release so I don't really want to mess with fixing
    this given we need the fix in liberty-rc2.

Change-Id: Ia7b70aa139b67cf58b5c0f9fbcd2a4deb465914e
Closes-Bug: #1502961

commit ef655379445693443146f8a3ed31cabb011d9937
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Oct 8 06:41:06 2015 +0000

Imported Translations from Zanata

For more information about this automatic import see:
https://wiki.openstack.org/wiki/Translations/Infrastructure

Change-Id: Idcac653033ab9808e06451a0dd690db4736834b2

commit eda3029aa74932f421d2992ac24f5ac3c92f347c
Author: Dan Smith <email address hidden>
Date: Tue Oct 6 10:58:18 2...

Reviewed:  https://review.openstack.org/235181
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8a3b9d6408627ce9a293d1e62d003757af655a1
Submitter: Jenkins
Branch:    master

commit 6df6ad3ff32f2b1fe2978df1032002548ad8eb66
Author: Davanum Srinivas <davanum@gmail.com>
Date:   Wed Oct 7 08:11:35 2015 -0700

Omnibus stable/liberty fix
    
    There are currently 3 different blocking issues in stable/liberty due
    to library releases: webob 1.5, oslo.db 3.0.0, and
    oslo.versionedobjects 0.11.0. This is a squashed fix for all of them
    as none can land without the others.
    
    Issue #1 - oslo.db
    
    Add testresources used by oslo.db fixture
    
    If we use oslo.db fixtures, we'll need the package or
    the next version of oslo.db release will break us.
    
    (Cherry-picked from 4bcc26487837b7ece7797f88622dea1b6d09bd94)
    
    Closes-Bug: #1503501
    
    Issue #2 - oslo.versionedobjects
    
    Drop unused obj_to_primitive() override
    
    This was a band-aid override until o.vo gained the obj_relationships fix
    that this method overrides. That has been in place since o.vo 0.8.0, which
    means this is long since no longer necessary (and is actually blocking our
    ability to absorb bug fixes to this code in o.vo). Further, we no longer
    use this directly because we're doing backports based on version manifests,
    which means we no longer consult child_versions _or_ obj_relationships.
    
    (cherry picked from commit 142f1d9cc4ace90956c665c40b1f78795f9f7e29)
    
    Issue #3 - webob
    
    Default ConvertedException code to 500
    
    webob 1.5.0 released on 10/11 has change f6c749011 which
    strictly enforces status codes in exceptions, and 0 is not
    a valid status code so tests fail.
    
    Change the default to 500 to match the default in the parent
    class in webob.
    
    Closes-Bug: #1505153
    (cherry picked from commit 10438c0fc34bd088e018e1a5e8ec57b396528792)
    
    Change-Id: I1e06e77308a7dd23209124f0807d61fb52470188

commit 606204354b5ed96852240020769c81acda9f9fc8
Author: Matt Riedemann <mriedem@us.ibm.com>
Date:   Mon Oct 5 20:32:58 2015 +0000

Revert "[libvirt] Move cleanup of imported files to imagebackend"
    
    This reverts commit 9ba70756de326ffaa8be43acfde12cad04ed0af2
    
    The change introduced an UnboundLocalError if we fail to
    create the config_drive_image variable. Also, the original
    change didn't have any unit tests and came late in the
    liberty release so I don't really want to mess with fixing
    this given we need the fix in liberty-rc2.
    
    Change-Id: Ia7b70aa139b67cf58b5c0f9fbcd2a4deb465914e
    Closes-Bug: #1502961

commit ef655379445693443146f8a3ed31cabb011d9937
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Oct 8 06:41:06 2015 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: Idcac653033ab9808e06451a0dd690db4736834b2

commit eda3029aa74932f421d2992ac24f5ac3c92f347c
Author: Dan Smith <dansmith@redhat.com>
Date:   Tue Oct 6 10:58:18 2015 -0700

Fix InstanceV1 backports to use context
    
    This fixes the InstanceV1 and InstanceListV1 backport code to properly
    initialize the temporary objects with a context. Without them, the
    objects are orphaned and can't do certain things.
    
    This should be backported to liberty
    
    Closes-Bug: #1503708
    Change-Id: I537734c0f10110ac64799cc1232edbc18a16c794
    (cherry picked from commit e571d88604353a0443ae4ca24e3ce72fb4c615f2)

commit 6cc3f4f6043e83d5d180a51bac5b9254cee8245e
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Tue Oct 6 06:39:20 2015 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: Ib23288ea4eefac2f44cc9ac6bcb672dba4c3f6cb

commit 8af97531e3fbd2f24c091f14c3a20cacf538cff2
Author: Jim Rollenhagen <jim@jimrollenhagen.com>
Date:   Thu Oct 1 21:12:53 2015 -0700

Ironic: Fix bad capacity reporting if instance_info is unset
    
    If node.instance_info is unset for a node that has an instance on it
    (for example, when upgrading Nova from before the patch that set these
    values on spawn), the Ironic driver reports 0 for both $resource and
    $resource_used. The resource tracker will correct $resource_used, and
    resources available will be reported as a negative number.
    
    Correct this by reporting the original value if instance_info is unset.
    
    Closes-Bug: #1502177
    Change-Id: I13c5e5430fd305cd0ee2f24cd95304660ccf11eb
    (cherry picked from commit 047da6498dbb3af71bcb9e6d0e2c38aa23b06615)

commit a79e09548844fdb02b96a7f033efc0b23e7882e3
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Mon Oct 5 06:33:17 2015 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I7ad39496a3e4bab408d208c88b2e2e4b9f776b7c

commit 83089e7e67ca0149f9637d73448edfcba1edbf84
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Sat Oct 3 06:34:44 2015 +0000

Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure
    
    Change-Id: I2ddf857098bbda7048caadb752b659f446d192cc

commit c24bf9e469dbc2df8e8b8900015197dc7c388e8c
Author: Matt Riedemann <mriedem@us.ibm.com>
Date:   Fri Oct 2 13:00:43 2015 -0700

libvirt: check if ImageMeta.disk_format is set before accessing it
    
    Commit 4f43bbafa4bfc2ab149d5db4eb55320c1602658b changed
    blockinfo.get_root_info to access the disk_format attribute on the
    passed in image_meta (ImageMeta object) like an object, but if that
    attribute is not set you'll get an error.
    
    image_meta can be an empty object in the case of evacuate where
    image_ref is passed as None to the compute manager's rebuild_instance
    method in the case of evacuate, so image_meta defaults to {}. So when
    the ImageMeta object is created in LibvirtDriver.spawn, there is nothing
    in it.
    
    We have to check if the attribute is set before accessing it using
    obj_attr_is_set because the ImageMeta class does not include the
    NovaObjectDictCompat mixin, so we can't use
    image_meta.get('disk_format').
    
    Closes-Bug: #1501831
    
    Change-Id: I95e543415b881603bb997ae0e159017c03e58a53
    (cherry picked from commit 2b1a2bc034e371a9f6d95f48ed8fa77e0854d675)

commit 579b02163e66502d22812f91d82e65f4019bedd4
Author: Tony Breeds <tony@bakeyournoodle.com>
Date:   Fri Oct 2 12:28:42 2015 +1000

Add Pillow to test-requirements.txt
    
    Pillow isn't directly used by nova but it is used by
    sphinxcontrib-seqdiag.
    
    Add it to test-requirements as a work around to:
    http://lists.openstack.org/pipermail/openstack-dev/2015-October/076030.html
    
    Change-Id: I4face1fdb56c30bfeffca8808f422565eb21de83
    (cherry picked from commit 03cae1c43124d03e747ea51cfcbc0737bbf4287a)

commit 2dbd0512ed4e51edab2a2f2d23d759ae8da9dc0c
Author: Andreas Jaeger <aj@suse.de>
Date:   Thu Sep 24 20:39:16 2015 +0200

Cleanup of Translations
    
    In preparation for the release, do some cleanups for translations.
    
    Removes all po files that are partially translated. The translation
    team has decided to exclude files with less than 66 % of translated
    content. There is no content lost, all data is in the translation
    server, we just remove it from the repository.
    
    This change needs to be done manually since the automatic import does
    not handle some of these cases.
    
    Change-Id: I662f6af336947188c74ea439ebba308bd4055fb8

commit fb23e596542d6bd052f5d3c5e193ffea7bb26028
Author: Nikola Dipanov <ndipanov@redhat.com>
Date:   Wed Sep 23 19:21:16 2015 +0100

rebuild: Apply migration context before calling the driver
    
    In I0233f964d8f294f0ffd9edcb16b1aaf93486177f we made sure that we are
    properly using the data stashed in the migration context for resource
    tracking, however - we missed that we need to also apply it when passing
    data to the driver to do the spawn (or rebuild if implemented).
    
    This patch adds a context manager that helps us temporarily apply the
    migration context to an instance object, as we want to make sure that we
    don't save the new data before it's ready to be used by the resource
    tracker.
    
    Change-Id: I9a4b86e176f408c1495d5308d3e7041292c2625a
    Closes-bug: #1499028
    (cherry picked from commit 66e0744471ce491371a67ffdd8ba3c105919e255)

commit 3be67646c46f7ca58b570770074c1f269c187a30
Author: He Jie Xu <hejie.xu@intel.com>
Date:   Mon Sep 28 13:24:06 2015 +0800

Correct Instance type check to work with InstanceV1
    
    When Liberty controller running with Kilo compute node, the Kilo
    compute node will send the InstanceV1 to the Liberty controller.
    Then all the checks like "isinstance(instance, objects.Instance)"
    on the Liberty controller won't works. Because objects.Instance
    pointed to objects.InstanceV2 now.
    
    This patch corrects all the Instance type check to make sure the
    upgrade works.
    
    Change-Id: Ib431c08dc6934631a71ef55c564fbbf4bde22642
    Closes-bug: #1500289
    (cherry picked from commit 093272489ceb13d4df26916c7ed8b66e39f7c1a4)

commit 51d74bdf18df39695ad5e500db6db830f21da36a
Author: Yingxin Cheng <yingxin.cheng@intel.com>
Date:   Wed Sep 9 11:20:19 2015 -0400

Fix MetricWeigher to use MonitorMetricList
    
    The commit I4dfbea27ce6c3eecc1a8658b1f9dc0feb2298705 changes
    'HostState.metrics' to 'objects.MonitorMetricList'. But MetricWeigher
    still use the old dictionary of 'host_manager.MetricItem'. This patch
    corrects the behavior of MetricWeigher with related tests, and delete
    unused MetricItem.
    
    Co-Authored-By: Sylvain Bauza <sbauza@redhat.com>
    
    Closes-Bug: #1493680
    (cherry picked from commit fd2b868d41a632748939b82ed1b8927adcdfb113)
    
    Fix NoneType error when calling MetricsWeigher
    
    (this is a squashed commit since commit fd2b868 was buggy)
    Given that dict comprehensions don't allow to have None as a list, we had
    a problem with the Ironic gate-ironic-inspector-dsvm job because
    host_state.metrics was set to None by default.
    
    Adding a unittest to make sure we don't regress.
    
    Closes-Bug: #1499271
    (cherry picked from commit ccea5d6b0ace535b375d3e63bd572885cb5dbc91)
    
    Change-Id: I9690bc4f6f1dca8f2bf8c5a83f16af6200015506

Revision history for this message

Thierry Carrez (ttx) wrote on 2015-12-02: Fix included in openstack/nova 13.0.0.0b1

This issue was fixed in the openstack/nova 13.0.0.0b1 development milestone.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.