Missing attributes in libvirt xml metadata cause ceilometer-compute to stop polling libvirt guests

Bug #1749960 reported by Lars Kellogg-Stedman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
High
Lars Kellogg-Stedman

Bug Description

We have discovered there are some nova instances in our (pike) environment that have empty "<nova:owner>" attributes in the metadata embedded in the libvirt XML. For example:

  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="14.0.3-9.el7ost"/>
      <nova:name>wm-solr</nova:name>
      <nova:creationTime>2017-05-24 15:29:24</nova:creationTime>
      <nova:flavor name="m1.large">
        <nova:memory>8192</nova:memory>
        <nova:disk>80</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner/>
      <nova:root type="image" uuid="a332ad63-8b38-4700-b818-b39fa69233a9"/>
    </nova:instance>
  </metadata>

In ceilometer/compute/discovery.py, ceilometer is asking for the owner/user and owner/project attributes with no error checking. This results in an AttributeError exception, which aborts the entire discover_libvirt_polling method and means that any remaining libvirt guests on the system will not be polled. This results in many missing metrics.

Following the theory that we should "be conservative in what you do, be liberal in what we accept from others", it seems as if it would be reasonable to catch these exceptions, report an error, and continue polling other guests on the system.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/545303

Changed in ceilometer:
assignee: nobody → Lars Kellogg-Stedman (larsks)
status: New → In Progress
tags: added: pike-backport-potential
Revision history for this message
gordon chung (chungg) wrote :

is this also a bug in Nova? i imagine not having user/project details will making actually querying for this data difficult.

Changed in ceilometer:
importance: Undecided → High
Revision history for this message
Lars Kellogg-Stedman (larsks) wrote :

This is also a bug in Nova, yes. There is some suggestion that https://review.openstack.org/#/c/399679/ fixed this in Nova.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/545303
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=d2af7e9ece2f903633bf4e95208b2e3ba6ac464d
Submitter: Zuul
Branch: master

commit d2af7e9ece2f903633bf4e95208b2e3ba6ac464d
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Feb 16 08:58:06 2018 -0500

    Gracefully handle missing metadata in libvirt xml

    Missing metadata in the libvirt domain xml for a nova instance would
    cause ceilometer-compute to abort, leading to missing metrics for the
    current and any subequent libvirt guests.

    This commit puts a try/except AttributeError block around all the code
    that is fetching attributes on the result of metadata_xml.find(...).

    Change-Id: I8adc609cc21c86de2daba326d24b73a80d6eb61f
    Closes-Bug: #1749960

Changed in ceilometer:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/545376

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/545403

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/545504

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ceilometer (stable/pike)

Change abandoned by Lars Kellogg-Stedman (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/545376
Reason: This patch was missing a 'continue'.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ceilometer (stable/queens)

Change abandoned by gordon chung (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/545403

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/545504
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=050b5293b3fa3b01a84b3e79e84f5ddb7fba0872
Submitter: Zuul
Branch: master

commit 050b5293b3fa3b01a84b3e79e84f5ddb7fba0872
Author: Lars Kellogg-Stedman <email address hidden>
Date: Sat Feb 17 00:01:01 2018 -0500

    the previous patch was missing a 'continue'

    This commit puts a try/except AttributeError block around all the code
    that is fetching attributes on the result of metadata_xml.find(...).

    Change-Id: I41aa76cf9def3e8c4bceef0280d15c1fd7c48e3d
    Closes-Bug: #1749960

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/547441

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/pike)

Reviewed: https://review.openstack.org/545376
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=48515b4841c462ead3d04928801304b093d7594d
Submitter: Zuul
Branch: stable/pike

commit 48515b4841c462ead3d04928801304b093d7594d
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Feb 16 08:58:06 2018 -0500

    Gracefully handle missing metadata in libvirt xml

    Missing metadata in the libvirt domain xml for a nova instance would
    cause ceilometer-compute to abort, leading to missing metrics for the
    current and any subequent libvirt guests.

    This commit puts a try/except AttributeError block around all the code
    that is fetching attributes on the result of metadata_xml.find(...).

    also pull in I41aa76cf9def3e8c4bceef0280d15c1fd7c48e3d

    Change-Id: I8adc609cc21c86de2daba326d24b73a80d6eb61f
    Closes-Bug: #1749960
    (cherry picked from commit d2af7e9ece2f903633bf4e95208b2e3ba6ac464d)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/queens)

Reviewed: https://review.openstack.org/545403
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=24caac82528be7678165bf12fb5b997852727ecd
Submitter: Zuul
Branch: stable/queens

commit 24caac82528be7678165bf12fb5b997852727ecd
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Feb 16 08:58:06 2018 -0500

    Gracefully handle missing metadata in libvirt xml

    Missing metadata in the libvirt domain xml for a nova instance would
    cause ceilometer-compute to abort, leading to missing metrics for the
    current and any subequent libvirt guests.

    This commit puts a try/except AttributeError block around all the code
    that is fetching attributes on the result of metadata_xml.find(...).

    also pull in I41aa76cf9def3e8c4bceef0280d15c1fd7c48e3d

    Change-Id: I8adc609cc21c86de2daba326d24b73a80d6eb61f
    Closes-Bug: #1749960
    (cherry picked from commit d2af7e9ece2f903633bf4e95208b2e3ba6ac464d)

tags: added: in-stable-queens
tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/ocata)

Reviewed: https://review.openstack.org/547441
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=afaa7f7c8cbe34b1573c7ade341a7c42b455c28e
Submitter: Zuul
Branch: stable/ocata

commit afaa7f7c8cbe34b1573c7ade341a7c42b455c28e
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Feb 16 08:58:06 2018 -0500

    Gracefully handle missing metadata in libvirt xml

    Missing metadata in the libvirt domain xml for a nova instance would
    cause ceilometer-compute to abort, leading to missing metrics for the
    current and any subequent libvirt guests.

    This commit puts a try/except AttributeError block around all the code
    that is fetching attributes on the result of metadata_xml.find(...).

    also pull in I41aa76cf9def3e8c4bceef0280d15c1fd7c48e3d

    Change-Id: I8adc609cc21c86de2daba326d24b73a80d6eb61f
    Closes-Bug: #1749960
    (cherry picked from commit d2af7e9ece2f903633bf4e95208b2e3ba6ac464d)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ceilometer 8.1.4

This issue was fixed in the openstack/ceilometer 8.1.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ceilometer 9.0.5

This issue was fixed in the openstack/ceilometer 9.0.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ceilometer 10.0.1

This issue was fixed in the openstack/ceilometer 10.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ceilometer 11.0.0

This issue was fixed in the openstack/ceilometer 11.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.