Missing attributes in libvirt xml metadata cause ceilometer-compute to stop polling libvirt guests

[Lars Kellogg-Stedman]

We have discovered there are some nova instances in our (pike) environment that have empty "<nova:owner>" attributes in the metadata embedded in the libvirt XML. For example:

  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="14.0.3-9.el7ost"/>
      <nova:name>wm-solr</nova:name>
      <nova:creationTime>2017-05-24 15:29:24</nova:creationTime>
      <nova:flavor name="m1.large">
        <nova:memory>8192</nova:memory>
        <nova:disk>80</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>4</nova:vcpus>
      </nova:flavor>
      <nova:owner/>
      <nova:root type="image" uuid="a332ad63-8b38-4700-b818-b39fa69233a9"/>
    </nova:instance>
  </metadata>

In ceilometer/compute/discovery.py, ceilometer is asking for the owner/user and owner/project attributes with no error checking. This results in an AttributeError exception, which aborts the entire discover_libvirt_polling method and means that any remaining libvirt guests on the system will not be polled. This results in many missing metrics.

Following the theory that we should "be conservative in what you do, be liberal in what we accept from others", it seems as if it would be reasonable to catch these exceptions, report an error, and continue polling other guests on the system.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/545303

[gordon chung (chungg)]

is this also a bug in Nova? i imagine not having user/project details will making actually querying for this data difficult.

[Lars Kellogg-Stedman (larsks)]

This is also a bug in Nova, yes. There is some suggestion that https://review.openstack.org/#/c/399679/ fixed this in Nova.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545303
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=d2af7e9ece2f903633bf4e95208b2e3ba6ac464d
Submitter: Zuul
Branch: master

commit d2af7e9ece2f903633bf4e95208b2e3ba6ac464d
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Feb 16 08:58:06 2018 -0500

    Gracefully handle missing metadata in libvirt xml

    Missing metadata in the libvirt domain xml for a nova instance would
    cause ceilometer-compute to abort, leading to missing metrics for the
    current and any subequent libvirt guests.

    This commit puts a try/except AttributeError block around all the code
    that is fetching attributes on the result of metadata_xml.find(...).

    Change-Id: I8adc609cc21c86de2daba326d24b73a80d6eb61f
    Closes-Bug: #1749960

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/545376

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/545403

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/545504

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lars Kellogg-Stedman (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/545376
Reason: This patch was missing a 'continue'.

[OpenStack Infra (hudson-openstack)]

Change abandoned by gordon chung (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/545403

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545504
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=050b5293b3fa3b01a84b3e79e84f5ddb7fba0872
Submitter: Zuul
Branch: master

commit 050b5293b3fa3b01a84b3e79e84f5ddb7fba0872
Author: Lars Kellogg-Stedman <email address hidden>
Date: Sat Feb 17 00:01:01 2018 -0500

    the previous patch was missing a 'continue'

    This commit puts a try/except AttributeError block around all the code
    that is fetching attributes on the result of metadata_xml.find(...).

    Change-Id: I41aa76cf9def3e8c4bceef0280d15c1fd7c48e3d
    Closes-Bug: #1749960

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/547441

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545376
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=48515b4841c462ead3d04928801304b093d7594d
Submitter: Zuul
Branch: stable/pike

commit 48515b4841c462ead3d04928801304b093d7594d
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Feb 16 08:58:06 2018 -0500

    Gracefully handle missing metadata in libvirt xml

    Missing metadata in the libvirt domain xml for a nova instance would
    cause ceilometer-compute to abort, leading to missing metrics for the
    current and any subequent libvirt guests.

    This commit puts a try/except AttributeError block around all the code
    that is fetching attributes on the result of metadata_xml.find(...).

    also pull in I41aa76cf9def3e8c4bceef0280d15c1fd7c48e3d

    Change-Id: I8adc609cc21c86de2daba326d24b73a80d6eb61f
    Closes-Bug: #1749960
    (cherry picked from commit d2af7e9ece2f903633bf4e95208b2e3ba6ac464d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545403
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=24caac82528be7678165bf12fb5b997852727ecd
Submitter: Zuul
Branch: stable/queens

commit 24caac82528be7678165bf12fb5b997852727ecd
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Feb 16 08:58:06 2018 -0500

    Gracefully handle missing metadata in libvirt xml

    Missing metadata in the libvirt domain xml for a nova instance would
    cause ceilometer-compute to abort, leading to missing metrics for the
    current and any subequent libvirt guests.

    This commit puts a try/except AttributeError block around all the code
    that is fetching attributes on the result of metadata_xml.find(...).

    also pull in I41aa76cf9def3e8c4bceef0280d15c1fd7c48e3d

    Change-Id: I8adc609cc21c86de2daba326d24b73a80d6eb61f
    Closes-Bug: #1749960
    (cherry picked from commit d2af7e9ece2f903633bf4e95208b2e3ba6ac464d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/547441
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=afaa7f7c8cbe34b1573c7ade341a7c42b455c28e
Submitter: Zuul
Branch: stable/ocata

commit afaa7f7c8cbe34b1573c7ade341a7c42b455c28e
Author: Lars Kellogg-Stedman <email address hidden>
Date: Fri Feb 16 08:58:06 2018 -0500

    Gracefully handle missing metadata in libvirt xml

    Missing metadata in the libvirt domain xml for a nova instance would
    cause ceilometer-compute to abort, leading to missing metrics for the
    current and any subequent libvirt guests.

    This commit puts a try/except AttributeError block around all the code
    that is fetching attributes on the result of metadata_xml.find(...).

    also pull in I41aa76cf9def3e8c4bceef0280d15c1fd7c48e3d

    Change-Id: I8adc609cc21c86de2daba326d24b73a80d6eb61f
    Closes-Bug: #1749960
    (cherry picked from commit d2af7e9ece2f903633bf4e95208b2e3ba6ac464d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ceilometer 8.1.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ceilometer 9.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ceilometer 10.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ceilometer 11.0.0 release.