ceilometer-agent-ipmi fails on unparseable data

Bug #1381600 reported by Chris Dent
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
High
Chris Dent
Juno
Fix Released
High
Eoghan Glynn

Bug Description

When the ceilometer-agent-ipmi calls out to ipmitool to get data it then parses the results into chunks. Each chunk is them parsed to extract a sample.

The intention is that any chunk which cannot be parsed is dropped and parsing continues. The code at https://github.com/openstack/ceilometer/blob/0820c95f0d335e978a92f5b65848d51623e4df78/ceilometer/ipmi/pollsters/sensor.py#L65 can throw an exception which needs to be trapped and a `continue` called.

A fix is in progress.

Given the comlexity of this chunk of code it would be useful to be able to test this in a complete environment (with local IPMI support). Existing manual testing has involved some hacks to get IPMI data from another machine.

Chris Dent (cdent)
Changed in ceilometer:
assignee: nobody → Chris Dent (chdent)
Eoghan Glynn (eglynn)
Changed in ceilometer:
importance: Undecided → High
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/128694

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/128694
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=8323cfe92af2ad27f6611be0a6887cac12b2c630
Submitter: Jenkins
Branch: master

commit 8323cfe92af2ad27f6611be0a6887cac12b2c630
Author: Chris Dent <email address hidden>
Date: Wed Oct 15 17:18:17 2014 +0100

    Handle poorly formed individual sensor readings

    ipmitool returns a series of multiple readings per run. Some of
    these readings may be of a form that parser code cannot handle. The
    desired behavior in this case is for the single reading to be
    dropped and for parsing to continue. This change ensures that
    behavior by continuing through the loop instead of allowing
    exceptions to cause the loop to exit.

    Tests have been added which verify the correct behavior.

    Note that though there was an opportunity to log each skipped
    reading this would lead to a very large number of log messages as
    many or even most samples from ipmitool are not parsed.

    Change-Id: I6a3193d5a6e12c69ca5c548e5fb58d8bc9646348
    Closes-Bug: #1381600

Changed in ceilometer:
status: In Progress → Fix Committed
Eoghan Glynn (eglynn)
tags: added: juno-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/129564

Alan Pevec (apevec)
tags: removed: juno-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/juno)

Reviewed: https://review.openstack.org/129564
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=b1fca33e5f4c5443389d131a212a058d2306ac23
Submitter: Jenkins
Branch: stable/juno

commit b1fca33e5f4c5443389d131a212a058d2306ac23
Author: Chris Dent <email address hidden>
Date: Wed Oct 15 17:18:17 2014 +0100

    Handle poorly formed individual sensor readings

    ipmitool returns a series of multiple readings per run. Some of
    these readings may be of a form that parser code cannot handle. The
    desired behavior in this case is for the single reading to be
    dropped and for parsing to continue. This change ensures that
    behavior by continuing through the loop instead of allowing
    exceptions to cause the loop to exit.

    Tests have been added which verify the correct behavior.

    Note that though there was an opportunity to log each skipped
    reading this would lead to a very large number of log messages as
    many or even most samples from ipmitool are not parsed.

    Change-Id: I6a3193d5a6e12c69ca5c548e5fb58d8bc9646348
    Closes-Bug: #1381600
    (cherry picked from commit 8323cfe92af2ad27f6611be0a6887cac12b2c630)

Eoghan Glynn (eglynn)
Changed in ceilometer:
milestone: none → kilo-1
Thierry Carrez (ttx)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: kilo-1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.