Nova compute *.percent metrics are always 0

Bug #1488696 reported by Joe Cropper
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Joe Cropper

Bug Description

It seems that with the recent update to the Nova virt metrics framework in Liberty, the 'percent' related metrics always 0 now in the database. After tracing through the code, I determined tha the virt_driver and resource tracker will still behaving properly and the compute_driver::get_host_cpu_stats work as expected.

It seems that the root cause is:

https://github.com/openstack/nova/blob/master/nova/objects/monitor_metric.py#L29

This shows that the metric value is expected to be an integer in nova object, but the percentage metrics are all floating points and range in value from [0, 1] -- e.g., so 17.5% has been historically represented as 0.175 using the monitor framework. This causes the percentage values to assume the value 0, as shown below in the snippet from `select metrics from compute_nodes`:

"cpu.user.percent", "value": 0,
"cpu.percent", "value": 0,

...so on and so forth. By the time the metrics get to this spot in the resource tracker:

https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L366

...the '*.percent' values are all 0.

I'm not sure if the intended behavior here was to only support integer-style values. If so, we probably need to do some "multiply by 100" logic when putting them into the MonitorMetric object and then divide by 100 (we'll lose precision, though) when we convert back to the values stored in the compute_nodes.metrics column, otherwise we will break backwards compatibility in terms of what folks were expecting to find in the DB.

Joe Cropper (jwcroppe)
description: updated
description: updated
Joe Cropper (jwcroppe)
summary: - Compute *.percent metrics are always 0
+ Compute *.percent Nova metrics are always 0
summary: - Compute *.percent Nova metrics are always 0
+ Nova compute *.percent metrics are always 0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/216923

Changed in nova:
assignee: nobody → Joe Cropper (jwcroppe)
status: New → In Progress
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/216923
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=87e1d8fab2b5888e711f07001025600fe709aaa7
Submitter: Jenkins
Branch: master

commit 87e1d8fab2b5888e711f07001025600fe709aaa7
Author: Joe Cropper <email address hidden>
Date: Tue Aug 25 20:53:55 2015 -0500

    Convert percent metrics back into the [0, 1] range

    With the recent revamp of the Nova metrics framework in the Liberty
    cycle, there was a problem introduced in terms of how the percentage-
    based metrics (e.g., cpu.percent) are represented. That is, prior to
    Liberty, we stored the percentages in compute_nodes.metrics as a
    floating point value in the range [0, 1]. For example, 17.5% was
    represented as 0.175.

    The revamped framework expects the MonitorMetric.value properties to
    be integers, so when the percentages such as 0.175 are stored in this
    object, they're erroneously converted to 0.

    This patch multiplies the 'problematic' fields by 100 in preparation
    for storage within the MonitorMetric object and then divides the
    same problematic fields by 100.0 when converting back to their JSON
    format, which will preserve RPC notifier and database compatibility.

    Added some unit tests to test the conversions as well as making some
    of the unit tests more explicit so this doesn't happen again.

    Change-Id: I17a9412c63eeeb78ba2b2600f9007e9b58fcbf68
    Closes-Bug: #1488696

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-3 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.