AttributeError may be raised if virt backend is libvirt in some case

Bug #1393415 reported by ZhiQiang Fan
30
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Low
ZhiQiang Fan

Bug Description

2014-11-17 16:04:01.563 5162 INFO ceilometer.agent [-] Polling pollster memory.usage in the context of meter_source
140000 2014-11-17 16:04:01.564 5162 DEBUG ceilometer.compute.pollsters.memory [-] Checking memory usage for instance 7e53172c-f05f-4fda-9855-af6775c1f4a8 get_samples /opt/stack/ceilometer/ceilometer/compute/pollsters/memory.py:31
140002 2014-11-17 16:04:01.573 5162 WARNING ceilometer.compute.virt.libvirt.inspector [-] Failed to inspect memory usage of instance-00000002, can not get info from libvirt
140003 2014-11-17 16:04:01.574 5162 ERROR ceilometer.compute.pollsters.memory [-] Could not get Memory Usage for 7e53172c-f05f-4fda-9855-af6775c1f4a8: 'NoneType' object has no attribute 'usage'
140004 2014-11-17 16:04:01.574 5162 TRACE ceilometer.compute.pollsters.memory Traceback (most recent call last):
140005 2014-11-17 16:04:01.574 5162 TRACE ceilometer.compute.pollsters.memory File "/opt/stack/ceilometer/ceilometer/compute/pollsters/memory.py", line 37, in get_samples
140006 2014-11-17 16:04:01.574 5162 TRACE ceilometer.compute.pollsters.memory 'usage': memory_info.usage}))
140007 2014-11-17 16:04:01.574 5162 TRACE ceilometer.compute.pollsters.memory AttributeError: 'NoneType' object has no attribute 'usage'

the libvirt will return None if it cannot get memory usage, we should add some check in memory pollster, if return is None, just skip the pollster, try to log with memory.usage will definitely raise unnecessary error which will bother cloud operator.

Tags: logging ops
ZhiQiang Fan (aji-zqfan)
Changed in ceilometer:
assignee: nobody → ZhiQiang Fan (aji-zqfan)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/134966

Changed in ceilometer:
status: New → In Progress
Rochelle Grober (rockyg)
tags: added: loging
tags: added: logging
removed: loging
tags: added: ops
gordon chung (chungg)
Changed in ceilometer:
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/134966
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=98b43aad172c894d3c4e288e40eaa3c4f822d47f
Submitter: Jenkins
Branch: master

commit 98b43aad172c894d3c4e288e40eaa3c4f822d47f
Author: ZhiQiang Fan <email address hidden>
Date: Mon Nov 17 23:14:03 2014 +0800

    Return a meaningful value or raise an excpetion for libvirt

    Currently, there are some cases libvirt cannot inspect instance's memory
    usage, and it is a normal behavior so libvirt just logs messages and
    returns None. However, memory pollster doesn't check return value, so
    AttributeError will be raised in such case, which leading to unnecessary
    exception log messages in every cycle and for each instance, this will
    bother cloud operator.

    This patch uses virt.inspector exceptions for libvirt, so when there is no
    valid value can be returned, we raise an exception, pollsters can catch
    those exceptions and log proper messages. Since all of them are non-fatal
    exception, log level is set to warn, which is excatly same as previous.

    Note, this patch refactors some code of libvirt to suit this change.

    Change-Id: I2c94aab90e827c75a602403d3c64fd6f67f73007
    Closes-Bug: #1393415

Changed in ceilometer:
status: In Progress → Fix Committed
Eoghan Glynn (eglynn)
Changed in ceilometer:
milestone: none → kilo-2
Thierry Carrez (ttx)
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ceilometer:
milestone: kilo-2 → 2015.1.0
Revision history for this message
Shinobu Kinjo (shinobu-kj) wrote :

I'm seeing same type of __warning__ like:

    2015-10-12 13:06:21.100 2189 WARNING ceilometer.compute.pollsters.memory [-]
   Cannot inspect data of MemoryUsagePollster for 41879320-053a-4742-a3f8-e03e707477ad,
   non-fatal reason: Failed to inspect memory usage of instance-0000003d,
   can not get info from libvirt

But it's only related to windows instance.
Is that same one or different?

Let me know your thought.

Revision history for this message
ZhiQiang Fan (aji-zqfan) wrote :

This bug (and its fix) only focus at the unexpected attribute error, what your log shows is what we expect to see.

If you do want to know why libvirt cannot get info for that instance, some investigation is needed, that is another story

Revision history for this message
Shinobu KINJO (shinobu) wrote :

@aji-zqfan,

Thank you for pointing it out to me.
I'm now clear.

Rgds,
Shinobu

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ceilometer (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/335886

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ceilometer (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/335900

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ceilometer (stable/mitaka)

Change abandoned by Jens Rosenboom (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/335886
Reason: Will repropose iff master patch got merged.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ceilometer (master)

Change abandoned by Jens Rosenboom (<email address hidden>) on branch: master
Review: https://review.openstack.org/335900

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers