compute agent's discovery process tries too often when connection refused

Bug #1454399 reported by Chris Dent
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Invalid
Undecided
Unassigned

Bug Description

If the nova-api is unavailable for some reason, for example during an upgrade, when the ceilometer compute-agent attempts to query the nova-api to get a list of local instances it will do it multiple times[1], around 51 times in 200ms.

Since it is trying to get exactly the same data each of those 51 times, this seems a bit excessive. Maybe it should just fail once and back off for a while on that particular error?

[1] A sample grenade run showing ceilometer-acompute log during the time before nova-api has started:

full log:
http://logs.openstack.org/72/182372/1/check/check-grenade-dsvm/1c95729/logs/new/screen-ceilometer-acompute.txt.gz

just errors:
http://logs.openstack.org/72/182372/1/check/check-grenade-dsvm/1c95729/logs/new/screen-ceilometer-acompute.txt.gz?level=ERROR

Changed in ceilometer:
assignee: nobody → Rohit Jaiswal (rohit-jaiswal-3)
Revision history for this message
Luo Gangyi (luogangyi) wrote :

I tried in my devstack enviroment(master) , and did't see that problem. Which version did you use?

Revision history for this message
Rohit Jaiswal (rohit-jaiswal-3) wrote :

This is because there are pollsters for as many supported meters - cpu, memory, disk and network - each having a couple of supported meters. Each pollster polls the underlying resource using nova client to get further details. Pollsters are configured to trigger at set intervals, when they fire off in quick succession. Not sure if there is a way to delay triggering a pollster if previous one fails on the same resource.

Revision history for this message
Qichen Hao (adsl123gg2008) wrote :

I can't download the log file,can you give us you erros log in comment?

ZhiQiang Fan (aji-zqfan)
Changed in ceilometer:
assignee: Rohit Jaiswal (rohit-jaiswal-3) → nobody
Revision history for this message
gordon chung (chungg) wrote :

we don't query api anymore

Changed in ceilometer:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.