The time of resource-list is too long
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| Ceilometer |
Fix Released
|
Medium
|
gordon chung |
Bug Description
In my all-in-one Havana environment,and ceilometer storage configured as mysql.
When using"time ceilometer meter-list", 193 meters listed in 1.275s:
http://
When using "time ceilometer resource-list", 40 resource listed in 2m14.775s:
http://
I have searched around the process of listing resource,and I fond there are time-consuming traversal of list.
I think the process should be optimized
Changed in ceilometer: | |
assignee: | nobody → Liusheng (liusheng) |
description: | updated |
Changed in ceilometer: | |
status: | New → Triaged |
importance: | Undecided → Medium |
David Hill (david-hill-ubisoft) wrote : | #1 |
Al Bailey (albailey1974) wrote : | #2 |
I am seeing the same performance issues using postgres backend.
The horizon dashboard "Resource Usage" (admin/
In the ceilometer code to query the resource list, it populates the resource_links attribute with a second SQL query per resource.
ZhiQiang Fan (aji-zqfan) wrote : | #3 |
In my all-in-one havana 2013.2.2.dev on sles 11 sp3 with mongodb as backend storage
here is my output:
# time ceilometer meter-list | wc -l
168
real 0m0.554s
user 0m0.352s
sys 0m0.028s
# time ceilometer resource-list | wc -l
37
real 0m3.089s
user 0m0.300s
sys 0m0.040s
The average is 3.089/37=0.083486s
# time ceilometer sample-list -m instance | wc -l
3921
real 0m7.610s
user 0m3.380s
sys 0m0.052s
The average of sample list is 0.761/3921=0.00194s
I think the problem may be caused by improper config option or in the impl_sqlalchemy implementation
as you mentioned that: "I fond there are time-consuming traversal of list." can you provide more detail information
or give a link points to specific line of code?
Thanks
ZhiQiang Fan (aji-zqfan) wrote : | #4 |
yes, I know your output is based on mysql, I just provide a mongo output for comparison,
thanks
gordon chung (chungg) wrote : | #5 |
have you tried an icehouse build? we did change ceilometer resource-list to drop some code which pulled down performance (https:/
that said, at quick glance, i still think we can improve the query that exists for resource-list
Liusheng (liusheng) wrote : | #6 |
gordon chung ,thanks ,yes, I have noticed that. the change has improved the performance of resource-list.
but the time of resource-list is still too long. as you said,we can improve the query.
Eoghan Glynn (eglynn) wrote : | #7 |
Note the suggestion I've made here pertaining to the performance of the mongodb driver:
https:/
could also potentially be applied to the sqlalchemy driver.
gordon chung (chungg) wrote : | #8 |
i think i have a better query for sql - testing this now.
Fix proposed to branch: master
Review: https:/
Changed in ceilometer: | |
assignee: | Liusheng (liusheng) → gordon chung (chungg) |
status: | Triaged → In Progress |
gordon chung (chungg) wrote : | #10 |
i've uploaded a patch that should fix performance issue to make it a bit more bearable now.... i saw about a 60-70% reduction on my machine.
the resource-list query can actually return in seconds with over a million samples. the reason it takes so long is the api currently builds a list of url links for related meters to resource. this accounts for almost all the time. i think we should change the behaviour so by default it doesn't generate these links.
Liusheng (liusheng) wrote : | #11 |
thanks for the patch
Changed in ceilometer: | |
milestone: | none → icehouse-rc1 |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 13e2ebcfb027534
Author: Gordon Chung <email address hidden>
Date: Thu Mar 13 13:00:58 2014 -0400
improve performance of resource-list in sql
simplify the query to retrieve list of resources. it is much simpler
to query against raw Sample data then to join against Resource table.
also, add meter_links param to allow ability to disable link generation of
relate meters.
Change-Id: Ia4ceafc568c4a2
Closes-Bug: #1264434
Implements: blueprint big-data-sql
Changed in ceilometer: | |
status: | In Progress → Fix Committed |
Changed in ceilometer: | |
status: | Fix Committed → Fix Released |
Changed in ceilometer: | |
milestone: | icehouse-rc1 → 2014.1 |
Matt Lesko (mattlesko-nih) wrote : | #13 |
Was the comment #10, https:/
/usr/bin/time ceilometer resource-list
Error communicating with http://
600.84 real 0.39 user 0.16 sys
mysql> select count(*) from ceilometer.
+----------+
| count(*) |
+----------+
| 882 |
+----------+
1 row in set (0.00 sec)
gordon chung (chungg) wrote : | #14 |
@Matt
there were minor improvements made to icehouse. larger improvements were made in juno.
just to dig deeper, can you confirm which ceilometer build you are using... also, what version of mysql do you have?
Matt Lesko (mattlesko-nih) wrote : | #15 |
openstack-
openstack-
openstack-
openstack-
openstack-
openstack-
python-
RDO Icehouse I believe.
Mysql: mysql-server-
So the comments on #10 were fixed in a different commit(s)?
gordon chung (chungg) wrote : | #16 |
there were additional commits that were not included/backported to icehouse.
in icehouse, we had a fundamental design issue in our sql backend which made read and writes very slow. in juno we worked to simplify the sql backend. some performance values can be found here: https:/
i'm adding python-
Matt Lesko (mattlesko-nih) wrote : | #17 |
Is this one of the commits? https:/
If it's showing up in Juno that'll be great.
gordon chung (chungg) wrote : | #18 |
@Matt,
yes, the above patch will be in Juno and will probably give the biggest performance benefits.
We are experiencing the same issue with ceilometer and mysql backend.
# ceilometer resource-list /obfuscated_ url:8777/ The read operation timed out
nError communicating with https:/
We only have 899 meters.
Dave