Collector is slow on storing samples to backend database
Bug #1291923 reported by
Mitsuru Kanabuchi
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ceilometer |
Won't Fix
|
Medium
|
Unassigned |
Bug Description
It takes about 18 minutes in storing 20,000 samples (floating ips in our case) to the backend database.
Stable/havana, with MySQL backend DB is used.
The polling interval is 10 minutes by default, so next polling cycle will start before storing completes.
Therefore, problems such that queue is choked with messages (samples) and/or resources for collector (cpu, memory) increase, might occur.
The followings can be considered to be reasons of the problem.
- Collector stores samples one at a time (i.e., no bulk insert).
- Schema level/code level problems may exist, as pointed out in the following article.
http://
Changed in ceilometer: | |
importance: | Undecided → Low |
status: | New → Triaged |
Changed in ceilometer: | |
assignee: | Mitsuru Kanabuchi (kanabuchi) → Rikimaru Honjo (honjo-rikimaru-c6) |
Changed in ceilometer: | |
milestone: | none → juno-3 |
Changed in ceilometer: | |
milestone: | juno-3 → juno-rc1 |
Changed in ceilometer: | |
milestone: | juno-rc1 → none |
To post a comment you must log in.
I think this is not a duplicate of Bug #1291054: ceilometer service is working on a single thread.
#1291054 seems to target to enable multiple workers for notification agent and collector. The fix will have great effect in speeding up storing samples to backend database.
However, #1291054 seems to solve our problem only partly. There are cases that single central agent needs to handle a large amount of samples. For example, 20,000 samples of floating ips, acquired from one polling to neutron, are made into one message and put into the queue. Only one collector can get the message and store 20,000 samples. Therefore, we believe that we need to consider the DB store speed of a single collector.
Ideas for solutions are:
1. implement some bulk insert feature into central agent.
2. avoid forming large number of samples into single message - divide message to contain smaller number of samples for each, so that multiple workers can take effect.