timeouts while benchmarking by rally

[Sergey Skripnick]

Rally job is failing with timeout during ceilometer benchmarks:

http://logs.openstack.org/60/211160/35/check/gate-rally-dsvm-rally/40248a3/rally-plot/results.html.gz

Seems this bug was introduced by this patch: https://review.openstack.org/#/c/218706/

Timeout error is gone if that patch reverted:

https://review.openstack.org/#/c/222243/
(direct link to results: http://logs.openstack.org/43/222243/1/check/gate-rally-dsvm-rally/813a5dd/rally-plot/results.html.gz)

[gordon chung (chungg)]

what event are you trying to query for? the basic idea is that you will only be returned events which are scoped to your project (if admin)

[Chris Dent (cdent)]

Sergey, one more piece of information (in addition to gordon's request) that might be useful in figuring out what the problem is here: How many events are already stored when the query that is timing out is made?

Gordon, how likely is it that the additional filtering is making the query slow down a significant amount?

[gordon chung (chungg)]

i don't think it's slowing it down. i think their querying on an event that is probably now restricted.

[Sergey Skripnick (eyerediskin)]

Here is part of mysql_slow.log:

(Query_time: 153.982554 Lock_time: 0.000407 Rows_sent: 1 Rows_examined: 1224979432)

http://dpaste.com/0FGKW8C

Here is explain:

http://dpaste.com/2S7DHAT

Here can be seen a lot of missing indexes. Also I believe this query may be optimized.

[Liusheng (liusheng)]

https://review.openstack.org/#/c/222991/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/222991
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=d5860a70720eecb7393f5b3183f898a9b14e0e00
Submitter: Jenkins
Branch: master

commit d5860a70720eecb7393f5b3183f898a9b14e0e00
Author: unknown <email address hidden>
Date: Mon Sep 14 15:16:52 2015 +0800

    Fix the heavy time cost of event-list

    Acorrding the current event-list API, an admin user is allowed to query
    all events which have traits.project_id value that match it's own project
    OR any event without any project_id trait. In get_events() in
    impl_sqlalchemy.py, this current query clause is implemented in a
    inefficient way. This change make it more efficient.

    Co-Authored-By: gordon chung<email address hidden>
    Change-Id: I645a7def0552e2574da540e180b4f57d8770e96e
    Closes-Bug: #1494440

[gordon chung (chungg)]

@Sergey, can we confirm the above patched resolved issue? thanks!