[ceilometer] Map-Reduce in resource-list command works long and blocks the db

[Ilya Tyaptin]

Currently for time request for resources we make a Map Reduce job on the all metering table (many GB in the longevity environments) that take a long time.

In the MongoDB 2.6+ we can avoid Map-Reduce and use find request instead. It will improve performance of this api request and doesn't block the database.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Ilya Tyaptin <email address hidden>
Review: https://review.fuel-infra.org/14893

[Nadya Privalova (nprivalova)]

Not merged in upstream. Plan to merge in upstream first and probably go to 9.0 and back port to 8.0

[Dmitry Sutyagin (dsutyagin)]

reproducible in MOS 7.0

Due to slow reply from /resources call with additional filter by timestamp (65 seconds in a specific case), it times out via haproxy, which causes relevant OSTF tests to fail

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Ilya Tyaptin <email address hidden>
Review: https://review.fuel-infra.org/21123

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0/mitaka
Change author: Ilya Tyaptin <email address hidden>
Review: https://review.fuel-infra.org/21129

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/21129
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 3d5380f87284da36308c81fabda61b8e76505ee2
Author: Ilya Tyaptin <email address hidden>
Date: Mon May 23 12:42:46 2016

Avoid Map-Reduce job during the get_resource

For avoiding map-reduce in the resource list requests
we add a atomic operations for update first_sample_timestamp and
last_sample_timestamp and make a find request on resource collection.

Time range in this query affects both last sample timestamp and
matches only timestamps which exist in requested period.

        +-----------------+
        | |
+------------------------------------+ R1
        | |
+-----------------+ | R2
        | |
        | +-------------+ R3
        | |
+-----+ | | R4
        | |
        | | +------+ R5
        | |
        +-----------------+

        Request timestamp range

Only resources R1, R2 and R3 will be returned.

Co-Authored-By: Igor Degtiarov <email address hidden>

Change-Id: I9f6787219e4ec04ab8eca11bcf4867d0f8becdc9
Closes-bug: #1528296
Cherry-picked from 92b61b8

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14893
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: 4d9271621211d8532022ab3d76c9a68e592b61b8
Author: Ilya Tyaptin <email address hidden>
Date: Fri May 20 12:55:54 2016

Avoid Map-Reduce job during the get_resource

For avoiding map-reduce in the resource list requests
we add a atomic operations for update first_sample_timestamp and
last_sample_timestamp and make a find request on resource collection.

Time range in this query affects both last sample timestamp and
matches only timestamps which exist in requested period.

        +-----------------+
        | |
+------------------------------------+ R1
        | |
+-----------------+ | R2
        | |
        | +-------------+ R3
        | |
+-----+ | | R4
        | |
        | | +------+ R5
        | |
        +-----------------+

        Request timestamp range

Only resources R1, R2 and R3 will be returned.

Co-Authored-By: Igor Degtiarov <email address hidden>

Change-Id: I9f6787219e4ec04ab8eca11bcf4867d0f8becdc9
Closes-bug: #1528296

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/21123
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 254106e8dafe9c5369522691de9d63c82a2da8aa
Author: Ilya Tyaptin <email address hidden>
Date: Tue May 31 14:55:19 2016

Avoid Map-Reduce job during the get_resource

For avoiding map-reduce in the resource list requests
we add a atomic operations for update first_sample_timestamp and
last_sample_timestamp and make a find request on resource collection.

Time range in this query affects both last sample timestamp and
matches only timestamps which exist in requested period.

        +-----------------+
        | |
+------------------------------------+ R1
        | |
+-----------------+ | R2
        | |
        | +-------------+ R3
        | |
+-----+ | | R4
        | |
        | | +------+ R5
        | |
        +-----------------+

        Request timestamp range

Only resources R1, R2 and R3 will be returned.

Co-Authored-By: Igor Degtiarov <email address hidden>

Change-Id: I9f6787219e4ec04ab8eca11bcf4867d0f8becdc9
Closes-bug: #1528296
Cherry-picked from 92b61b8

[Dmitry (dtsapikov)]

Verified on 7.0+mu4

Script
https://github.com/vitalygusev/ceilo-scripts/blob/master/mongo-generator.py

was used with parameters:

--users 10 --projects 10 --resources_per_user_project 10 --samples_per_user_project 10

Performance was checked using:

ceilometer resource-list -l 100 | wc -l
ceilometer resource-list -q

[Artem Minasyan (aminasyan)]

Verfied on ISO: http://paste.openstack.org/show/510473/

[Dmitry (dtsapikov)]

Verified on 8.0+mu2

[Nadya Privalova (nprivalova)]

I have added '10.0-reviewed' tag not to forget to include this into Newton-based product. We cannot merge this change to upstream because of Mongo version (2.6 cannot be supported in CI, but this change requires 2.6 at least).

[Nastya Urlapova (aurlapova)]

Since we don't support Ceilo in 10.0 issue was moved to Invalid.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/newton
Change author: Ilya Tyaptin <email address hidden>
Review: https://review.fuel-infra.org/33379

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 11.0/ocata
Change author: Ilya Tyaptin <email address hidden>
Review: https://review.fuel-infra.org/33777

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/ocata
Change author: Ilya Tyaptin <email address hidden>
Review: https://review.fuel-infra.org/34487

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ilya Tyaptin <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33379

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ilya Tyaptin <email address hidden> on branch: mcp/ocata
Review: https://review.fuel-infra.org/34487

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Roman Podoliaka <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/33777
Reason: we don't use 11.0/ocata anymore - mcp/ocata is the correct branch name