Fuel-stats analytics consumes all memory on the servers

[Alexander Kislitsky]

On large amount of data analytics consumes all memory and uWSGI process killed by Segmantation Fault.

Strace output:
mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)

Steps to reproduce:
- generate about 20000 different installation_info
- generate about 110000 oswl_stats for installation_infos
- generate about 1000000 action_logs for installation_infos
- set 2Gb memory in vm
- run analytics service
- try to fetch 'all' CSV report

Expected:
Report is downloaded

Actual:
500 HTTP error

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/301127

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/301127
Committed: https://git.openstack.org/cgit/openstack/fuel-stats/commit/?id=ebe8d4eb46d58656ead0f79e0256bf70fde0bccf
Submitter: Jenkins
Branch: master

commit ebe8d4eb46d58656ead0f79e0256bf70fde0bccf
Author: Alexander Kislitsky <email address hidden>
Date: Mon Apr 4 15:58:27 2016 +0300

    Memory consumption reduced in fuel-stats analytics

    We had huge memory consumption on fetching OSWLs due to fetching
    clusters data in each OSWL row.

    Joined clusters data removed from each OSWL row. Explicit
    fetching of version info from clusters is introduced to CSV
    exporter.

    We are using SQLAlchemy yield_per for fetching large query results.
    Yeld_per page parameter reduced from 1000 to 100.

    We still have issues with memory consumption in all
    reports generation. Thus we are temporary removing link
    on 'All reports' from web UI.

    Example uWSGI configuration changed to start http service.

    Change-Id: I66c97058c0f15f8a15b626ab1fa222106754a455
    Partial-Bug: #1564427

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/303371

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/303371
Committed: https://git.openstack.org/cgit/openstack/fuel-stats/commit/?id=e200196347d5860bbc510c43598e5dad0831d82c
Submitter: Jenkins
Branch: master

commit e200196347d5860bbc510c43598e5dad0831d82c
Author: Alexander Kislitsky <email address hidden>
Date: Fri Apr 8 14:57:14 2016 +0300

    Connection hanging issue fixed

    On huge reports data we can have 500 HTTP error by timeout limitation.
    In case of streaming data from the DB cursor we had hanged
    DB operation inside the open transaction. For fix the issue releasing of
    SQLAlchemy session was added to collector and analytics apps.

    Additional changes:

    - We stop to migrate action logs data to Elasticsearch due to
      no one UI report uses this data.
    - Parameters fixed in OSWLs 'export' call in 'all_reports'.

    Change-Id: I009081a2304b36fb4ab3705b6fa09a8a3df5710b
    Closes-Bug: #1564427