Very high memory usage for vitrage-graph process

[Idan Hefetz]

Testing performance with a graph of ~60,000 vertices and numerous parallel calls to topology show api.
Memory allocation, in vitrage-graph main process, keeps increasing and reaches ~10gb before we stopped it.

After some investigation, the fault is with the api-handler, returning huge dictionaries.
This also slows down the event processing as it is in the same process, traversing the same memory.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/566794

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/566794
Committed: https://git.openstack.org/cgit/openstack/vitrage/commit/?id=d485c6157dd38c316d6b46a73dc595b421dfc94b
Submitter: Zuul
Branch: stable/queens

commit d485c6157dd38c316d6b46a73dc595b421dfc94b
Author: Idan Hefetz <email address hidden>
Date: Tue May 8 06:59:41 2018 +0000

    handle severe memory problem with get_topology

    A few fixes in api_handler solved this issue
     - not calling json.dumps on the returned dict
     - force non parallel api execution, to avoid multiple concurrent allocations
     - call gc.collect as part of get_topology
     - api handler runs in different process
     - using generators to avoid creation of large collections
     - removing excessive debug logging

    Change-Id: Ied24d8c0b1d30149e45d63e95cf1b3b2afdf504f
    Closes-Bug: #1769826

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/vitrage 2.3.0 release.