Memory use is quadratic in the number of stacks

[Zane Bitter]

Since the change to deal with nested stacks over RPC rather than work with the whole tree of stacks in-memory, we are using enormous amounts of memory.

This is partly because the Stack.root_stack property works by loading the parent *resource* from the parent stack - meaning the lazily-loaded resources are loaded even though we don't need them for the purposes of getting the parent stack.

That is irrelevant, however, because we then call total_resources() on the root stack, and that loads every single stack in the tree and all of their resources in order to count the total number resources. Thus, memory use is O(n^2) in the size of the tree :(

[Zane Bitter (zaneb)]

Fun fact: the total_resources() function was introduced (for bug 1215100) to prevent users from exhausting memory.

[Steve Baker (steve-stevebaker)]

It looks like this could be reasonably well optimised by changing root_stack and total_resources to use db queries instead of stack/resource navigation.

In engine.Stack, implement root_stack_id which queries for models.Stack.owner_id until the root is reached (bonus points for doing it in a single query)

change engine.Stack.root_stack to use root_stack_id, and deprecate it

change total_resources to do queries which build a list of stack IDs by recursively searching for owner_id. total resources is the count of models.Resource where stack_id in all stack IDs

I *might* have a crack at this on the plane. If I do I'll assign it to me

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/185153

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/185153
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=71a1e261401976901d60411207559f4f3e228def
Submitter: Jenkins
Branch: master

commit 71a1e261401976901d60411207559f4f3e228def
Author: Steve Baker <email address hidden>
Date: Tue May 19 10:11:25 2015 -0700

    Find root stack ID with database operations

    This change adds a (currently unused) database function
    stack_get_root_id to find the root stack ID for any stack.

    Scheduler hints are moved to using Stack.root_stack_id() in this change.
    Remaining uses of Stack.root_stack() will switch to using
    Stack.root_stack_id() later in the series.

    Change-Id: I9914b0df5af119edea6346db0d7c62124fbb1313
    Partial-Bug: #1455589

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/185894

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/185154
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=be41d08a44c6769743241515e8afb16e09077422
Submitter: Jenkins
Branch: master

commit be41d08a44c6769743241515e8afb16e09077422
Author: Steve Baker <email address hidden>
Date: Fri May 22 10:51:52 2015 -0700

    Count all nested stack resources with DB operations

    The count is performed in 2 parts
    - A function which builds a list of all nested stacks
      by recursively calling stack_get_all_by_owner_id
    - A query which counts resources which belong to the list
      of stacks

    Considering this will be the basis for fixing bug #1455589 then this
    approach is appropriate for backporting to stable/kilo, but master would
    ideally replace this soon with a new Stack attribute that stores the
    calculated resource count.

    Partial-Bug: #1455589

    Change-Id: Ifa2e5609fd9a6853e20037e94a5e16696bb378ee

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/185155
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ca32acbecea28f3674ac59738d74966ed406f465
Submitter: Jenkins
Branch: master

commit ca32acbecea28f3674ac59738d74966ed406f465
Author: Steve Baker <email address hidden>
Date: Fri May 22 14:39:54 2015 -0700

    Switch total_resources to use stack_count_total_resources

    This change uses database queries to count the total number of
    resources in a nested stack tree instead of loading every stack
    and every resource in that stack.

    This should result in a significant improvement in memory use
    for stacks with many nested stacks and resources.

    Closes-Bug: #1455589
    Change-Id: I8c2f1fa9114dbf68ad4a5f99dd3929b6958b5d7a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/185894
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=601a45d2c9c8d812c841bd47d3abf4a206467fd9
Submitter: Jenkins
Branch: master

commit 601a45d2c9c8d812c841bd47d3abf4a206467fd9
Author: Michal Jastrzebski (inc0) <email address hidden>
Date: Wed May 27 03:07:44 2015 +0200

    Unlimited option max_resources_per_stack

    Since biggest issue for memory problems is nested stack max
    resources validation, this patch creates option to remove limit
    which skips whole validation whatsoever.

    Change-Id: I7c851fcad2d4a41b2f9df42244862b04304812b0
    Partial-Bug: #1455589

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/192464

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/192465

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/192466

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/192464

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/192465

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/192466

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/192468

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/192469

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/192470

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/192468
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=3a7e7cbd3e09d8144752bbf431a41b0635f2be11
Submitter: Jenkins
Branch: stable/kilo

commit 3a7e7cbd3e09d8144752bbf431a41b0635f2be11
Author: Steve Baker <email address hidden>
Date: Tue May 19 10:11:25 2015 -0700

    Find root stack ID with database operations

    This change adds a (currently unused) database function
    stack_get_root_id to find the root stack ID for any stack.

    Scheduler hints are moved to using Stack.root_stack_id() in this change.
    Remaining uses of Stack.root_stack() will switch to using
    Stack.root_stack_id() later in the series.

    Cherry-picked from heat 71a1e261401976901d60411207559f4f3e228def
    Partial-Bug: #1455589

    Change-Id: I9914b0df5af119edea6346db0d7c62124fbb1313

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/192465

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/192469

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/192470

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/192465
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=99f95f0e9c57a295c0f03ab539117d71fb5b45aa
Submitter: Jenkins
Branch: stable/kilo

commit 99f95f0e9c57a295c0f03ab539117d71fb5b45aa
Author: Steve Baker <email address hidden>
Date: Fri May 22 10:51:52 2015 -0700

    Count all nested stack resources with DB operations

    The count is performed in 2 parts
    - A function which builds a list of all nested stacks
      by recursively calling stack_get_all_by_owner_id
    - A query which counts resources which belong to the list
      of stacks

    Considering this will be the basis for fixing bug #1455589 then this
    approach is appropriate for backporting to stable/kilo, but master would
    ideally replace this soon with a new Stack attribute that stores the
    calculated resource count.

    Partial-Bug: #1455589

    Change-Id: Ifa2e5609fd9a6853e20037e94a5e16696bb378ee

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/218193

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/192466
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=a7484244eb2a3f8110b80a279ad656cc3ba8d445
Submitter: Jenkins
Branch: stable/kilo

commit a7484244eb2a3f8110b80a279ad656cc3ba8d445
Author: Steve Baker <email address hidden>
Date: Fri May 22 14:39:54 2015 -0700

    Switch total_resources to use stack_count_total_resources

    This change uses database queries to count the total number of
    resources in a nested stack tree instead of loading every stack
    and every resource in that stack.

    This should result in a significant improvement in memory use
    for stacks with many nested stacks and resources.

    Cherry-picked from heat ca32acbecea28f3674ac59738d74966ed406f465
    Closes-Bug: #1455589
    Change-Id: I8c2f1fa9114dbf68ad4a5f99dd3929b6958b5d7a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/218193
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9258321f2925209e296ed18443e710cf28ed8f3a
Submitter: Jenkins
Branch: stable/kilo

commit 9258321f2925209e296ed18443e710cf28ed8f3a
Author: Michal Jastrzebski (inc0) <email address hidden>
Date: Wed May 27 03:07:44 2015 +0200

    Unlimited option max_resources_per_stack

    Since biggest issue for memory problems is nested stack max
    resources validation, this patch creates option to remove limit
    which skips whole validation whatsoever.

    Conflicts:
     heat/engine/resources/stack_resource.py
     heat/tests/engine/test_stack_create.py

    Cherry-picked from heat: 601a45d2c9c8d812c841bd47d3abf4a206467fd9
    Related-Bug: #1455589
    Change-Id: I7c851fcad2d4a41b2f9df42244862b04304812b0

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/juno
Review: https://review.openstack.org/229227

[Zane Bitter (zaneb)]

FWIW Juno was not affected by the quadraticness thing.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steve Baker (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/229227
Reason: Given that in Juno the whole stack is already in memory and counting all resources doesn't cross RPC boundaries there is no particular benefit to backporting this change.