Loading nested stacks into memory at the same time as the parent stack concentrates memory use into a single engine, and tends to result in a high-water-mark effect. Since Kilo, when we split operations on nested stacks so that they happen over RPC in different engine worker processes, we've been endeavouring to eliminate loading of nested stacks using the StackResource.nested() call, most notably in these patches:
https://review.openstack.org/#/c/384718/
https://review.openstack.org/#/c/383839/
which together reduced TripleO memory usage in the gate by >20%.[1]
However, there remain other places where nested() is still called - many hidden by the monstrosity that is grouputils. The following changes could further improve memory usage:
* StackResource._validate_nested_resources() should use the current template or something - anything - to ascertain the number of resources currently in the stack in preference to loading the nested stack to count them. AFAICT this will result in every nested stack being loaded into memory simultaneously for validation purposes whenever we do a stack update, unless the max_resources_per_stack config option is set to -1. (TripleO *does* set it to -1, but most installations do not.)
* InstanceGroup._replace() and ResourceGroup._replace() (including ResourceGroup._count_black_listed()) should use some other method of determining the current size when doing a rolling update.
* AutoscalingGroup, InstanceGroup, ResourceGroup, and ResourceChain should use outputs from the nested stack (added in Pike by https://review.openstack.org/#/c/475931/) to get attribute values from nested resources, and only fall back on the grouputils functions when the outputs don't exist (which would happen if the stack has not been updated since before Pike).
* ResourceChain should add outputs to the nested stack to get reference IDs for nested resources, and only fall back on the grouputils functions when the outputs don't exist.
* grouputils.get_members() should be reimplemented or replaced with something that does a resource list over RPC to get the list of non-failed child resources. (And get_size() should be reimplemented to just return len(get_members(...)).)
[1] http://lists.openstack.org/pipermail/openstack-dev/2017-January/109748.html
Fix proposed to branch: master
Review: https:/