Comment 20 for bug 1626675

I ran the same analysis on the periodic job. The data is much more sparse, but it appears there is a jump between the 3rd and the 19th of August (before the RAM increase):

https://zaneb.fedorapeople.org/tripleo-memory/20160930/periodic_20160803-20160819_memused.png

(Note that this may understate the increase, which looks like ~0.75GiB in the flat part of the graph, because there seems to be a hard ceiling below 6GiB that like affects the latter jobs - presumably swap is enabled, but we don't track that.)

The two closest builds showing different behaviour are http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/05ecf21/ and http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/1b96976/ - they use the Heat builds 6c9f33d and 2f412de, respectively. Of course it's possible that changes in the test itself or changes in TripleO rather than Heat are responsible for the increase.

However, assuming that a problem lies between those two Heat commits, the most suspicious one appears to be https://review.openstack.org/#/c/323614/16 - always loading the raw_template from the database just to look up the name/id in identify_stack sounds like it could easily be hard on memory, and it's consistent with Steve's observation that memory use increases when you're hitting the API from the client (internally I don't think identify_stack is used that much), although none of us have been able to reproduce that. (It's possible even that eager loading in general might interact poorly with Crag's memory-saving fix that added the template_files table.)

The second-most suspicious commit is probably https://review.openstack.org/#/c/350287/ since it involves doing more work when creating a Software Config/Deployment, which is used extensively in TripleO but not in my test above. It doesn't appear to be doing anything obviously wrong, but then I would say that :)

The other patches all seem fairly benign to me, but I would welcome more eyeballs.