Neutron test suite leaks memory like a sieve (still)

[Kevin L. Mitchell]

Reported this originally as bug #1065276. When we try to run unit tests on neutron, we have to run them on an 8 core box, because the test suite will not pass successfully otherwise. One of our developers needed to add 100G of swap—and yes, that's gigabytes—just to get them to pass locally for him while he was trying to track down a problem.

[Mithil Arun (arun-mithil)]

I'm not sure about this. I run the neutron unit tests on a single core, 2GB RAM VM all the time, and it works fine for me.

[Kevin Benton (kevinbenton)]

I'm in the same boat as Mithil. I've never had a memory problem on my 4 GB VM.

How are you running the tests, and how do they fail?

[Jason Kölker (jason-koelker)]

Last night running under tox combined the processes ballooned up to 7.8G of memory in use. Running under tox 1.6.1 and python 2.7. I suspect it has to do with the test runner not calling cleanup or some tests keeping references around after they "complete". I didn't look into it much as all I needed was the tests to pass to cut a package. But this is definitely still an issue from time to time.

[Doug Hellmann (doug-hellmann)]

I'm seeing test failures in the gate because of running out of memory.

A logstash search for message:"oom-killer" shows many jobs with neutron processes being killed off.

[Kevin L. Mitchell (klmitch)]

Have we answered your question? Do you need any more data? Has anything been done to address the problem?

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Kevin L. Mitchell (klmitch)]

There has been no activity for 60 days because no one will tell us if they need any more information.

[Eugene Nikanorov (enikanorov)]

Most of developers use VMs with quite low cpu/mem resources and don't experience problems.

Especially 100G swap looks strange.

Are you working on master? How do you run UTs?

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Gandharva (gandharva-s)]

I agree with Mithil and Kevin, its working fine for me on my single core 2gb ram machine and I am running it on master.