Neutron test suite leaks memory like a sieve (still)

Bug #1353554 reported by Kevin L. Mitchell
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Expired
Undecided
Unassigned

Bug Description

Reported this originally as bug #1065276. When we try to run unit tests on neutron, we have to run them on an 8 core box, because the test suite will not pass successfully otherwise. One of our developers needed to add 100G of swap—and yes, that's gigabytes—just to get them to pass locally for him while he was trying to track down a problem.

Revision history for this message
Mithil Arun (arun-mithil) wrote :

I'm not sure about this. I run the neutron unit tests on a single core, 2GB RAM VM all the time, and it works fine for me.

Revision history for this message
Kevin Benton (kevinbenton) wrote :

I'm in the same boat as Mithil. I've never had a memory problem on my 4 GB VM.

How are you running the tests, and how do they fail?

Changed in neutron:
status: New → Incomplete
Revision history for this message
Jason Kölker (jason-koelker) wrote :

Last night running under tox combined the processes ballooned up to 7.8G of memory in use. Running under tox 1.6.1 and python 2.7. I suspect it has to do with the test runner not calling cleanup or some tests keeping references around after they "complete". I didn't look into it much as all I needed was the tests to pass to cut a package. But this is definitely still an issue from time to time.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote :

I'm seeing test failures in the gate because of running out of memory.

A logstash search for message:"oom-killer" shows many jobs with neutron processes being killed off.

Revision history for this message
Kevin L. Mitchell (klmitch) wrote :

Have we answered your question? Do you need any more data? Has anything been done to address the problem?

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
Revision history for this message
Kevin L. Mitchell (klmitch) wrote :

There has been no activity for 60 days because no one will tell us if they need any more information.

Changed in neutron:
status: Expired → New
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

Most of developers use VMs with quite low cpu/mem resources and don't experience problems.

Especially 100G swap looks strange.

Are you working on master? How do you run UTs?

Changed in neutron:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
Revision history for this message
Gandharva (gandharva-s) wrote :

I agree with Mithil and Kevin, its working fine for me on my single core 2gb ram machine and I am running it on master.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.