Unit tests periodically oom

[Sean Dague]

For the following logstash query - http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Killed%20process%5C%22%20AND%20message%3A%5C%22(python)%5C%22

unit tests are periodically getting hit by the Out of Memory killer in Linux, indicating that they are running out of memory on these boxes. This is probably do to some kind of memory leak in the unit tests of something getting created during setup that is not fully torn down, or something getting created in global state during a test which is not reaped.

An example failure - http://logs.openstack.org/33/352533/10/gate/gate-cinder-python34-db/1fc4184/console.html

[Scott DAngelo (scott-dangelo)]

I've reproduced this using:
Ubuntu 14.04
VM has 2GB Ram and 1CPU

and running Cinder unit tests:
./runtests

[Scott DAngelo (scott-dangelo)]

Some more info:
Failure is consistent for py34. I've had successful runs and failure on py27.
Failure doesn't occur (for my limited tests runs) without devstack running.
6/7 failures during VolumeMigrationTestCase....

scott@dev2gb:~/tests$ grep inprogress *
Failure_1_23:{0} cinder.tests.unit.test_volume.VolumeMigrationTestCase.test_migrate_volume_generic_timeout_error [] ... inprogress
Failure_1_44:{0} cinder.tests.unit.test_volume.VolumeTestCase.test_attach_detach_not_multiattach_volume_for_instances [] ... inprogress
Failure_2_08:{0} cinder.tests.unit.test_volume.VolumeMigrationTestCase.test_retype_volume_driver_success [] ... inprogress
Failure_2_26:{0} cinder.tests.unit.test_volume.VolumeMigrationTestCase.test_migrate_volume_generic_migrate_volume_completion_error [] ... inprogress
Failure_2_41:{0} cinder.tests.unit.test_volume.VolumeMigrationTestCase.test_migrate_volume_generic_timeout_error [] ... inprogress
Failure_2_50:{0} cinder.tests.unit.test_volume.VolumeMigrationTestCase.test_retype_volume_migration_bad_policy [] ... inprogress
Failure_4_03:{0} cinder.tests.unit.test_volume.VolumeTestCase.test_begin_detaching_fails_available [] ... inprogress

[Eric Harney (eharney)]

Results of running unit tests with a small memory ulimit:

http://paste.openstack.org/raw/563924/

[Scott DAngelo (scott-dangelo)]

Disregard that INFO ^^ in Comment #4. That test was invalid. The oom_killer still kicks in with zonemanager removed.

[Matt Riedemann (mriedem)]

We aren't seeing this in the gate anymore, so is something fixed?