Lost connection to MySQL server because process "p_mysqld" on controller had murdered by OOM killer

[Alexey. Kalashnikov]

Detailed bug description:
 Swarm test 'Deploy cluster with multiple services using local mirror' had failed with
"AssertionError: Failed 7 OSTF tests; should fail 0 tests. Names of failed tests" message.

OSTF failed cause p_mysqld process had murdered by OOM killer:
cut from ostf.log:
2016-08-01 02:28:23 ERROR (nose_storage_plugin) fuel_health.tests.smoke.test_neutron_actions.TestNeutron.test_check_neutron_objects_creation
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/smoke/test_neutron_actions.py", line 95, in test_check_neutron_objects_creation
    net_id=network['id'])
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 183, in verify
    " Please refer to OpenStack logs for more details.")
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Step 6 failed: Server can not be created. Please refer to OpenStack logs for more details.

cut from syslog:
<3>Aug 1 02:28:22 node-1 kernel: [ 3910.822371] Out of memory: Kill process 23759 (mysqld) score 39 or sacrifice child
<3>Aug 1 02:28:22 node-1 kernel: [ 3910.823967] Killed process 23759 (mysqld) total-vm:2548552kB, anon-rss:111952kB, file-rss:0kB
<27>Aug 1 02:28:29 node-1 ocf-mysql-wss: ERROR: p_mysqld: mysql_status(): MySQL is not running

Steps to reproduce:
  1. Revert snapshot 'prepare_slaves_5' with default set of mirrors
  2. Run 'fuel-mirror' to create mirror repositories
  3. Create cluster with many components to check as many
     packages in local mirrors have correct dependencies
  4. Run 'fuel-mirror' to replace cluster repositories
     with local mirrors
  5. Check that repositories are changed
  6. Deploy cluster
  7. Check running services with OSTF

Expected results:
 All steps passed

Actual result:
 OSTF tests failed:
http://paste.openstack.org/show/545089/

Description of the environment:

Parameters:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.multirole/13/parameters/

shotgun2 short-report
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-library9.0-9.0.0-1.mos8495.noarch
 rubygem-astute-9.0.0-1.mos753.noarch
 fuel-release-9.0.0-1.mos6349.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8748.noarch
 shotgun-9.0.0-1.mos90.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-openstack-metadata-9.0.0-1.mos8748.noarch
 python-packetary-9.0.0-1.mos142.noarch
 nailgun-mcagents-9.0.0-1.mos753.noarch
 fuel-utils-9.0.0-1.mos8495.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-misc-9.0.0-1.mos8495.noarch
 fuel-ostf-9.0.0-1.mos938.noarch
 fuel-notify-9.0.0-1.mos8495.noarch
 fuel-nailgun-9.0.0-1.mos8748.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-mirror-9.0.0-1.mos142.noarch
 fuel-migrate-9.0.0-1.mos8495.noarch

Additional information:

console output:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.multirole/13/console

logs:
https://drive.google.com/open?id=0B0EB6QSDWt2vTUx1dDJnVjh3dU0

[Oleksiy Molchanov (omolchanov)]

Please add more RAM for controllers.

[Tatyana Kuterina (tkuterina)]

Reproduced on 9.1 snapshot #93

Acceptance test: Deployment with 3 controllers, NeutronTUN, with Murano, Sahara and Ceilometer
Test group: mixed_components_murano_sahara_ceilometer

https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNT2NDX3NnSFJxRnc/view?usp=sharing

[Alexey. Kalashnikov (akalashnikov)]

Again mysql had killed by oom killer, this time on acceptance run:
https://mirantis.testrail.com/index.php?/tests/view/14695139

I think, we supposed to consider seriously possibility of increase amount of RAM on controllers, this might be helpful to preventing many floating fails on step "Run OSTF tests".

[Alexander Kurenyshev (akurenyshev)]

Oleksiy,
please tell us what amount of ram will be enough? For now there are 3GB on controllers for that suits

[Roman Podoliaka (rpodolyaka)]

This was also the problem with the recent run of 10.0 smoke: http://paste.openstack.org/show/581152/

https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.smoke_neutron/685/console

[Oleksiy Molchanov (omolchanov)]

Hi,

Sorry for long delay, was on a vacation. Let's try to increase RAM to 4G.