Mysqld is killed by OOM-killer, because of lack of resources

[Nikita Karpin]

Detailed bug description:

During CI jobs [examples - 1, 2, 3] When running OSTF tests on successfully deployed environment, some of them are failed because mysqld is not running. From logs it is obvious that mysqld is killed by OOM killer:

<3>Sep 12 14:02:06 node-1 kernel: [ 2585.823435] Out of memory: Kill process 9339 (mysqld) score 44 or sacrifice child
<3>Sep 12 14:02:06 node-1 kernel: [ 2585.823518] Killed process 9339 (mysqld) total-vm:2547960kB, anon-rss:118464kB, file-rss:0kB

[1] https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4628

[2] https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.smoke_neutron/656/

[3] https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.smoke_neutron/652/

Steps to reproduce:

Can be reproduced during smoke_neutron job.

Expected results:

All ostf tests pass

Actual result:

Some ostf test failed

Description of the environment:

1 Controller + 2 Compute,Cinder

Operation system: Ubuntu Xenial

Versions of components: Fuel 10

[Nikita Karpin (mkarpin)]

same issue is seen here:

https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4678/
https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4683/
https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4681/
https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4679/

and looks like in many others builds

in all the cases, something like this can be seen in controller's kern.log:

<3>Sep 14 00:23:25 node-3 kernel: [ 2603.871961] Out of memory: Kill process 2156 (mysqld) score 43 or sacrifice child
<3>Sep 14 00:23:25 node-3 kernel: [ 2603.871965] Killed process 2122 (mysqld) total-vm:2547440kB, anon-rss:112436kB, file-rss:2444kB

also OSTF tests in all cases are failing like this:

 Create volume and boot instance from it success
 Launch instance, create snapshot, launch instance from snapshot failure
 Create instance flavor success
 Request volume list success
 Launch instance with file injection failure
 Request user list success
 Request image list using Glance v2 success
 Create security group failure
 Launch instance failure
 Request active services list success
 Check that required services are running success
 Create user and authenticate with it. failure
 Request image list using Glance v1 success
 Request snapshot list success
 Create keypair failure
 Request flavor list success
 Check create, update and delete image actions using Glance v2 success
 Request image list using Nova success
 Create volume and attach it to instance success
 Request instance list success
 Request stack list success
 Check network parameters failure
 Request list of networks success
 Check network connectivity from instance via floating IP failure
 Request absolute limits list success

This issue really blocks CI in upstream openstack-puppet project

[Nikita Karpin (mkarpin)]

the issue persists with iso 665 too
https://ci.fuel-infra.org/job/master.puppet-openstack.fuel-library.pkgs.ubuntu.smoke_neutron/4695/