Mysqld is killed by OOM-killer, because of lack of resources
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
New
|
High
|
Fuel CI |
Bug Description
Detailed bug description:
During CI jobs [examples - 1, 2, 3] When running OSTF tests on successfully deployed environment, some of them are failed because mysqld is not running. From logs it is obvious that mysqld is killed by OOM killer:
<3>Sep 12 14:02:06 node-1 kernel: [ 2585.823435] Out of memory: Kill process 9339 (mysqld) score 44 or sacrifice child
<3>Sep 12 14:02:06 node-1 kernel: [ 2585.823518] Killed process 9339 (mysqld) total-vm:2547960kB, anon-rss:118464kB, file-rss:0kB
[1] https:/
[2] https:/
[3] https:/
Steps to reproduce:
Can be reproduced during smoke_neutron job.
Expected results:
All ostf tests pass
Actual result:
Some ostf test failed
Description of the environment:
1 Controller + 2 Compute,Cinder
Operation system: Ubuntu Xenial
Versions of components: Fuel 10
Changed in fuel: | |
status: | New → Confirmed |
importance: | Undecided → High |
assignee: | nobody → Fuel Sustaining (fuel-sustaining-team) |
milestone: | none → 10.0 |
Changed in fuel: | |
assignee: | Fuel Sustaining (fuel-sustaining-team) → Fuel CI (fuel-ci) |
tags: | added: area-ci |
Changed in fuel: | |
status: | Confirmed → New |
tags: | removed: area-ci |
same issue is seen here:
https:/ /ci.fuel- infra.org/ job/master. puppet- openstack. fuel-library. pkgs.ubuntu. smoke_neutron/ 4678/ /ci.fuel- infra.org/ job/master. puppet- openstack. fuel-library. pkgs.ubuntu. smoke_neutron/ 4683/ /ci.fuel- infra.org/ job/master. puppet- openstack. fuel-library. pkgs.ubuntu. smoke_neutron/ 4681/ /ci.fuel- infra.org/ job/master. puppet- openstack. fuel-library. pkgs.ubuntu. smoke_neutron/ 4679/
https:/
https:/
https:/
and looks like in many others builds
in all the cases, something like this can be seen in controller's kern.log:
<3>Sep 14 00:23:25 node-3 kernel: [ 2603.871961] Out of memory: Kill process 2156 (mysqld) score 43 or sacrifice child
<3>Sep 14 00:23:25 node-3 kernel: [ 2603.871965] Killed process 2122 (mysqld) total-vm:2547440kB, anon-rss:112436kB, file-rss:2444kB
also OSTF tests in all cases are failing like this:
Create volume and boot instance from it success
Launch instance, create snapshot, launch instance from snapshot failure
Create instance flavor success
Request volume list success
Launch instance with file injection failure
Request user list success
Request image list using Glance v2 success
Create security group failure
Launch instance failure
Request active services list success
Check that required services are running success
Create user and authenticate with it. failure
Request image list using Glance v1 success
Request snapshot list success
Create keypair failure
Request flavor list success
Check create, update and delete image actions using Glance v2 success
Request image list using Nova success
Create volume and attach it to instance success
Request instance list success
Request stack list success
Check network parameters failure
Request list of networks success
Check network connectivity from instance via floating IP failure
Request absolute limits list success
This issue really blocks CI in upstream openstack-puppet project