Out of memory in apache2 process after Rally keystone tests

[Ivan Lozgachev]

SW version:
MOS 9.0 RC2 build 495

Environment:
3 controller + mongo nodes
5 compute + ceph nodes
1 elasticsearch
1 influxdb

Detailed bug description:
During Rally benchmarking Keystone gets hang with strange observation. `keystone user-list` command raises 504 Gateway timeout. At the same time `keystone endpoint list` works well. Several Rally tests are failed because of 504 error.

From syslog:
<3>Jun 23 10:04:25 node-2 kernel: [171009.310711] Out of memory: Kill process 31782 (apache2) score 92 or sacrifice child
<3>Jun 23 10:04:25 node-2 kernel: [171009.310713] Killed process 31782 (apache2) total-vm:5079100kB, anon-rss:4525040kB, file-rss:1544kB
<129>Jun 23 10:07:35 node-2 haproxy[7618]: Server keystone-2/node-2 is DOWN, reason: Layer7 timeout, check duration: 10000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
<3>Jun 23 10:07:57 node-2 kernel: [171221.518638] Out of memory: Kill process 20504 (apache2) score 94 or sacrifice child
<3>Jun 23 10:07:57 node-2 kernel: [171221.541984] Killed process 20504 (apache2) total-vm:5148200kB, anon-rss:4603044kB, file-rss:1188kB
<11>Jun 23 10:16:26 node-2 epmd: epmd: node name already occupied rabbitmq-cli-54
<129>Jun 23 10:23:20 node-2 haproxy[7618]: Server keystone-2/node-1 is DOWN, reason: Layer7 timeout, check duration: 10000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
<11>Jun 23 10:25:40 node-2 epmd: epmd: node name already occupied rabbitmq-cli-27
<129>Jun 23 10:27:19 node-2 haproxy[7618]: Server keystone-2/node-1 is DOWN, reason: Layer7 timeout, check duration: 10001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Fuel snapshot http://mos-scale-share.mirantis.com/fuel-snapshot-2016-06-23_10-25-08.tar.gz

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Ivan Lozgachev (ilozgachev)]

Log from Slack channel http://paste.openstack.org/show/521853/

[Dina Belova (dbelova)]

This bug is triggered by running load with 200+ concurrent requests from load generator (Rally in our case) -> moving its solving to 9.0-updates

[Alexander Makarov (amakarov)]

The problem is in 110k+ users in the keystone database generated by rally.
It's well known that keystone has poor identity part - it's more about tokens.

[Dina Belova (dbelova)]

This is a new enhancement / research task to see if we can improve identity part of keystone. Adding non-release and enhancement tags + keeping it for 10.0

[Boris Bobrov (bbobrov)]

> <3>Jun 23 10:07:57 node-2 kernel: [171221.518638] Out of memory: Kill process 20504 (apache2) score 94 or sacrifice child

There is not enough memory on your node. Buy more RAM.