Out of memory in apache2 process after Rally keystone tests
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mirantis OpenStack |
Won't Fix
|
High
|
MOS Keystone | ||
10.0.x |
Won't Fix
|
High
|
MOS Keystone | ||
9.x |
Won't Fix
|
High
|
MOS Keystone |
Bug Description
SW version:
MOS 9.0 RC2 build 495
Environment:
3 controller + mongo nodes
5 compute + ceph nodes
1 elasticsearch
1 influxdb
Detailed bug description:
During Rally benchmarking Keystone gets hang with strange observation. `keystone user-list` command raises 504 Gateway timeout. At the same time `keystone endpoint list` works well. Several Rally tests are failed because of 504 error.
From syslog:
<3>Jun 23 10:04:25 node-2 kernel: [171009.310711] Out of memory: Kill process 31782 (apache2) score 92 or sacrifice child
<3>Jun 23 10:04:25 node-2 kernel: [171009.310713] Killed process 31782 (apache2) total-vm:5079100kB, anon-rss:4525040kB, file-rss:1544kB
<129>Jun 23 10:07:35 node-2 haproxy[7618]: Server keystone-2/node-2 is DOWN, reason: Layer7 timeout, check duration: 10000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
<3>Jun 23 10:07:57 node-2 kernel: [171221.518638] Out of memory: Kill process 20504 (apache2) score 94 or sacrifice child
<3>Jun 23 10:07:57 node-2 kernel: [171221.541984] Killed process 20504 (apache2) total-vm:5148200kB, anon-rss:4603044kB, file-rss:1188kB
<11>Jun 23 10:16:26 node-2 epmd: epmd: node name already occupied rabbitmq-cli-54
<129>Jun 23 10:23:20 node-2 haproxy[7618]: Server keystone-2/node-1 is DOWN, reason: Layer7 timeout, check duration: 10000ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
<11>Jun 23 10:25:40 node-2 epmd: epmd: node name already occupied rabbitmq-cli-27
<129>Jun 23 10:27:19 node-2 haproxy[7618]: Server keystone-2/node-1 is DOWN, reason: Layer7 timeout, check duration: 10001ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Fuel snapshot http://
Changed in mos: | |
assignee: | nobody → MOS Keystone (mos-keystone) |
tags: | added: area-keystone |
Changed in mos: | |
status: | New → Confirmed |
importance: | Undecided → High |
(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:
actual result
expected result
steps to reproduce
For more detailed information on the contents of each of the listed sections see https:/ /wiki.openstack .org/wiki/ Fuel/How_ to_contribute# Here_is_ how_you_ file_a_ bug