Too few wsgi workers for keystone
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mirantis OpenStack |
Invalid
|
Undecided
|
MOS Keystone |
Bug Description
We faced with an issue during boot_and_list rally scenario: http://
We run rally on dedicated harware node with 48 CPUs and 256 RAM
rally@6bb2d5d38
0.3.2~dev84
We got 500 HTTP error:
from rally log: http://
The error was from nova-api:
find 500 HTTP code at the time from haproxy log: http://
look at nova log on the node which we know from haproxy: http://
can't find a keystone node from haproxy log : http://
There is node error in keystone logs at the same time. But I've noticed that there is only 6 processes for keystone wsgi workers: http://
And it seems all of the workers was overloaded:
atop from node-1 controller: http://
atop from node-2 controller: http://
atop from node-3 controller: http://
Environment description:
3controllers,
[root@fuel ~]# cat /etc/fuel/
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "8.0"
api: "1.0"
build_number: "570"
build_id: "570"
fuel-nailgun_sha: "558ca91a854cf2
python-
fuel-agent_sha: "658be72c4b42d3
fuel-
astute_sha: "b81577a5b7857c
fuel-library_sha: "c2a335b5b725f1
fuel-ostf_sha: "3bc76a63a9e7d1
fuel-mirror_sha: "fb45b80d7bee58
fuelmenu_sha: "78ffc73065a967
shotgun_sha: "63645dea384a37
network-
fuel-upgrade_sha: "616a7490ec7199
fuelmain_sha: "d605bcbabf3153
!!! NOTE: RabbitMQ was replaced by ZeroMQ in this installation. Actually, rabbitmq wasn't removed, but zeromq was installed and oslo-messaging worked via zeromq.
Will try to generate Diagnostic Snapshot, but there is bug which can prevent to do that (https:/
Logs from controller nodes are here: http://
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
summary: |
- too few wsgi workers for keystone + Too few wsgi workers for keystone |
Changed in mos: | |
milestone: | 8.0-mu-1 → 8.0-updates |
According to keystone.log the average request to API takes >20 second. e.g.: wsgi_main_ access 192.168.0.103 - - [11/Mar/ 2016:12: 47:31 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 22387336 "-" "python- keystoneclient" wsgi_main_ access 192.168.0.103 - - [11/Mar/ 2016:12: 47:43 +0000] "POST /v2.0/tokens HTTP/1.1" 200 4577 10758361 "-" "neutron/7.0.2 keystonemiddlew are.auth_ token/2. 3.2" wsgi_main_ access 192.168.0.103 - - [11/Mar/ 2016:12: 47:32 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 22235663 "-" "python- keystoneclient" wsgi_main_ access 192.168.0.101 - - [11/Mar/ 2016:12: 47:33 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 21911819 "-" "python- keystoneclient" wsgi_main_ access 192.168.0.103 - - [11/Mar/ 2016:12: 47:32 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 23379620 "-" "python- keystoneclient" wsgi_main_ access 192.168.0.28 - - [11/Mar/ 2016:12: 47:35 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 22400081 "-" "python- keystoneclient" wsgi_main_ access 192.168.0.28 - - [11/Mar/ 2016:12: 47:35 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 23467167 "-" "python- keystoneclient" wsgi_main_ access 192.168.0.28 - - [11/Mar/ 2016:12: 47:36 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 22843466 "-" "python- keystoneclient" wsgi_main_ access 192.168.0.103 - - [11/Mar/ 2016:12: 47:36 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 23528153 "-" "python- keystoneclient" wsgi_main_ access 192.168.0.28 - - [11/Mar/ 2016:12: 47:37 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 23840234 "-" "python- keystoneclient" wsgi_main_ access - - - [11/Mar/ 2016:12: 47:48 +0000] "GET /v3 HTTP/1.0" 500 251 12117172 "-" "-" wsgi_main_ access 192.168.0.101 - - [11/Mar/ 2016:12: 47:39 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7687 22441397 "-" "python- keystoneclient" wsgi_main_ access - - - [11/Mar/ 2016:12: 47:49 +0000] "GET /v3 HTTP/1.0" 500 251 12886038 "-" "-" wsgi_main_ access 192.168.0.28 - - [11/Mar/ 2016:12: 47:38 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 23778944 "-" "python- keystoneclient" wsgi_main_ access - - - [11/Mar/ 2016:12: 47:49 +0000] "GET /v3 HTTP/1.0" 500 251 13210862 "-" "-" wsgi_main_ access 192.168.0.101 - - [11/Mar/ 2016:12: 47:41 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 21187830 "-" "python- keystoneclient" wsgi_main_ access 192.168.0.101 - - [11/Mar/ 2016:12: 47:40 +0000] "GET /v3/auth/tokens HTTP/1.1" 200 7649 22641118 "-" "python- keystoneclient"
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_
<46>Mar 11 12:48:03 node-1 keystone_