| 2015-08-17 15:17:52 |
Bogdan Dobrelya |
bug |
|
|
added bug |
| 2015-08-17 15:17:57 |
Bogdan Dobrelya |
fuel: milestone |
|
7.0 |
|
| 2015-08-17 15:18:04 |
Bogdan Dobrelya |
fuel: assignee |
|
Fuel Library Team (fuel-library) |
|
| 2015-08-17 15:18:06 |
Bogdan Dobrelya |
fuel: importance |
Undecided |
Critical |
|
| 2015-08-17 15:37:33 |
Alex Schultz |
fuel: status |
New |
Confirmed |
|
| 2015-08-17 15:37:35 |
Alex Schultz |
fuel: assignee |
Fuel Library Team (fuel-library) |
Alex Schultz (alex-schultz) |
|
| 2015-08-17 17:30:36 |
Bogdan Dobrelya |
summary |
Keystone HAProxy backend is too SLOW under wsgi |
Keystone wsgi runs out of connections limits for the apache proces |
|
| 2015-08-18 10:06:07 |
Bogdan Dobrelya |
description |
Under wsgi, the keystone seems too slow to accept connections in time, thus haproxy sporadically marks it <NOSRV> making a deployment fail. This is a summary bug from these two: https://bugs.launchpad.net/fuel/+bug/1485597 and https://bugs.launchpad.net/fuel/+bug/1485591
This bug is also related to https://bugs.launchpad.net/fuel/+bug/1434216
Note, current configuration options are:
timeout http-request 20s
timeout connect 10s
We should either to increase them or to fix the slow backend to accept connections better. |
This is a summary bug from these two: https://bugs.launchpad.net/fuel/+bug/1485597 and https://bugs.launchpad.net/fuel/+bug/1485591
We should:
1) tune Apache2 with amount of simultaneous connections by increasing Apache2 connection limits (ServerLimit, MaxClients, StartServers, MinSpareServers and MaxSpareServers)
2) (see http://veithen.github.io/2014/01/01/how-tcp-backlog-works-in-linux.html) tune Linux sysctls to allow Apache2 to process more connections:
number of connections per a socket (somaxconn), tcp_max_syn_backlog global value, and probably enable tcp_abort_on_overflow and tcp_tw_reuse as well |
|
| 2015-08-18 15:30:32 |
Bogdan Dobrelya |
description |
This is a summary bug from these two: https://bugs.launchpad.net/fuel/+bug/1485597 and https://bugs.launchpad.net/fuel/+bug/1485591
We should:
1) tune Apache2 with amount of simultaneous connections by increasing Apache2 connection limits (ServerLimit, MaxClients, StartServers, MinSpareServers and MaxSpareServers)
2) (see http://veithen.github.io/2014/01/01/how-tcp-backlog-works-in-linux.html) tune Linux sysctls to allow Apache2 to process more connections:
number of connections per a socket (somaxconn), tcp_max_syn_backlog global value, and probably enable tcp_abort_on_overflow and tcp_tw_reuse as well |
This is a summary bug from these two: https://bugs.launchpad.net/fuel/+bug/1485597 and https://bugs.launchpad.net/fuel/+bug/1485591
We should:
1) tune Apache2 with amount of simultaneous connections by increasing Apache2 connection limits:
MaxClients = ServerLimit * ThreadsPerChild (25 default), instead of the current value 100
ServerLimit = $::processorcount, while the current is 4
2) tune Linux sysctls to allow Apache2 to process more connections:
number of connections per a socket (somaxconn), tcp_max_syn_backlog global value, and (optional) enable tcp_abort_on_overflow and tcp_tw_reuse as well. |
|
| 2015-08-18 15:31:02 |
Bogdan Dobrelya |
description |
This is a summary bug from these two: https://bugs.launchpad.net/fuel/+bug/1485597 and https://bugs.launchpad.net/fuel/+bug/1485591
We should:
1) tune Apache2 with amount of simultaneous connections by increasing Apache2 connection limits:
MaxClients = ServerLimit * ThreadsPerChild (25 default), instead of the current value 100
ServerLimit = $::processorcount, while the current is 4
2) tune Linux sysctls to allow Apache2 to process more connections:
number of connections per a socket (somaxconn), tcp_max_syn_backlog global value, and (optional) enable tcp_abort_on_overflow and tcp_tw_reuse as well. |
This is a summary bug from these two: https://bugs.launchpad.net/fuel/+bug/1485597 and https://bugs.launchpad.net/fuel/+bug/1485591
We should:
1) tune Apache2 MPM worker (/etc/apache2/mods-available/worker.conf) with amount of simultaneous connections by increasing Apache2 connection limits:
MaxClients = ServerLimit * ThreadsPerChild (25 default), instead of the current value 100
ServerLimit = $::processorcount, while the current is 4
2) tune Linux sysctls to allow Apache2 to process more connections:
number of connections per a socket (somaxconn), tcp_max_syn_backlog global value, and (optional) enable tcp_abort_on_overflow and tcp_tw_reuse as well. |
|
| 2015-08-18 15:31:26 |
Bogdan Dobrelya |
tags |
|
haproxy keystone |
|
| 2015-08-18 15:31:53 |
Bogdan Dobrelya |
summary |
Keystone wsgi runs out of connections limits for the apache proces |
Keystone wsgi might run out of connections limits for the apache proces |
|
| 2015-08-18 15:32:18 |
Bogdan Dobrelya |
bug |
|
|
added subscriber Aleksandr Didenko |
| 2015-08-18 15:40:04 |
Aleksandr Didenko |
fuel: status |
Confirmed |
Invalid |
|
| 2015-08-18 16:02:42 |
Bogdan Dobrelya |
description |
This is a summary bug from these two: https://bugs.launchpad.net/fuel/+bug/1485597 and https://bugs.launchpad.net/fuel/+bug/1485591
We should:
1) tune Apache2 MPM worker (/etc/apache2/mods-available/worker.conf) with amount of simultaneous connections by increasing Apache2 connection limits:
MaxClients = ServerLimit * ThreadsPerChild (25 default), instead of the current value 100
ServerLimit = $::processorcount, while the current is 4
2) tune Linux sysctls to allow Apache2 to process more connections:
number of connections per a socket (somaxconn), tcp_max_syn_backlog global value, and (optional) enable tcp_abort_on_overflow and tcp_tw_reuse as well. |
This is a summary bug from these two: https://bugs.launchpad.net/fuel/+bug/1485597 and https://bugs.launchpad.net/fuel/+bug/1485591
We should:
1) tune Apache2 MPM worker (/etc/apache2/mods-available/worker.conf) with amount of simultaneous connections by increasing Apache2 connection limits from RAM based formula https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/osnailyfacter/manifests/apache_mpm.pp to CPU based:
MaxClients = ServerLimit * ThreadsPerChild (25 default), instead of the current value 100
ServerLimit = $::processorcount, while the current is 4
2) tune Linux sysctls to allow Apache2 to process more connections:
number of connections per a socket (somaxconn), tcp_max_syn_backlog global value, and (optional) enable tcp_abort_on_overflow and tcp_tw_reuse as well. |
|
| 2015-08-18 19:23:34 |
Bogdan Dobrelya |
fuel: status |
Invalid |
Confirmed |
|
| 2015-08-18 19:23:42 |
Bogdan Dobrelya |
tags |
haproxy keystone |
haproxy keystone scale |
|
| 2015-08-18 19:23:50 |
Bogdan Dobrelya |
tags |
haproxy keystone scale |
apache2 haproxy keystone scale wsgi |
|
| 2015-08-18 19:43:03 |
Bogdan Dobrelya |
summary |
Keystone wsgi might run out of connections limits for the apache proces |
Keystone wsgi might run out of connections limits for the apache2 process |
|
| 2015-08-18 20:00:32 |
OpenStack Infra |
fuel: status |
Confirmed |
In Progress |
|
| 2015-08-19 08:51:03 |
OpenStack Infra |
fuel: status |
In Progress |
Fix Committed |
|
| 2015-09-07 10:09:08 |
Oleksiy Molchanov |
tags |
apache2 haproxy keystone scale wsgi |
apache2 haproxy keystone on-verification scale wsgi |
|
| 2015-09-07 12:33:44 |
Oleksiy Molchanov |
fuel: status |
Fix Committed |
Fix Released |
|
| 2015-09-07 13:16:08 |
Oleksiy Molchanov |
tags |
apache2 haproxy keystone on-verification scale wsgi |
apache2 haproxy keystone scale wsgi |
|
