| 2021-10-13 16:27:11 |
Nobuto Murata |
bug |
|
|
added bug |
| 2021-10-13 16:27:44 |
Nobuto Murata |
bug task added |
|
layer-openstack |
|
| 2021-10-13 16:27:55 |
Nobuto Murata |
bug task added |
|
charm-keystone |
|
| 2021-10-13 16:28:11 |
Nobuto Murata |
bug task added |
|
charm-openstack-dashboard |
|
| 2021-10-13 16:35:21 |
Nobuto Murata |
attachment added |
|
novaclient_output.txt https://bugs.launchpad.net/charm-openstack-dashboard/+bug/1947010/+attachment/5532502/+files/novaclient_output.txt |
|
| 2021-10-13 16:36:34 |
Nobuto Murata |
description |
Charmed OpenStack doesn't set an explicit KeepAliveTimeout for OpenStack API and dashboard services, and the default value of KeepAliveTimeout in Apache2 is used which is 5 seconds.
When clients (including OpenStack services themselves talking to other services) open a session using keystoneauth1.session.Session. It tries to use a connection pool and respect HTTP Keep alive timeout. However, due to the suboptimal KeepAliveTimeout on the server side, clients have to re-establish TCP connections whenever an interval of two requests is more than 5 seconds.
I've seen a lot of not-normally terminated sessions ("CD" status) at haproxy in front of apache2. And it's because of the connection reset from the client side due to an expiration of Keep alive timeout.
> CD The client unexpectedly aborted during data transfer. This can be
> caused by a browser crash, by an intermediate equipment between the
> client and haproxy which decided to actively break the connection,
> by network routing issues between the client and haproxy, or by a
> keep-alive session between the server and the client terminated first
> by the client.
By increasing the KeepAliveTimeout value, we can achieve:
- far less frequent not-normally terminated sessions (haproxy logs will be a lot cleaner)
- TCP connections are properly reused in the session, which keystoneauth1.session.Session was designed for
- avoid intermittent but fatal errors of "Remote end closed connection without response" from occurring |
Charmed OpenStack doesn't set an explicit KeepAliveTimeout for OpenStack API and dashboard services, and the default value of KeepAliveTimeout in Apache2 is used which is 5 seconds.
When clients (including OpenStack services themselves talking to other services) open a session using keystoneauth1.session.Session. It tries to use a connection pool and respect HTTP Keep alive timeout. However, due to the suboptimal KeepAliveTimeout on the server side, clients have to re-establish TCP connections whenever an interval of two requests is more than 5 seconds.
I've seen a lot of not-normally terminated sessions ("CD" status) at haproxy in front of apache2. And it's because of the connection reset from the client side due to an expiration of Keep alive timeout.
> CD The client unexpectedly aborted during data transfer. This can be
> caused by a browser crash, by an intermediate equipment between the
> client and haproxy which decided to actively break the connection,
> by network routing issues between the client and haproxy, or by a
> keep-alive session between the server and the client terminated first
> by the client.
By increasing the KeepAliveTimeout value, we can achieve:
- far less frequent not-normally terminated sessions (haproxy logs will be a lot cleaner)
- TCP connections are properly reused in the session, which keystoneauth1.session.Session was designed for
- avoid intermittent but fatal errors of "Remote end closed connection without response" from occurring
Actual issues in the field:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1944467 |
|
| 2021-10-13 16:56:59 |
Nobuto Murata |
bug watch added |
|
https://github.com/psf/requests/issues/4664 |
|
| 2021-10-13 16:56:59 |
Nobuto Murata |
bug watch added |
|
http://bugs.python.org/issue41345 |
|
| 2021-10-13 16:56:59 |
Nobuto Murata |
attachment added |
|
keystoneclient_output.txt https://bugs.launchpad.net/charm-openstack-dashboard/+bug/1947010/+attachment/5532504/+files/keystoneclient_output.txt |
|
| 2021-10-14 00:53:22 |
OpenStack Infra |
charm-keystone: status |
New |
In Progress |
|
| 2021-10-14 00:57:04 |
OpenStack Infra |
charm-openstack-dashboard: status |
New |
In Progress |
|
| 2021-10-14 01:00:14 |
Nobuto Murata |
layer-openstack: status |
New |
In Progress |
|
| 2021-10-14 01:00:23 |
Nobuto Murata |
charm-helpers: status |
New |
In Progress |
|
| 2021-10-14 01:08:59 |
Nobuto Murata |
bug |
|
|
added subscriber Canonical Field Medium |
| 2021-10-14 22:54:55 |
Billy Olsen |
charm-keystone: status |
In Progress |
Fix Committed |
|
| 2021-10-14 22:54:58 |
Billy Olsen |
charm-keystone: importance |
Undecided |
High |
|
| 2021-10-14 22:55:42 |
Billy Olsen |
charm-keystone: status |
Fix Committed |
In Progress |
|
| 2021-10-14 23:11:57 |
Billy Olsen |
layer-openstack: status |
In Progress |
Fix Committed |
|
| 2021-10-18 22:38:16 |
OpenStack Infra |
charm-keystone: status |
In Progress |
Fix Committed |
|
| 2021-11-24 14:35:45 |
OpenStack Infra |
charm-openstack-dashboard: status |
In Progress |
Fix Committed |
|
| 2021-11-24 22:23:54 |
Felipe Reyes |
layer-openstack: milestone |
|
22.04 |
|
| 2021-11-24 22:23:58 |
Felipe Reyes |
charm-openstack-dashboard: milestone |
|
22.04 |
|
| 2021-11-24 22:24:02 |
Felipe Reyes |
charm-keystone: milestone |
|
22.04 |
|
| 2021-11-24 23:02:21 |
Nobuto Murata |
charm-helpers: status |
In Progress |
Fix Committed |
|
| 2022-02-14 07:30:07 |
Nobuto Murata |
removed subscriber Canonical Field Medium |
|
|
|
| 2022-02-14 07:30:22 |
Nobuto Murata |
bug task added |
|
charm-guide |
|
| 2022-02-14 07:30:30 |
Nobuto Murata |
charm-guide: assignee |
|
Nobuto Murata (nobuto) |
|
| 2022-04-05 01:58:31 |
Nobuto Murata |
bug |
|
|
added subscriber Canonical Field Medium |
| 2022-04-13 03:04:33 |
OpenStack Infra |
charm-guide: status |
New |
In Progress |
|
| 2022-04-14 03:13:33 |
OpenStack Infra |
charm-guide: status |
In Progress |
Fix Released |
|
| 2022-05-10 13:41:42 |
OpenStack Infra |
tags |
|
in-stable-xena |
|
| 2022-05-10 15:30:18 |
Alex Kavanagh |
charm-keystone: status |
Fix Committed |
Fix Released |
|
| 2022-05-10 16:47:44 |
Alex Kavanagh |
layer-openstack: status |
Fix Committed |
Fix Released |
|
| 2022-05-10 16:47:45 |
Alex Kavanagh |
charm-openstack-dashboard: status |
Fix Committed |
Fix Released |
|
| 2022-06-02 00:48:54 |
Nobuto Murata |
charm-helpers: status |
Fix Committed |
Fix Released |
|
| 2022-06-16 02:47:11 |
OpenStack Infra |
tags |
in-stable-xena |
in-stable-wallaby in-stable-xena |
|
| 2022-06-16 02:59:54 |
OpenStack Infra |
tags |
in-stable-wallaby in-stable-xena |
in-stable-victoria in-stable-wallaby in-stable-xena |
|
| 2022-06-16 03:12:16 |
OpenStack Infra |
tags |
in-stable-victoria in-stable-wallaby in-stable-xena |
in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena |
|
