Suboptimal KeepAliveTimeout terminates keystoneauth1.session.Session unnecessarily and leads to "Remote end closed connection without response"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Charm Helpers |
Fix Released
|
Undecided
|
Unassigned | ||
OpenStack Base Layer |
Fix Released
|
Undecided
|
Unassigned | ||
OpenStack Charm Guide |
Fix Released
|
Undecided
|
Nobuto Murata | ||
OpenStack Dashboard Charm |
Fix Released
|
Undecided
|
Unassigned | ||
OpenStack Keystone Charm |
Fix Released
|
High
|
Unassigned |
Bug Description
Charmed OpenStack doesn't set an explicit KeepAliveTimeout for OpenStack API and dashboard services, and the default value of KeepAliveTimeout in Apache2 is used which is 5 seconds.
When clients (including OpenStack services themselves talking to other services) open a session using keystoneauth1.
I've seen a lot of not-normally terminated sessions ("CD" status) at haproxy in front of apache2. And it's because of the connection reset from the client side due to an expiration of Keep alive timeout.
> CD The client unexpectedly aborted during data transfer. This can be
> caused by a browser crash, by an intermediate equipment between the
> client and haproxy which decided to actively break the connection,
> by network routing issues between the client and haproxy, or by a
> keep-alive session between the server and the client terminated first
> by the client.
By increasing the KeepAliveTimeout value, we can achieve:
- far less frequent not-normally terminated sessions (haproxy logs will be a lot cleaner)
- TCP connections are properly reused in the session, which keystoneauth1.
- avoid intermittent but fatal errors of "Remote end closed connection without response" from occurring
Actual issues in the field:
https:/
Changed in charm-keystone: | |
status: | In Progress → Fix Committed |
importance: | Undecided → High |
status: | Fix Committed → In Progress |
Changed in layer-openstack: | |
status: | In Progress → Fix Committed |
Changed in layer-openstack: | |
milestone: | none → 22.04 |
Changed in charm-openstack-dashboard: | |
milestone: | none → 22.04 |
Changed in charm-keystone: | |
milestone: | none → 22.04 |
Changed in charm-helpers: | |
status: | In Progress → Fix Committed |
Changed in charm-guide: | |
assignee: | nobody → Nobuto Murata (nobuto) |
Changed in charm-keystone: | |
status: | Fix Committed → Fix Released |
Changed in layer-openstack: | |
status: | Fix Committed → Fix Released |
Changed in charm-openstack-dashboard: | |
status: | Fix Committed → Fix Released |
Changed in charm-helpers: | |
status: | Fix Committed → Fix Released |
1st example: if a second request is 6 seconds after the first request (> KeepAliveTimeou t=5), tcp connection is not reused under keystoneauth1. session. Session.
The line to highlight is "urllib3. connectionpool: Resetting dropped connection" which is resetting the TCP connection and to re-establish another TCP connection.
https:/ /docs.openstack .org/python- novaclient/ latest/ user/python- api.html identity import v3
====
from keystoneauth1.
from keystoneauth1 import session
from novaclient import client
import logging
logging. basicConfig( level=logging. DEBUG)
from time import sleep
auth = v3.Password( auth_url= "https:/ /192.168. 151.112: 5000/v3",
username= "admin" , password= "MY_PASSWORD" ,
project_ name="admin" ,
user_ domain_ name="admin_ domain" ,
project_ domain_ name="admin_ domain" )
sess = session. Session( auth=auth, "/home/ ubuntu/ snap/openstackc lients/ common/ root-ca. crt")
verify=
nova = client.Client(2, session=sess)
nova.servers.list()
sleep(6)
nova.flavors.list()
====
>>> nova.flavors.list() uth.session: REQ: curl -g -i --cacert "/home/ ubuntu/ snap/openstackc lients/ common/ root-ca. crt" -X GET https:/ /192.168. 151.115: 8774/v2. 1/flavors/ detail -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256} 5849e99d70d26ae 11c4c62e4940105 9c89a66c6a1172b 9bd867e1b36477a 07f8" connectionpool: Resetting dropped connection: 192.168.151.115 connectionpool:https:/ /192.168. 151.115: 8774 "GET /v2.1/flavors/ detail HTTP/1.1" 200 478 uth.session: RESP: [200] Connection: Keep-Alive Content-Length: 478 Content-Type: application/json Date: Wed, 13 Oct 2021 15:41:02 GMT Keep-Alive: timeout=5, max=100 OpenStack- API-Version: compute 2.1 Server: Apache/2.4.41 (Ubuntu) Vary: OpenStack- API-Version, X-OpenStack- Nova-API- Version X-OpenStack- Nova-API- Version: 2.1 x-compute- request- id: req-f299ee15- cad8-429a- 954c-a1b79047e3 5f x-openstack- request- id: req-f299ee15- cad8-429a- 954c-a1b79047e3 5f
DEBUG:keystonea
DEBUG:urllib3.
DEBUG:urllib3.
DEBUG:keystonea
...