Requests to neutron API do not use retries
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Low
|
melanie witt | ||
Queens |
Fix Released
|
Low
|
s10 | ||
Rocky |
Fix Released
|
Low
|
melanie witt | ||
Stein |
Fix Released
|
Low
|
melanie witt | ||
Train |
Fix Released
|
Low
|
melanie witt |
Bug Description
We have a customer bug report downstream [1] where nova occasionally fails to carry out server actions requiring calls to neutron API if haproxy happens to close a connection after idle time of 10 seconds at nearly the same time as an incoming request that attempts to re-use the connection while it is being torn down. Here is an excerpt from [1]:
The result of our investigation, the cause is follows.
1. neutron-client in nova uses connection pool ( urllib3/requests ) for http.
2. Sometimes, http connection is reused for different requests.
3. Connection between neutron-client and haproxy is closed from haproxy when it is in idle for 10 seconds.
4. If reusing connection from client side and closing connection from haproxy side are happend almost same time,
client gets RST and end with "bad status line".
To address this problem, we can add a new config option for neutron client (similar to the existing config options we have for cinder client and glance client retries) to be more resilient during such scenarios.
Changed in nova: | |
assignee: | melanie witt (melwitt) → Lee Yarwood (lyarwood) |
Changed in nova: | |
assignee: | Lee Yarwood (lyarwood) → melanie witt (melwitt) |
Fix proposed to branch: master /review. opendev. org/712226
Review: https:/