Timeout when reading response headers from daemon process 'octavia-api' after restart

Bug #1909004 reported by Jake Hill on 2020-12-22

This bug report will be marked for expiration in 59 days if no further activity occurs. (find out why)

10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Octavia Charm
Undecided
Unassigned

Bug Description

Perhaps or not related to #1885936 but I think not so raising new bug.

Install cs:openstack-base-70 with the attached overlay for octavia. When everything is green run e.g.

  watch openstack loadbalancer list

in a terminal. You can see the empty list of loadbalancers.

Forcably restart the octavia unit, e.g.

  juju ssh octavia/0 -- sudo reboot

The unit seems to come back, but the API no longer responds. The once the unit has settled again, the client returns;

Unable to establish connection to https://192.168.200.193:9876/v2.0/lbaas/loadbalancers: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

In /var/log/apache2/octavia_error.log there are these messages;

[Tue Dec 22 11:13:38.488467 2020] [wsgi:error] [pid 3449:tid 139866935383808] [client 127.0.0.1:50482] Timeout when reading response headers from daemon process 'octavia-api': /usr/bin/octavia-wsgi
[Tue Dec 22 11:15:14.200171 2020] [wsgi:error] [pid 3449:tid 139866901812992] [client 127.0.0.1:52026] Timeout when reading response headers from daemon process 'octavia-api': /usr/bin/octavia-wsgi
[Tue Dec 22 11:16:49.404853 2020] [wsgi:error] [pid 3448:tid 139867030517504] [client 127.0.0.1:53520] Timeout when reading response headers from daemon process 'octavia-api': /usr/bin/octavia-wsgi
[Tue Dec 22 11:18:24.722825 2020] [wsgi:error] [pid 3449:tid 139866658555648] [client 127.0.0.1:55038] Timeout when reading response headers from daemon process 'octavia-api': /usr/bin/octavia-wsgi
[Tue Dec 22 11:20:00.493323 2020] [wsgi:error] [pid 3449:tid 139866264295168] [client 127.0.0.1:56534] Timeout when reading response headers from daemon process 'octavia-api': /usr/bin/octavia-wsgi
[Tue Dec 22 11:21:35.836805 2020] [wsgi:error] [pid 3449:tid 139866650162944] [client 127.0.0.1:58016] Timeout when reading response headers from daemon process 'octavia-api': /usr/bin/octavia-wsgi
[Tue Dec 22 11:23:11.372837 2020] [wsgi:error] [pid 3449:tid 139866415298304] [client 127.0.0.1:59540] Timeout when reading response headers from daemon process 'octavia-api': /usr/bin/octavia-wsgi
[Tue Dec 22 11:24:47.687568 2020] [wsgi:error] [pid 3449:tid 139866364942080] [client 127.0.0.1:32792] Timeout when reading response headers from daemon process 'octavia-api': /usr/bin/octavia-wsgi

Jake Hill (routergod) wrote :
Liam Young (gnuoy) wrote :

I could not reproduce this issue. Please could include the logs from octavia/0:var/log/octavia ?

$ openstack loadbalancer list

$ juju ssh octavia/0 -- sudo reboot

Connection to 172.20.0.8 closed by remote host.
Connection to 172.20.0.8 closed.

<Wait for restart>

$ openstack loadbalancer list

$

Changed in charm-octavia:
status: New → Incomplete
Jake Hill (routergod) wrote :

Thanks for looking. I've included apache2 and octavia logs in the tarball. It looks perhaps like some problem with the WSGI invocation, the octavia logs don't give me anything.

Will also add a paste of a console session pinging the API during the reboot. It is a bit odd, the API does appear to come back for at least one request post-reboot.

Jake Hill (routergod) wrote :

Console session

Jake Hill (routergod) wrote :

Apologies that was pretty pointless as I did not give enough time for the WGSI timeout. Here is a replacement /var/log/apache2/octavia_error.log

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers