gunicorn worker timeout blocks appliance config update

Bug #1462100 reported by Adam Gandelman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Astara
Fix Released
Medium
Adam Gandelman
akanda
Fix Released
Medium
Adam Gandelman

Bug Description

this is probably only an issue in dev environments where we're running the appliance /w non-accelerated nested virt, which describes where the devstack gate jobs run.

in such envs, the appliance tries to restart the metadata service as part of its config update. this can take a long time, sometimes >=a 30s, which is the default gunicorn worker timeout. in the devstack slaves, it appears to consistently take longer than 30s causing the config updates to fail with a BadStatusLine error on the client (rug) side, ie http://logs.openstack.org/94/188194/1/experimental/check-functional-dsvm-akanda-appliance-src/e1e0c28/logs/screen-ak-rug.txt.gz#_2015-06-03_23_40_18

2015-06-03 23:40:18:ERROR:akanda.rug.state.d2db15e9-288d-4400-aeef-9b8dfef266d4:13546:p01:t00:failed to update config
Traceback (most recent call last):
  File "/opt/stack/new/akanda-rug/akanda/rug/vm_manager.py", line 357, in configure
    config
  File "/opt/stack/new/akanda-rug/akanda/rug/api/akanda_client.py", line 72, in update_config
    timeout=cfg.CONF.config_timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 518, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Again, this causes the config update to fail and the router never reaches the active state.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to akanda-appliance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188596

Changed in akanda:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to akanda-appliance (master)

Reviewed: https://review.openstack.org/188596
Committed: https://git.openstack.org/cgit/stackforge/akanda-appliance/commit/?id=a12136b2039bbc341b6d11ccc6e77d0791ef6f02
Submitter: Jenkins
Branch: master

commit a12136b2039bbc341b6d11ccc6e77d0791ef6f02
Author: Adam Gandelman <email address hidden>
Date: Thu Jun 4 14:15:21 2015 -0700

    Bump gunicorn worker timeout to 60

    The default 30s worker timeout for gunicorn is not sufficient for
    some slower environments, as the metadata service takes a while to
    restart. This bumps it to 60s to ensure we dont fail config updates
    on worker timeouts.

    Change-Id: If27329b58d97e6aef687a069615dbda763dba8c1
    Closes-bug: #1462100

Changed in akanda:
status: In Progress → Fix Committed
Sean Roberts (sarob)
tags: added: akanda-appliance
Changed in akanda:
milestone: none → liberty-1
assignee: nobody → Adam Gandelman (gandelman-a)
importance: Undecided → Medium
Sean Roberts (sarob)
Changed in akanda:
status: Fix Committed → Fix Released
Changed in akanda:
milestone: liberty-1 → 7.0.0
Changed in astara:
milestone: none → 7.0.0
assignee: nobody → Adam Gandelman (gandelman-a)
importance: Undecided → Medium
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.