keystone failed to start on centos7 after change to use uwsgi

Bug #1686210 reported by Jerry Zhao
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
devstack
Fix Released
High
Ian Wienand

Bug Description

Since this change https://review.openstack.org/#/c/456344/15, keystone has been failing to start on centos7.

[Wed Apr 19 06:34:19.516447 2017] [core:notice] [pid 17582] AH00052: child pid 26997 exit signal Segmentation fault (11)

http://paste.openstack.org/show/607926/
The apache erorr log attached.

Revision history for this message
Ian Wienand (iwienand) wrote :

We've debugged this to the graceful reload in [1]. we can work around this with a restart

the reason it works in the devstack gate is that we have TLS enabled there, so apache gets a restart from that. so apache is in a bad state looping, but we don't notice.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1445540

Changed in devstack:
assignee: nobody → Ian Wienand (iwienand)
importance: Undecided → High
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to devstack (master)

Fix proposed to branch: master
Review: https://review.openstack.org/459904

Changed in devstack:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to devstack (master)

Reviewed: https://review.openstack.org/459904
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=f6a2d2cd4edd06408690081d6207ff73b76f543a
Submitter: Jenkins
Branch: master

commit f6a2d2cd4edd06408690081d6207ff73b76f543a
Author: Ian Wienand <email address hidden>
Date: Wed Apr 26 10:50:29 2017 +1000

    Always restart apache

    As described in [1], it seems that mod_wsgi is not "graceful" reload
    safe. Upon re-init, it can end up in a segfault loop.

    The "reload" (not *restart*) after setting up uwsgi was added with
    I1d89be1f1b36f26eaf543b99bde6fdc5701474fe but not causing an issue
    until uwsgi was enabled.

    We do not notice in the gate, because the TLS setup ends up doing a
    restart after this setup. In the period between the
    write_uwsgi_config and that restart, Apache is sitting in a segfault
    loop, but we never noticed because we don't try talking to it. Other
    jobs that don't do any further apache configuration have started
    failing, however.

    Looking at the original comments around "reload_apache_server" I'm not
    sure if it is still necessary. [2] shows it is not used outside these
    two calls.

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1445540
    [2] http://codesearch.openstack.org/?q=reload_apache_server&i=nope&files=&repos=

    Closes-Bug: #1686210
    Change-Id: I5234bae0595efdcd30305a32bf9c121072a3625e

Changed in devstack:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.