Latest devstack deploy on fedora fails with error

[Rabi Mishra]

Standard devstack deploy on fedora25 fails with the below error.

[Call Trace]
./stack.sh:1072:start_keystone
/home/stack/devstack/lib/keystone:563:die
[ERROR] /home/stack/devstack/lib/keystone:563 keystone did not start
Error on exit
World dumping... see /opt/stack/logs/worlddump-2017-04-20-034323.txt for details

It seems we've defaulted to use uwsgi and unix domain sockets recently and from the httpd error log I can see the following error.

[Thu Apr 20 10:02:10.077975 2017] [:error] [pid 2301] [client 192.168.0.71:42384] failed to make connection to backend: httpd-UDS:0
[Thu Apr 20 10:02:11.137116 2017] [proxy:error] [pid 2304] (2)No such file or directory: AH02454: uwsgi: attempt to connect to Unix domain socket /tmp/keystone-wsgi-public.socket (uwsgi-uds-keystone-wsgi-public) failed

So it seems httpd is not able to see the sockets due to PrivateTemp setting of httpd systemd unit file.

[Unit]
....
[Service]
....
PrivateTmp=true

[Ian Wienand (iwienand)]

Yeah, I think centos has a similar issue in the gate. I haven't debugged it yet but let's keep looking...

[Rabi Mishra (rabi)]

I've fixed it locally by creating a custom dir /var/sockets with a+rwx permissions and used it here[1] and it works fine. But wanted to see if there is a better solution before I push the patch.

[1] https://github.com/openstack-dev/devstack/blob/master/lib/apache#L233

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/458373

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/458373
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=aa26baacb8206967d61aef74686be94e3f8c90b8
Submitter: Jenkins
Branch: master

commit aa26baacb8206967d61aef74686be94e3f8c90b8
Author: rabi <email address hidden>
Date: Thu Apr 20 10:55:16 2017 +0530

    Create custom dir for uwsgi domain sockets

    On Centos, apache has a private view of /tmp and thus can't see this
    socket, causing keystone to fail. This happened after
    I46294fb24e3c23fa19fcfd7d6c9ee8a932354702.

    Move it to /var/run.

    Closes-Bug: #1684360
    Change-Id: I47f091656802719c259752454ec88bf50760b967

[souig amel (soumel)]

can anyone explain it to me i am new in linux;
so you go to this "I46294fb24e3c23fa19fcfd7d6c9ee8a932354702." you download the file and move it to /var/run? that's it will it resolve that problem?

[Tom Barron (tpb)]

Hmm, I've got a CentOS job [1] still failing in gate today with "keystone did not start" [2] even though it's running a devstack with this fix [3].

About five lines into the two keystone screenlogs [4] [5] there is a "No such file or directory" message w.r.t. an open of the path "./python_plugin.so".

Do we also need a local view of the python plugin?

[1] http://logs.openstack.org/77/456577/3/check/gate-manila-tempest-minimal-dsvm-lvm-centos-7-nv/de6d69b/

[2] http://logs.openstack.org/77/456577/3/check/gate-manila-tempest-minimal-dsvm-lvm-centos-7-nv/de6d69b/logs/devstacklog.txt.gz#_2017-04-24_11_18_39_963

[3] http://logs.openstack.org/77/456577/3/check/gate-manila-tempest-minimal-dsvm-lvm-centos-7-nv/de6d69b/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-04-24_11_11_29_028

[4] http://logs.openstack.org/77/456577/3/check/gate-manila-tempest-minimal-dsvm-lvm-centos-7-nv/de6d69b/logs/screen-key-a.txt.gz

[5] http://logs.openstack.org/77/456577/3/check/gate-manila-tempest-minimal-dsvm-lvm-centos-7-nv/de6d69b/logs/screen-key-p.txt.gz

[Tom Barron (tpb)]

Ian remarked to me on IRC that the "No such file or directory" business about "./python_plugin.so" is expected with pip install of uwsgi and is not a clue as to the failure, and indeed I am able to run devstack locally on CentOS 7.3 and keystone comes up fine even though it has that warning.

Locally I am running with screen:

2017-04-19 01:46:09.947 | +^[[38;5;242mfunctions-common:run_process:1563 ^[(B^[[m screen_process key-a '/usr/bin/uwsgi --ini /etc/keystone/keystone-uwsgi-admin.ini' ''

whereas in gate we have:

2017-04-24 11:17:39.587 | + lib/keystone:start_keystone:551 : run_process key-a '/usr/bin/uwsgi --ini /etc/keystone/keystone-uwsgi-admin.ini' ''

and hit the timeout waiting for keystone to start. I will restack locally with USE_SCREEN=False and compare logs again.

[golfreeze (golfreeze)]

###temp fixed for me.
cd /var/run
mkdir uwsgi
chmod 777 uwsgi

### restart keystone from devstack again
systemctl restart <email address hidden>

Try to login your dashboard.