httpd (pid 1) already running when starting keystone or horizon container

[Waldemar Znoinski]

After hard restarting (powercycle due to network misconfigured - connectivity lost) the physical machine started and all except for 3 containers didn't start - see below. Heat was not working before that but Horizon and Keystone have same/similar error (behave same). These container keep restartarting despite the apache2 -DFOREGROUND being executed with exit code 0 as per my debbugging.

1. world dump

Docker version 1.8.2, build 0a8c2e3

root@silvixa00385229:~]$ docker ps | grep -i restarting
57304b4517ea kollaglue/ubuntu-source-horizon:ubuntu14.04 "kolla_start" 13 days ago Restarting (0) 29 seconds ago horizon
75b01bd47706 kollaglue/ubuntu-source-heat-api:ubuntu14.04 "kolla_start" 13 days ago Restarting (1) 2 minutes ago heat_api
4c52c62f81a4 kollaglue/ubuntu-source-keystone:ubuntu14.04 "kolla_start" 13 days ago Restarting (0) 31 seconds ago keystone

docker logs horizon | tail -15
httpd (pid 1) already running
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Removing existing destination: /etc/apache2/conf-enabled/000-default.conf
INFO:__main__:Copying /var/lib/kolla/config_files/horizon.conf to /etc/apache2/conf-enabled/000-default.conf
INFO:__main__:Setting permissions for /etc/apache2/conf-enabled/000-default.conf
INFO:__main__:Removing existing destination: /etc/openstack-dashboard/local_settings
INFO:__main__:Copying /var/lib/kolla/config_files/local_settings to /etc/openstack-dashboard/local_settings
INFO:__main__:Setting permissions for /etc/openstack-dashboard/local_settings
INFO:__main__:Writing out command to execute
Running command: '/usr/sbin/apache2 -DFOREGROUND'
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
httpd (pid 1) already running

2. inside horizon container:
root@silvixa00385229:/var/run/apache2# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 18176 3064 ? Ss+ 16:02 0:00 /bin/bash
root 16 0.0 0.0 18180 3332 ? Ss 16:02 0:00 /bin/bash
root 43 0.0 0.0 15572 2116 ? R+ 16:03 0:00 ps aux
root@silvixa00385229:/var/run/apache2# ls -l /var/run/apache2/
total 4
-rw-r--r-- 1 root root 2 Mar 7 11:06 apache2.pid
srwx------ 1 www-data root 0 Mar 7 11:06 wsgi.1.0.1.sock
root@silvixa00385229:/var/run/apache2#

It looks like leftovers in /var/run/apache2 (a pid file) even when the apache2 is not actually running.

[Waldemar Znoinski (wznoinsk)]

removing /var/run/apache2/* inside horizon container fixes the problem

[Vikram Hosakote (vhosakot)]

Good test.

I think ansible/roles/horizon/tasks/start.yml must be enhanced to first delete all the old/stale files in /run/apache2 before starting the horizon container.

[Steven Dake (sdake)]

Is this master or liberty?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/295458
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=28cc39e2fee14d58831e49458a66d6da0a4cd406
Submitter: Jenkins
Branch: master

commit 28cc39e2fee14d58831e49458a66d6da0a4cd406
Author: SamYaple <email address hidden>
Date: Mon Mar 21 18:56:52 2016 +0000

    Fix bad apache2 close

    Change-Id: Id19450a566da1bf7b824f0050e7c3e3f73581a9b
    Closes-Bug: #1557036

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 2.0.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 1.1.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 3.0.0.0b1 development milestone.