I'm running containers using the liberty2 release and have seen a couple of times the apache server in the keystone container not start up due to leftover state files in /var/run. This is running the kollaglue/centos-rdo-keystone:liberty2 pulled from the public Docker registry.
I'd like to see the cleanup of /var/run (or equivalent for other OS's) everywhere to avoid this happening to other services in other containers.
Here's a log from the latest startup of the container:
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
ERROR: database mysql is not available @ 192.168.1.3
Running the kolla-common script
Checking connectivity to the DB
database is active @ 192.168.1.3
Checking for required variables
Setting up Keystone DB
Configuring keystone.conf
Setting up PKI
No handlers could be found for logger "oslo_config.cfg"
2015-08-24 05:32:40.336 32 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/private/cakey.pem already exists
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Initializing Keystone DB
No handlers could be found for logger "oslo_config.cfg"
Starting Keystone
Verifying Keystone is running
waiting for Keystone @ http://192.168.1.3:35357/v2.0
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
httpd (pid 52) already running
waiting for Keystone @ http://192.168.1.3:35357/v2.0
waiting for Keystone @ http://192.168.1.3:35357/v2.0
The waiting for keystone message continues indefinitely.
I've logged in to the container and here's what's in /var/run/httpd:
tpot@userver:~$ docker exec -it compose_keystone_1 bash
[root@userver /]# ls /var/run/httpd/
authdigest_shm.52 htcacheclean httpd.pid wsgi.52.0.1.sock wsgi.52.0.2.sock
Process listing is:
[root@userver /]# ps auxww
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.1 0.0 11744 1596 ? Ss 01:01 0:00 /bin/bash /opt/kolla/config-internal.sh
root 743 0.7 0.0 13464 2024 ? Ss 01:07 0:00 bash
root 780 0.0 0.0 4320 368 ? S 01:08 0:00 sleep 1
root 781 0.0 0.0 21372 1308 ? R+ 01:08 0:00 ps auxww
Fix is to enter the container and rm -rf /var/run/httpd/* and then restart it. The log now looks more useful with just a single "waiting for Keystone" message.
Running the kolla-common script
Checking connectivity to the DB
database is active @ 192.168.1.3
Checking for required variables
Setting up Keystone DB
Configuring keystone.conf
Setting up PKI
No handlers could be found for logger "oslo_config.cfg"
2015-08-20 22:34:35.852 32 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/private/cakey.pem already exists
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Initializing Keystone DB
No handlers could be found for logger "oslo_config.cfg"
Starting Keystone
Verifying Keystone is running
waiting for Keystone @ http://192.168.1.3:35357/v2.0
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
keystone is active @ http://192.168.1.3:35357/v2.0
Creating Keystone tenant, user, role, service and endpoints
using existing tenant admin (e1a0d4522a1c4d6f81f79497394799f9)
using existing user admin (cab433e4f2304ec3996ae8983b9e123e)
updating enabled=True for user admin
updating password for user admin
using existing role admin (b0c0987b16b3424c847ca4e221fbaea3)
Just cutting and pasting comments from my gerrit submission I made incorrectly instead of filing a bug on launchpad:
Sam Yaple Aug 24 4:42 PM
Debian based systems do not use httpd, but instead apache2. Additionally, with different versions of packages and configuration options Debian-based systems *may* not be affected.
Either way, this will need to be in a block that only runs on Centos/RHEL/Fedora based systems.
Steven Dake Aug 24 10:15 PM
note there is an environment variable set in the docker files as follows;
https:/