config.json is missing from keystone container after restart of keystone container

[James Bagwell]

Summary:
After deploying openstack stein (containerized) , a restart of the keystone containers on the overcloud controllers fails and the docker exits in status 2. The following error is seen below:

command: docker start keystone

docker logs keystone:
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
ERROR:__main__:Unexpected error:
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 412, in main
    config = load_config()
  File "/usr/local/bin/kolla_set_configs", line 294, in load_config
    config = load_from_file()
  File "/usr/local/bin/kolla_set_configs", line 282, in load_from_file
    with open(config_file) as f:
IOError: [Errno 2] No such file or directory: '/var/lib/kolla/config_files/config.json

docker ps -a | grep keystone shows the container exited (2).
"dumb-init --singl..." 3 hours ago Exited (2) 3 hours ago keystone

The config.json exists inside the container prior to the restart, but disappears completely after restart, resulting the error above.

Steps to reproduce:
1: Deploy Stein openstack containerized.
2: Restart keystone containers on overcloud controllers.
3: check the status of keystone container - in failed state.

Expected results:
Keystone container starts and remains healthy, which assumes the config.json exists in the container , in expected location of /var/lib/kolla/config_files/

The contents of config.json , which EXISTS before restarting keystone , contains:

()[root@overcloud-controller-bvt-0 config_files]# cat config.json
{
    "command": "/usr/sbin/httpd",
    "config_files": [
        {
            "dest": "/etc/keystone/fernet-keys",
            "merge": false,
            "preserve_properties": true,
            "source": "/var/lib/kolla/config_files/src/etc/keystone/fernet-keys"
        },
        {
            "dest": "/etc/httpd/conf.d",
            "merge": false,
            "preserve_properties": true,
            "source": "/var/lib/kolla/config_files/src/etc/httpd/conf.d"
        },
        {
            "dest": "/",
            "merge": true,
            "preserve_properties": true,
            "source": "/var/lib/kolla/config_files/src/*"
        }
    ]

Environment:

Openstack Stein - tripleo deployment , containerized. Centos 7.7.1908

[James Bagwell (jimbagwell)]

Have discovered that maybe there's a problem with our restart handler in ansible:

I have been using , to restart the container:
handlers:
    - name: restart keystone
      docker_container:
        name: keystone
        image: 172.31.0.1:8787/cbis/centos-binary-keystone:stein-latest
        command: "kolla_start"
        state: started
        restart: yes

When I dont use the ansible module above, but instead use shell (below) for the same task, the config.json remains intact on the container and the container starts.

  handlers:
    - name: restart keystone
      shell: "docker restart keystone"

[kobig (kobi.ginon)]

additional info:

any container that we restart with docker_container module with ansible-playbook
will result with a failure to get up back with the below error

Note: strange thing is that simple shell docker restart [container name] is successful
      we also tried to go to latest ansible-playbook version without success

An example for the ansible task
###############################
  - name: restart manila_api
    docker_container:
      name: manila_api
      image: 172.31.0.1:8787/cbis/centos-binary-manila-api:stein-latest
      command: "kolla_start"
      state: started
      restart: yes

####result of docker logs from the container

9f966dddc5b6 172.31.0.1:8787/cbis/centos-binary-manila-api:stein-latest "dumb-init --singl..." 13 minutes ago Exited (2) 13 minutes ago manila_api
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
ERROR:__main__:Unexpected error:
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 412, in main
    config = load_config()
  File "/usr/local/bin/kolla_set_configs", line 294, in load_config
    config = load_from_file()
  File "/usr/local/bin/kolla_set_configs", line 282, in load_from_file
    with open(config_file) as f:
IOError: [Errno 2] No such file or directory: '/var/lib/kolla/config_files/config.json'

################Ansible playbook version
############################
(undercloud) [stack@undercloud ~]$ ansible-playbook --version
ansible-playbook 2.8.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/stack/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.5 (default, Aug 7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

[wes hayutin (weshayutin)]

@Kevin, please have a look and help me get this into the right hands.