heat_api and heat_api_cfn docker container has "Restarting" state on controller node

[Artem Hrechanychenko]

Description
===========

[heat-admin@overcloud-controller-0 ~]$ sudo docker ps |grep heat_api
42e84db16e96 192.168.24.1:8787/tripleoupstream/centos-binary-heat-api-cfn:latest "kolla_start" 53 minutes ago Restarting (1) 26 minutes ago heat_api_cfn
7a9d2c48337a 192.168.24.1:8787/tripleoupstream/centos-binary-heat-api:latest "kolla_start" 54 minutes ago Restarting (1) 26 minutes ago heat_api

http://pastebin.test.redhat.com/492689
http://pastebin.test.redhat.com/492691

Steps to reproduce
==================
Deploy overcloud Ocata

openstack overcloud deploy \
    --libvirt-type qemu \
    --ntp-server clock.redhat.com \
    --control-scale 1 \
    --control-flavor oooq_control \
    --compute-flavor oooq_compute \
    --templates $THT \
    -e $THT/environments/low-memory-usage.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
-e /home/stack/network-environment.yaml

Upgrade Undercloud node

### UPGRADE UNDERCLOUD ###

    # master repos
    cd /home/stack/ && source stackrc

    sudo curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-master/current-passed-ci/delorean.repo
    sudo curl -L -o /etc/yum.repos.d/delorean-current.repo https://trunk.rdoproject.org/centos7/current/delorean.repo
    sudo sed -i 's/\[delorean\]/\[delorean-current\]/' /etc/yum.repos.d/delorean-current.repo
    sudo /bin/bash -c 'printf "\nincludepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common*,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,puppet-*" >> /etc/yum.repos.d/delorean-current.repo'
    sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo

    sudo systemctl stop openstack-*
    sudo systemctl stop neutron-*
    sudo systemctl stop httpd

    sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient
openstack undercloud upgrade

 # create an envrionment file to make overcloud fetch the images from the undercloud
    # (192.168.24.1 is undercloud IP that must be pingable from the overcloud)
    echo > ~/containers-default-parameters.yaml 'parameter_defaults:
      DockerNamespace: 192.168.24.1:8787/tripleoupstream
      DockerNamespaceIsRegistry: true
    '
    sleep 3
    printf "\n"

    cat > ~/containers-upgrade-repos.yaml <<'EOEF'
parameter_defaults:
  UpgradeInitCommand: |
    set -ex
    pushd /etc/yum.repos.d/
    rm -rf delorean*
    REPO_PREFIX=/etc/yum.repos.d
    DELOREAN_REPO_URL=https://trunk.rdoproject.org/centos7/current-tripleo
    DELOREAN_REPO_FILE=delorean.repo
    sudo curl -Lvo $REPO_PREFIX/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo
    sudo sed -i -e 's%priority=.*%priority=30%' $REPO_PREFIX/delorean-deps.repo
    cat $REPO_PREFIX/delorean-deps.repo
    # Enable last known good RDO Trunk Delorean repository
    sudo curl -Lvo $REPO_PREFIX/delorean.repo $DELOREAN_REPO_URL/$DELOREAN_REPO_FILE
    sudo sed -i -e 's%priority=.*%priority=20%' $REPO_PREFIX/delorean.repo
    cat $REPO_PREFIX/delorean.repo
    # Enable latest RDO Trunk Delorean repository
    sudo curl -Lvo $REPO_PREFIX/delorean-current.repo https://trunk.rdoproject.org/centos7/current/delorean.repo
    sudo sed -i -e 's%priority=.*%priority=10%' $REPO_PREFIX/delorean-current.repo
    sudo sed -i 's/\[delorean\]/\[delorean-current\]/' $REPO_PREFIX/delorean-current.repo
    sudo /bin/bash -c "cat <<-EOF>>$REPO_PREFIX/delorean-current.repo
    includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common*,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,puppet-*
    EOF"
    cat $REPO_PREFIX/delorean-current.repo
    popd
    yum clean all
EOEF

download_docker_images_to_local_registry
    cd /home/stack/ && source stackrc
    sudo openstack overcloud container image upload --verbose --config-file /usr/share/tripleo-common/container-images/overcloud_containers.yaml

Upgrade overcloud
 cd /home/stack/ && source stackrc
    export THT=/usr/share/openstack-tripleo-heat-templates
    openstack overcloud deploy \
    --libvirt-type qemu \
    --ntp-server clock.redhat.com \
    --control-scale 1 \
    --control-flavor oooq_control \
    --compute-flavor oooq_compute \
    --templates $THT \
    -e $THT/environments/low-memory-usage.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
    -e /home/stack/network-environment.yaml \
    -e $THT/environments/docker.yaml \
    -e $THT/environments/major-upgrade-composable-steps-docker.yaml \
    -e ~/containers-default-parameters.yaml \
-e ~/containers-upgrade-repos.yaml

Check docker containers state on overcloud nodes

Expected result
===============
Containers has running state

Actual result
=============
heat-api and heat-api-cfn containers has restarting state

[Artem Hrechanychenko (ahrechan)]

http://paste.openstack.org/show/612280/
http://paste.openstack.org/show/612279/