containerized keystone intermittently fails undercloud: grpc: the connection is unavailable

[Bogdan Dobrelya]

Undercloud install failed to deploy keystone containers: https://pastebin.com/khZxCLZf

Observed in https://review.openstack.org/#/c/553427

http://logs.openstack.org/27/553427/9/check/tripleo-ci-centos-7-undercloud-containers/3e01f30/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

The issue might be not keystone/healthcheck specific though, given the intermittent nature, as it seems it is from the related (?) moby issues [0], [1].

Docker (1.13.1) info [2]

[0] https://github.com/moby/moby/issues/31849
[1] https://github.com/moby/moby/issues/31074
[2] http://logs.openstack.org/27/553427/9/check/tripleo-ci-centos-7-undercloud-containers/3e01f30/logs/undercloud/var/log/extra/docker/docker_allinfo.log.txt.gz

[Bogdan Dobrelya (bogdando)]

There is also messages pointing out that docker-containerd had bee restarted a few times:

logs/undercloud/var/log/journal.txt.gz:Mar 27 15:54:08 undercloud.localdomain dockerd-current[28158]: time="2018-03-27T15:54:08.008583683Z" level=info msg="libcontainerd: new containerd process, pid: 28168"
logs/undercloud/var/log/journal.txt.gz:Mar 27 15:57:59 undercloud.localdomain dockerd-current[28158]: time="2018-03-27T15:57:59.259077567Z" level=info msg="libcontainerd: new containerd process, pid: 4141"
logs/undercloud/var/log/journal.txt.gz:Mar 27 15:58:00 undercloud.localdomain dockerd-current[28158]: time="2018-03-27T15:58:00.014093381Z" level=info msg="libcontainerd: new containerd process, pid: 4211"

[Steve Baker (steve-stevebaker)]

Here it is happening during crond docker-puppet.py

http://logs.openstack.org/72/569672/5/gate/tripleo-ci-centos-7-undercloud-containers/d5cdc86/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-05-31_22_02_27

[Marios Andreou (marios-b)]

Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

[Launchpad Janitor (janitor)]

[Expired for tripleo because there has been no activity for 60 days.]