Failed to Starting kolla-toolbox container

[MarginHu]

when I run "kolla-ansible deploy -i inventory/multinode" , it reports failed to start kolla-toolbox container with "Unknown status message: Digest: sha256:684d4b4ac3988df5e49640e7097e697930588d8b42fcba70e8bece4998aa1d2f"

the original info is in http://pastebin.com/XRd4HkAQ

my kolla-ansible version: 4.0.0.0rc1
Image was built from centos binary.

[Steven Dake (sdake)]

need docker version
need ansible version
need output of the command docker images

[MarginHu (margin2017)]

the info is in http://pastebin.com/hb60Dgas

[MarginHu (margin2017)]

I find a way to solve the issue,
1.remove the docker image with "docker rmi "

2.rebuild the image with kolla-build

3.run the deploy command.

but I don't know the root cause.

Moreover,after apply above method, the error appeared on other images again.

TASK [common : Starting cron container] ****************************************
fatal: [kode5]: FAILED! => {"changed": false, "failed": true, "msg": "Unknown status message: Digest: sha256:bca287c5d30b5e370b755ab041ea9e1fcf110286f7c397038d0bf2600f1a2206"}
fatal: [kode4]: FAILED! => {"changed": false, "failed": true, "msg": "Unknown status message: Digest: sha256:bca287c5d30b5e370b755ab041ea9e1fcf110286f7c397038d0bf2600f1a2206"}

[MarginHu (margin2017)]

now, I met similar issue on kolla-toolbox image, but this time the workaround is not effective.

I found a clue on kode4 , /var/log/messages has the following info.
Feb 27 14:06:05 kode4 ansible-kolla_docker: Invoked with tls_key=None image=192.168.103.16:5000/bgi/centos-binary-kolla-toolbox:ocata-rc1-1 labels={} tls_verify=False pid_mode=None tls_cacert=None auth_password=None environment={'ANSIBLE_LIBRARY': '/usr/share/ansible', 'ANSIBLE_NOCOLOR': '1'} auth_registry=None volumes_from=None tls_cert=None privileged=True api_version=auto remove_on_exit=True restart_retries=10 detach=True auth_username=None name=kolla_toolbox security_opt=[] cap_add=[] restart_policy=None auth_email=None ipc_mode=None volumes=['/etc/kolla//kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'] action=start_container common_options={'auth_email': None, 'restart_policy': 'unless-stopped', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'auth_registry': '192.168.103.16:5000', 'restart_retries': '10', 'auth_password': None, 'auth_username': 'sam'}
Feb 27 14:06:05 kode4 dockerd-current: time="2017-02-27T14:06:05.889807416+08:00" level=info msg="{Action=auth, Username=root, LoginUID=0, PID=10667}"
Feb 27 14:06:05 kode4 dockerd-current: time="2017-02-27T14:06:05.892787982+08:00" level=info msg="Error logging in to v2 endpoint, trying next endpoint: Get https://192.168.103.16:5000/v2/: http: server gave HTTP response to HTTPS client"
Feb 27 14:06:05 kode4 dockerd-current: time="2017-02-27T14:06:05.904874073+08:00" level=info msg="{Action=create, Username=root, LoginUID=0, PID=10667}"
Feb 27 14:06:05 kode4 dockerd-current: time="2017-02-27T14:06:05.909848231+08:00" level=warning msg="Error getting v2 registry: Get https://192.168.103.16:5000/v2/: http: server gave HTTP response to HTTPS client"
Feb 27 14:06:05 kode4 dockerd-current: time="2017-02-27T14:06:05.909889123+08:00" level=error msg="Attempting next endpoint for pull after error: Get https://192.168.103.16:5000/v2/: http: server gave HTTP response to HTTPS client"
Feb 27 14:07:34 kode4 systemd-logind: Removed session 222.

[MarginHu (margin2017)]

when I rerun the deploy command , it reports a new error.

TASK [common : Starting kolla-toolbox container] *******************************
fatal: [kode5]: FAILED! => {"changed": true, "failed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_RjTukh/ansible_module_kolla_docker.py\", line 781, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_RjTukh/ansible_module_kolla_docker.py\", line 608, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/lib/python2.7/site-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/lib/python2.7/site-packages/docker/api/container.py\", line 368, in start\\n self._raise_for_status(res)\\n File \"/usr/lib/python2.7/site-packages/docker/client.py\", line 173, in _raise_for_status\\n raise errors.NotFound(e, response, explanation=explanation)\\nNotFound: 404 Client Error: Not Found (\"{\"message\":\"invalid header field value \\\\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\\\\\\\"process_linux.go:359: container init caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\"rootfs_linux.go:54: mounting \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/containers/e6c1ab3ea666a871f9c6416c397a520aedfd85b4a4e50c65a94ef98213048b4a/secrets\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" to rootfs \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/btrfs/subvolumes/32263df0ddae40923b60e2a78cc4a903ead21a7367a0498a510e3fc4e9c23896\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" at \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"/var/lib/docker/btrfs/subvolumes/32263df0ddae40923b60e2a78cc4a903ead21a7367a0498a510e3fc4e9c23896/run/secrets\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" caused \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"no such file or directory\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"\\\\\\\\\\\\\"\\\\\\\\n\\\\\"\"}\")\\n'"}
fatal: [kode4]: FAILED! => {"changed": true, "failed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_sLTnpX/ansible_module_kolla_docker.py\", line 781, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_sLTnpX/ansible_module_kolla_docker.py\", line 608, in start_container\\n self.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/lib/python2.7/site-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *args, **kwargs)\\n File \"/usr/lib/python2.7/site-packages/docker/api/container.py\", line 368, in start\\n self._raise_for_status(res)\\n File \"/usr/lib/python2.7/site-packages/docker/client.py\", line 173, in _raise_for_status\\n raise errors.NotFound(e, response, explanation=explanation)\\nNotFound: 404 Client Error: Not Found (\"{\"message\":\"invalid header field value \\\\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\...

[Eduardo Gonzalez (egonzalez90)]

The issue with HTTPS response is likely a misconfiguration of docker daemon regarding to insecure-registry. Please, follow this steps to configure insecure registry in all nodes.
https://github.com/openstack/kolla-ansible/blob/master/doc/multinode.rst#configure-docker-on-all-nodes

[MarginHu (margin2017)]

@Eduardo，you're right about the issue in comment #6.

but after I fix the issue #6, origin issue is existed,Finally I find that the issue is disappeared after I use docker-engine-1.12.6-1.el7.centos.x86_64 this docker community version instead of redhat docker version docker-1.12.5-14.el7.centos.

[MarginHu (margin2017)]

the issue at comment #6 has reproduced, I use docker-1.12.6-16.el7.centos.x86_64 in centos repository instead of docker-engine-1.12.6-1.el7.centos.x86_64.
because I found system often trigger nmi deadlock If run docker-engine-1.12.6-1.el7.centos.x86_64

so now I must solve the issue, I google it is similar with https://bugzilla.redhat.com/show_bug.cgi?id=1410118

can you help me ?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b2 development milestone.

[Jeffrey Zhang (jeffrey4l)]

disable duo to lack more info.