Comment 4 for bug 1752874

in stable/pike noop jobs From http://logs.openstack.org/85/564285/29/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/003d01b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-11_01_27_19

2018-07-11 01:27:19 | 2018-07-11 01:27:10Z [overcloud]: CREATE_FAILED Resource CREATE failed: resources.AllNodesDeploySteps: Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR

After digging more, we found that: http://logs.openstack.org/85/564285/29/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/003d01b/logs/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz#_2018-07-11_01_26_34_125

2018-07-11 01:26:34,125 p=11925 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.0.3-stable-3.0-jewel-centos-7-x86_64 image (3 retries left).
2018-07-11 01:26:44,944 p=11925 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.0.3-stable-3.0-jewel-centos-7-x86_64 image (2 retries left).
2018-07-11 01:26:56,001 p=11925 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.0.3-stable-3.0-jewel-centos-7-x86_64 image (1 retries left).
2018-07-11 01:27:06,881 p=11925 u=mistral | fatal: [192.168.24.9]: FAILED! => {"attempts": 3, "changed": false, "cmd": ["timeout", "300s", "docker", "pull", "docker.io/ceph/daemon:v3.0.3-stable-3.0-jewel-centos-7-x86_64"], "delta": "0:00:00.428785", "end": "2018-07-11 01:27:06.726508", "msg": "non-zero return code", "rc": 1, "start": "2018-07-11 01:27:06.297723", "stderr": "error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/f1/f19fe1c264b51153863c03539124928d04b59919b4aeea28cf807122f2052bd2/data?verify=1531275426-cdP6wPgskL0k2WfQU%2FaXl6%2F7zmQ%3D: dial tcp: lookup production.cloudflare.docker.com on 127.0.0.1:53: server misbehaving", "stderr_lines": ["error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/f1/f19fe1c264b51153863c03539124928d04b59919b4aeea28cf807122f2052bd2/data?verify=1531275426-cdP6wPgskL0k2WfQU%2FaXl6%2F7zmQ%3D: dial tcp: lookup production.cloudflare.docker.com on 127.0.0.1:53: server misbehaving"], "stdout": "Trying to pull repository docker.io/ceph/daemon ... \nv3.0.3-stable-3.0-jewel-centos-7-x86_64: Pulling from docker.io/ceph/daemon\n469cfcc7a4b3: Pulling fs layer\nb33fc2819b5f: Pulling fs layer\n22c964f95d0b: Pulling fs layer\n4a0859b511e1: Pulling fs layer\nbb2f6e7d7d49: Pulling fs layer\nb4b0f148e402: Pulling fs layer\nd9b052310e3f: Pulling fs layer\n4a0859b511e1: Waiting\nbb2f6e7d7d49: Waiting\nb4b0f148e402: Waiting\nddf7aae59323: Pulling fs layer\nd9b052310e3f: Waiting\nddf7aae59323: Waiting", "stdout_lines": ["Trying to pull repository docker.io/ceph/daemon ... ", "v3.0.3-stable-3.0-jewel-centos-7-x86_64: Pulling from docker.io/ceph/daemon", "469cfcc7a4b3: Pulling fs layer", "b33fc2819b5f: Pulling fs layer", "22c964f95d0b: Pulling fs layer", "4a0859b511e1: Pulling fs layer", "bb2f6e7d7d49: Pulling fs layer", "b4b0f148e402: Pulling fs layer", "d9b052310e3f: Pulling fs layer", "4a0859b511e1: Waiting", "bb2f6e7d7d49: Waiting", "b4b0f148e402: Waiting", "ddf7aae59323: Pulling fs layer", "d9b052310e3f: Waiting", "ddf7aae59323: Waiting"]}

Looks like we are again seeing this issue.