"Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: server misbehaving" while Trying to pull repository docker.io/ceph/daemon

[wes hayutin]

FYI.. it appears that some of the promotion jobs are hitting issues in the overcloud deployment when attempting to pull the ceph container from docker.io.

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/2ba379b/undercloud/var/log/extra/errors.txt.gz#_2018-03-02_11_55_39_975618

The overcloud deployment should not pull containers from docker.io directly specifically for this reason.

This is a bug in the config of the deployment:
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/2ba379b/undercloud/home/jenkins/containers-default-parameters.yaml.txt.gz

  DockerCephDaemonImage: docker.io/ceph/daemon:tag-stable-3.0-luminous-centos-7

Should be:

  DockerCephDaemonImage: 192.168.24.1:8787/master/daemon:tag-stable-3.0-luminous-centos-7

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/549216

[John Trowbridge (trown)]

We have always handled ceph this way, so I am not sure what is new here. That said, I agree it is incorrect to download the ceph container to the undercloud only to never actually use it from there.

https://review.openstack.org/549216 hard codes the ceph namespace for overcloud deploy to be from the undercloud registry as we do for all the other containers.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549216
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=6d378a625374a3a445f3c0d380fde2c6fc1fe937
Submitter: Zuul
Branch: master

commit 6d378a625374a3a445f3c0d380fde2c6fc1fe937
Author: John Trowbridge <email address hidden>
Date: Fri Mar 2 08:48:00 2018 -0500

    Use undercloud registry for ceph in overcloud deployment

    We download all of the containers to the undercloud (including ceph),
    but then we still pull ceph from docker.io during overcloud deploy.

    Instead we should hardcode the namespace for the overcloud deploy
    to be the undercloud registry (as we do for all the other containers).

    Change-Id: I7cf95bfee7267c7b1a451de20129dbe22d946834
    Closes-Bug: #1752874

[chandan kumar (chkumar246)]

in stable/pike noop jobs From http://logs.openstack.org/85/564285/29/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/003d01b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-11_01_27_19

2018-07-11 01:27:19 | 2018-07-11 01:27:10Z [overcloud]: CREATE_FAILED Resource CREATE failed: resources.AllNodesDeploySteps: Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR

After digging more, we found that: http://logs.openstack.org/85/564285/29/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/003d01b/logs/undercloud/var/log/mistral/ceph-install-workflow.log.txt.gz#_2018-07-11_01_26_34_125

2018-07-11 01:26:34,125 p=11925 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.0.3-stable-3.0-jewel-centos-7-x86_64 image (3 retries left).
2018-07-11 01:26:44,944 p=11925 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.0.3-stable-3.0-jewel-centos-7-x86_64 image (2 retries left).
2018-07-11 01:26:56,001 p=11925 u=mistral | FAILED - RETRYING: pulling docker.io/ceph/daemon:v3.0.3-stable-3.0-jewel-centos-7-x86_64 image (1 retries left).
2018-07-11 01:27:06,881 p=11925 u=mistral | fatal: [192.168.24.9]: FAILED! => {"attempts": 3, "changed": false, "cmd": ["timeout", "300s", "docker", "pull", "docker.io/ceph/daemon:v3.0.3-stable-3.0-jewel-centos-7-x86_64"], "delta": "0:00:00.428785", "end": "2018-07-11 01:27:06.726508", "msg": "non-zero return code", "rc": 1, "start": "2018-07-11 01:27:06.297723", "stderr": "error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/f1/f19fe1c264b51153863c03539124928d04b59919b4aeea28cf807122f2052bd2/data?verify=1531275426-cdP6wPgskL0k2WfQU%2FaXl6%2F7zmQ%3D: dial tcp: lookup production.cloudflare.docker.com on 127.0.0.1:53: server misbehaving", "stderr_lines": ["error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/f1/f19fe1c264b51153863c03539124928d04b59919b4aeea28cf807122f2052bd2/data?verify=1531275426-cdP6wPgskL0k2WfQU%2FaXl6%2F7zmQ%3D: dial tcp: lookup production.cloudflare.docker.com on 127.0.0.1:53: server misbehaving"], "stdout": "Trying to pull repository docker.io/ceph/daemon ... \nv3.0.3-stable-3.0-jewel-centos-7-x86_64: Pulling from docker.io/ceph/daemon\n469cfcc7a4b3: Pulling fs layer\nb33fc2819b5f: Pulling fs layer\n22c964f95d0b: Pulling fs layer\n4a0859b511e1: Pulling fs layer\nbb2f6e7d7d49: Pulling fs layer\nb4b0f148e402: Pulling fs layer\nd9b052310e3f: Pulling fs layer\n4a0859b511e1: Waiting\nbb2f6e7d7d49: Waiting\nb4b0f148e402: Waiting\nddf7aae59323: Pulling fs layer\nd9b052310e3f: Waiting\nddf7aae59323: Waiting", "stdout_lines": ["Trying to pull repository docker.io/ceph/daemon ... ", "v3.0.3-stable-3.0-jewel-centos-7-x86_64: Pulling from docker.io/ceph/daemon", "469cfcc7a4b3: Pulling fs layer", "b33fc2819b5f: Pulling fs layer", "22c964f95d0b: Pulling fs layer", "4a0859b511e1: Pulling fs layer", "bb2f6e7d7d49: Pulling fs layer", "b4b0f148e402: Pulling fs layer", "d9b052310e3f: Pulling fs layer", "4a0859b511e1: Waiting", "bb2f6e7d7d49: Waiting", "b4b0f148e402: Waiting", "ddf7aae59323: Pulling...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/581607

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chandan Kumar (<email address hidden>) on branch: master
Review: https://review.openstack.org/581607
Reason: As ceph is not used on undercloud, so this patch is not needed

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.