Exception: Error inspecting image: docker://docker.io/ server misbehaving

Bug #1795718 reported by wes hayutin on 2018-10-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Unassigned

Bug Description

2018-10-02 16:52:36.977 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] rendering j2 template to file: /home/zuul/tripleo-heat-installer-templates/./network/config/multiple-nics/compute-dvr.yaml
2018-10-02 16:52:36.977 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] jinja2 rendering normal template controller-v6.j2.yaml
2018-10-02 16:52:36.977 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] rendering j2 template to file: /home/zuul/tripleo-heat-installer-templates/./network/config/multiple-nics/controller-v6.yaml
2018-10-02 16:53:02.480 13286 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] Exception: Error inspecting image: docker://docker.io/tripleorocky/centos-binary-barbican-api:current-tripleo
time="2018-10-02T16:53:02Z" level=fatal msg="pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: server misbehaving"

2018-10-02 16:53:02.490 13286 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ] None
2018-10-02 16:53:03.657 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
2018-10-02 16:53:03.692 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning ansible directory /home/zuul/undercloud-ansible-vpyQEW
2018-10-02 16:53:03.718 13286 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Install artifact is located at /home/zuul/undercloud-install-20181002165302.tar.bzip2

http://logs.openstack.org/73/607173/1/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/703ce71/logs/undercloud/home/zuul/install-undercloud.log.txt.gz#_2018-10-02_16_53_02_480

Source:
http://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/image/image_uploader.py#n298

Maybe the msg needs to be updated?

    if process.returncode != 0:
            not_found_msgs = (
                u'manifest unknown',
                # returned by docker.io
                u'requested access to the resource is denied'
            )

Alex Schultz (alex-schultz) wrote :

So i think the issue here might be that unbound is not properly caching the registry-1.docker.io dns lookup due to the config.

http://logs.openstack.org/25/602425/5/check/tripleo-ci-centos-7-containers-multinode/e2bb2ec/logs/undercloud/etc/unbound/conf.d/ttl.conf.txt.gz
# Ansible managed

server:
    cache-min-ttl: 900
    cache-max-ttl: 86400

Unfortunately the TTL for docker.io is 900 so it's skipping the cache.

Oct 02 16:52:43 unbound[1731:1] info: incoming scrubbed packet: ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 0
;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
docker.io. IN DS

;; ANSWER SECTION:

;; AUTHORITY SECTION:
io. 900 IN SOA a0.nic.io. noc.afilias-nst.info. 1497465179 10800 3600 2764800 900

;; ADDITIONAL SECTION:
;; MSG SIZE rcvd: 90
http://logs.openstack.org/73/607173/1/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/703ce71/logs/undercloud/var/lib/unbound/unbound.log.txt.gz

wes hayutin (weshayutin) on 2018-10-08
tags: removed: alert
Changed in tripleo:
milestone: stein-1 → stein-2
Changed in tripleo:
milestone: stein-2 → stein-3
Changed in tripleo:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers