tripleo

contacting container registry fails even w/ unbound, requires retry

Bug #1779213 reported by wes hayutin on 2018-06-28

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	High	Emilien Macchi	tripleo rocky-3

Bug Description

2018-06-28 18:27:52 | Exception: Error inspecting image: docker://docker.io/tripleomaster/centos-binary-heat-engine:current-tripleo
2018-06-28 18:27:52 | time="2018-06-28T18:27:51Z" level=fatal msg="pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: no such host"

Code in play
http://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/image/image_uploader.py#n325

Logs
======

http://logs.openstack.org/13/528213/42/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/58315c1/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-06-28_18_27_52

http://logs.openstack.org/13/528213/42/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/58315c1/logs/undercloud/etc/resolv.conf.txt.gz

http://logs.openstack.org/13/528213/42/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/58315c1/logs/undercloud/etc/unbound/conf.d/ttl.conf.txt.gz

Tags:

Steve Baker (steve-stevebaker) on 2018-06-28

Changed in tripleo:
assignee:	nobody → Steve Baker (steve-stevebaker)
tags:	added: ci

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-07-03: Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.openstack.org/579734

Changed in tripleo:
status:	Triaged → In Progress

OpenStack Infra (hudson-openstack) on 2018-07-04

Changed in tripleo:
assignee:	Steve Baker (steve-stevebaker) → Emilien Macchi (emilienm)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-07-04: Fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/579734
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=2dbcc9036aed8f47467e81b028f2fae76dd28816
Submitter: Zuul
Branch: master

commit 2dbcc9036aed8f47467e81b028f2fae76dd28816
Author: Steve Baker <email address hidden>
Date: Tue Jul 3 14:08:08 2018 +1200

Retry inspect of container images

    This change parses the error output of a failed skopeo inspect to
    detect if it is a missing image error, then adds tenacity retries for all
    other error types.

Failures are being seen in CI due to transient AWS DNS issues, so this change
should make image inspections more resilient.

Change-Id: If5ad384cfc8783c3b10c4cc567e42f06e5c6880b
Closes-Bug: #1779213

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-07-12: Fix proposed to tripleo-common (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/582297

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-07-13: Fix merged to tripleo-common (stable/queens)

Reviewed: https://review.openstack.org/582297
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=bd92a6ad476caf16e8d9dbf18d290f260b00e94b
Submitter: Zuul
Branch: stable/queens

commit bd92a6ad476caf16e8d9dbf18d290f260b00e94b
Author: Steve Baker <email address hidden>
Date: Tue Jul 3 14:08:08 2018 +1200

Retry inspect of container images

    This change parses the error output of a failed skopeo inspect to
    detect if it is a missing image error, then adds tenacity retries for all
    other error types.

Failures are being seen in CI due to transient AWS DNS issues, so this change
should make image inspections more resilient.

    Change-Id: If5ad384cfc8783c3b10c4cc567e42f06e5c6880b
    Closes-Bug: #1779213
    (cherry picked from commit 2dbcc9036aed8f47467e81b028f2fae76dd28816)