tripleo-container-image-prepare fails when there is a readtimeout

[Alex Schultz]

The tripleo-container-image-prepare lookup for image labels does not appear to be retried and causes installation failures. If I'm reading the code correctly we only retry for ImageUploaderExceptions but the ReadTimeout exception isn't included.

See https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/image_uploader.py#L363

Failure in the gate:

http://logs.openstack.org/94/608094/5/gate/tripleo-ci-centos-7-containers-multinode/6903d01/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

Traceback (most recent call last):
  File "/usr/bin/tripleo-container-image-prepare", line 119, in <module>
    env, roles_data, cleanup=args.cleanup, dry_run=args.dry_run)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 183, in container_images_prepare_multi
    modify_only_with_labels=modify_only_with_labels,
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 288, in container_images_prepare
    images, modify_only_with_labels)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 526, in filter_images_with_labels
    url, insecure=insecure, session=session)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 302, in _image_labels
    i = BaseImageUploader._inspect(image_url, insecure, session)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 241, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 330, in call
    start_time=start_time)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 279, in iter
    return fut.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 333, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 416, in _inspect
    config_r = config_f.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 429, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 62, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 525, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 644, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 222, in resolve_redirects
    **adapter_kwargs
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 526, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='production.cloudflare.docker.com', port=443): Read timed out. (read timeout=30)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/611217

[Noam Angel (noama)]

I also get this with rocky

[Steve Baker (steve-stevebaker)]

The code has just landed, can you please create a new bug for the rocky issue and attach a stack trace?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/611217
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=c10ccfd0cc40ba6549e583b576378d5344fac596
Submitter: Zuul
Branch: master

commit c10ccfd0cc40ba6549e583b576378d5344fac596
Author: Steve Baker <email address hidden>
Date: Wed Oct 17 15:32:44 2018 +1300

    Improve inspect error retry behaviour

    This makes the following changes:
    - retries on any requests exceptions instead of ImageUploaderException
      which is not raised in _inspect
    - adds retry decorator to the authenticate function, also on requests
      exceptions only
    - when authenticate first pings the endpoint, call raise_for_status if
      the status code is not 200 or 401.

    Change-Id: If4f619e35042283313912ff681d86a307ecea97e
    Closes-Bug: #1798232

[Noam Angel (noama)]

cmd:
sudo openstack tripleo container image prepare --verbose --output-env-file /home/stack/jenkins/workspace/deploy_overcloud/rocky/scripts/../templates/containers-default-parameters.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml -e /home/stack/jenkins/workspace/deploy_overcloud/rocky/scripts/../templates/containers-prepare-parameter.yaml --roles-file /home/stack/jenkins/workspace/deploy_overcloud/rocky/scripts/../templates/roles_data.yaml

log:
Completed upload for image docker.io/tripleorocky/centos-binary-nova-libvirt:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
imagename: docker.io/tripleorocky/centos-binary-swift-proxy-server:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
Running skopeo inspect --tls-verify=false docker://192.168.24.1:8787/tripleorocky/centos-binary-swift-proxy-server:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5-updated
Pulling docker.io/tripleorocky/centos-binary-swift-proxy-server
Running ansible-playbook command: ['ansible-playbook', '-vvvvv', '/tmp/tripleo-modify-image-playbook-s2OHKG/playbook.yaml']
Completed upload for image docker.io/tripleorocky/centos-binary-sensu-client:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
imagename: docker.io/tripleorocky/centos-binary-zaqar:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
Running skopeo inspect --tls-verify=false docker://192.168.24.1:8787/tripleorocky/centos-binary-zaqar:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5-updated
Pulling docker.io/tripleorocky/centos-binary-zaqar
Pushing 192.168.24.1:8787/tripleorocky/centos-binary-skydive-agent
Running ansible-playbook command: ['ansible-playbook', '-vvvvv', '/tmp/tripleo-modify-image-playbook-oYztxQ/playbook.yaml']
Completed upload for image docker.io/tripleorocky/centos-binary-swift-account:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
imagename: docker.io/ceph/daemon:v3.1.0-stable-3.1-luminous-centos-7-x86_64
Running skopeo inspect --tls-verify=false docker://192.168.24.1:8787/ceph/daemon:v3.1.0-stable-3.1-luminous-centos-7-x86_64-updated
Pulling docker.io/ceph/daemon
Completed upload for image docker.io/tripleorocky/centos-binary-swift-container:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
Completed upload for image docker.io/tripleorocky/centos-binary-swift-object:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
Completed upload for image docker.io/tripleorocky/centos-binary-skydive-analyzer:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
Completed upload for image docker.io/tripleorocky/centos-binary-skydive-agent:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
Pushing 192.168.24.1:8787/tripleorocky/centos-binary-swift-proxy-server
Running ansible-playbook command: ['ansible-playbook', '-vvvvv', '/tmp/tripleo-modify-image-playbook-hWXaCh/playbook.yaml']
Pushing 192.168.24.1:8787/tripleorocky/centos-binary-zaqar
Completed upload for image docker.io/tripleorocky/centos-binary-swift-proxy-server:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
Pushing 192.168.24.1:8787/ceph/daemon
Completed upload for image docker.io/tripleorocky/centos-binary-zaqar:31ff628fbefca9bd1d5181e500ad7bed9a471b2a_1bb5bbf5
Completed upload for image docker.io/ceph/daemon:v3...

[Steve Baker (steve-stevebaker)]

Could you raise a new bug please with this error log?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 10.1.0 release.