container image prepare does not fail correctly if there are problems downloading images

Bug #1835837 reported by Alex Schultz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Alex Schultz

Bug Description

The tripleo-container-image-prepare does not actually exit with an error code to indicate a failure has occurred. What ends up happening is a deployment failure occurs later when the images are missing.

Something like this ends up in the output logs:

2019-07-08 19:42:32.909 21842 ERROR root [ ] Image prepare failed: 401 Client Error: Unauthorized for url: http://mirror.bhs1.ovh.openstack.org:8082/v2/tripleomaster/centos-binary-cinder-api/blobs/sha256:5112c01f82905602dc09b071dd4b8a47652e8b7e9e0fd013471a830fc7fdf270
Traceback (most recent call last):
  File "/usr/bin/tripleo-container-image-prepare", line 132, in <module>
    env, roles_data, cleanup=args.cleanup, dry_run=args.dry_run)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/kolla_builder.py", line 225, in container_images_prepare_multi
    uploader.upload()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 206, in upload
    uploader.run_tasks()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 1701, in run_tasks
    for result in p.map(upload_task, self.upload_tasks):
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 605, in result_iterator
    yield future.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 429, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/thread.py", line 62, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 1753, in upload_task
    return uploader.upload_image(task)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 990, in upload_image
    multi_arch=t.multi_arch
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 292, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 358, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 331, in iter
    raise retry_exc.reraise()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 167, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
    return self.__get_result()
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 361, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 1253, in _copy_registry_to_registry
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 401 Client Error: Unauthorized for url: http://mirror.bhs1.ovh.openstack.org:8082/v2/tripleomaster/centos-binary-cinder-api/blobs/sha256:5112c01f82905602dc09b071dd4b8a47652e8b7e9e0fd013471a830fc7fdf270

The deployment continues and ultimately fails when trying to pull down the containers:

2019-07-08 19:50:45 | "Error: error pulling image \"192.168.24.1:8787/tripleomaster/centos-binary-swift-proxy-server:aec64907ec7d9bd9bed48dad9b878d7773c91943_27a89b3f-updated-20190708191242\": unable to pull 192.168.24.1:8787/tripleomaster/centos-binary-swift-proxy-server:aec64907ec7d9bd9bed48dad9b878d7773c91943_27a89b3f-updated-20190708191242: unable to pull image: Error initializing source docker://192.168.24.1:8787/tripleomaster/centos-binary-swift-proxy-server:aec64907ec7d9bd9bed48dad9b878d7773c91943_27a89b3f-updated-20190708191242: Error reading manifest aec64907ec7d9bd9bed48dad9b878d7773c91943_27a89b3f-updated-20190708191242 in 192.168.24.1:8787/tripleomaster/centos-binary-swift-proxy-server: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/tripleomaster/centos-binary-swift-proxy-server/manifests/aec64907ec7d9bd9bed48dad9b878d7773c91943_27a89b3f-updated-20190708191242 was not found on this server.</p>\\n</body></html>\\n\"",

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.opendev.org/669766

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/669766
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=4596f0f332a8ba6e03399c994cd9e6a7803f44f3
Submitter: Zuul
Branch: master

commit 4596f0f332a8ba6e03399c994cd9e6a7803f44f3
Author: Alex Schultz <email address hidden>
Date: Mon Jul 8 16:03:32 2019 -0600

    Exit 1 when container image prepare fails

    Currently if an exception is thrown during the container image prepare,
    we log it but we still return 0. Anything that is running the command
    doesn't know it had issues.

    Change-Id: I87be8c3c7268399d86856eb47e801258d117a172
    Closes-Bug: #1835837

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/669884

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/stein)

Reviewed: https://review.opendev.org/669884
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=10d504871b40724f0d8e24dd39447a5935f1b428
Submitter: Zuul
Branch: stable/stein

commit 10d504871b40724f0d8e24dd39447a5935f1b428
Author: Alex Schultz <email address hidden>
Date: Mon Jul 8 16:03:32 2019 -0600

    Exit 1 when container image prepare fails

    Currently if an exception is thrown during the container image prepare,
    we log it but we still return 0. Anything that is running the command
    doesn't know it had issues.

    Change-Id: I87be8c3c7268399d86856eb47e801258d117a172
    Closes-Bug: #1835837
    (cherry picked from commit 4596f0f332a8ba6e03399c994cd9e6a7803f44f3)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 11.1.0

This issue was fixed in the openstack/tripleo-common 11.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 10.8.1

This issue was fixed in the openstack/tripleo-common 10.8.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.