tripleo

container builds fails randomly with ERROR:- error locating item named "manifest" for image with ID "c5d52ad8e571fc7e9105cd5852abd9abea732b8c870d30893da4e87ce6d9c3b8": file does not exist\

Bug #1894856 reported by yatin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Emilien Macchi
tripleo victoria-3 "tripleo victoria"

Bug Description

The job fails with TraceBack like below:-
2020-09-07 10:19:21 | Exception information: Unexpected error while running command.
2020-09-07 10:19:21 | Command: sudo buildah bud --volume /etc/yum.repos.d:/etc/yum.repos.d:z --volume /etc/pki/rpm-gpg:/etc/pki/rpm-gpg:z --format docker --tls-verify=False --layers --logfile /tmp/container-builds/55a84b8a-cd40-4e6c-a7cb-39b6daad7b32/base/os/placement-api/placement-api-build.log -t 127.0.0.1:8787/tripleomaster/openstack-placement-api:04d7e9b3125e0f17e08e43271a62bb96 /tmp/container-builds/55a84b8a-cd40-4e6c-a7cb-39b6daad7b32/base/os/placement-api
2020-09-07 10:19:21 | Exit code: 1
2020-09-07 10:19:21 | Stdout: ''
2020-09-07 10:19:21 | Stderr: 'error checking if cached image exists from a previous build: error getting history of "c5d52ad8e571fc7e9105cd5852abd9abea732b8c870d30893da4e87ce6d9c3b8": error creating new image from reference to image "c5d52ad8e571fc7e9105cd5852abd9abea732b8c870d30893da4e87ce6d9c3b8": error locating item named "manifest" for image with ID "c5d52ad8e571fc7e9105cd5852abd9abea732b8c870d30893da4e87ce6d9c3b8": file does not exist\n'
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build Traceback (most recent call last):
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build super(Command, self).run(parsed_args)
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build return super(Command, self).run(parsed_args)
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/cliff/command.py", line 187, in run
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build return_code = self.take_action(parsed_args) or 0
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleoclient/v2/tripleo_container_image.py", line 644, in take_action
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build bb.build_all()
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build File "/home/zuul/workspace/venv_build/lib/python3.6/site-packages/tripleo_common/image/builder/buildah.py", line 232, in build_all
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build exceptions='\n'.join(exceptions)
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build RuntimeError:
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build The following errors were detected during container build(s):
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build Exception information:
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build The following errors were detected during container build(s):
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build Exception information:
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build The following errors were detected during container build(s):
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build Exception information: Unexpected error while running command.
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build Command: sudo buildah bud --volume /etc/yum.repos.d:/etc/yum.repos.d:z --volume /etc/pki/rpm-gpg:/etc/pki/rpm-gpg:z --format docker --tls-verify=False --layers --logfile /tmp/container-builds/55a84b8a-cd40-4e6c-a7cb-39b6daad7b32/base/os/placement-api/placement-api-build.log -t 127.0.0.1:8787/tripleomaster/openstack-placement-api:04d7e9b3125e0f17e08e43271a62bb96 /tmp/container-builds/55a84b8a-cd40-4e6c-a7cb-39b6daad7b32/base/os/placement-api
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build Exit code: 1
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build Stdout: ''
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build Stderr: 'error checking if cached image exists from a previous build: error getting history of "c5d52ad8e571fc7e9105cd5852abd9abea732b8c870d30893da4e87ce6d9c3b8": error creating new image from reference to image "c5d52ad8e571fc7e9105cd5852abd9abea732b8c870d30893da4e87ce6d9c3b8": error locating item named "manifest" for image with ID "c5d52ad8e571fc7e9105cd5852abd9abea732b8c870d30893da4e87ce6d9c3b8": file does not exist\n'
2020-09-07 10:19:21 | 2020-09-07 10:19:21.629 38653 ERROR tripleoclient.v2.tripleo_container_image.Build 

Some example logs:-
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_009/749728/1/gate/tripleo-build-containers-ubi-8/009cb5b/logs/build.log
https://f2a0e1d5e065af30b773-76da2058f7382f685ecdff725a6049b3.ssl.cf5.rackcdn.com/749728/1/gate/tripleo-ci-centos-8-content-provider/6617656/logs/undercloud/home/zuul/container_image_build.log

Emilien Macchi (emilienm)
Changed in tripleo:
milestone: none → victoria-3
status: New → Triaged
importance: Undecided → High
OpenStack Infra (hudson-openstack)
Changed in tripleo:
assignee: nobody → Kevin Carter (kevin-carter)
status: Triaged → In Progress
OpenStack Infra (hudson-openstack)
Changed in tripleo:
assignee: Kevin Carter (kevin-carter) → Emilien Macchi (emilienm)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/750366
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=16ff908a2d8638a200bb0d40e043de411d52d2f0
Submitter: Zuul
Branch: master

commit 16ff908a2d8638a200bb0d40e043de411d52d2f0
Author: Kevin Carter <email address hidden>
Date: Tue Sep 8 08:57:05 2020 -0500

    Update container build to better reign in threads

    This restructures the buildah build process to ensure we're
    managing our thread pool correctly and not spawing more
    threads than needed or expected. By updating the thread
    process we'll ensure we're building each container only
    once and that we're maintaining build order.

    Tests have been added to ensure the new methods are called,
    and that the data processing is consistent.

    Closes-Bug: #1894856
    Change-Id: I647212442c378aabbd1887a0b499c19bbc74b5f7
    Signed-off-by: Kevin Carter <email address hidden>

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/752323

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/752324

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/ussuri)

Reviewed: https://review.opendev.org/752323
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=10e5e6cc2a31b288564dd227b4681c4fc2fb8530
Submitter: Zuul
Branch: stable/ussuri

commit 10e5e6cc2a31b288564dd227b4681c4fc2fb8530
Author: Kevin Carter <email address hidden>
Date: Tue Sep 8 08:57:05 2020 -0500

    Update container build to better reign in threads

    This restructures the buildah build process to ensure we're
    managing our thread pool correctly and not spawing more
    threads than needed or expected. By updating the thread
    process we'll ensure we're building each container only
    once and that we're maintaining build order.

    Tests have been added to ensure the new methods are called,
    and that the data processing is consistent.

    Closes-Bug: #1894856
    Change-Id: I647212442c378aabbd1887a0b499c19bbc74b5f7
    Signed-off-by: Kevin Carter <email address hidden>
    (cherry picked from commit 16ff908a2d8638a200bb0d40e043de411d52d2f0)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-common (stable/train)

Change abandoned by "Alex Schultz <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-common/+/752324
Reason: we'll just skip this since it's not py2 compatible

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.