tripleo

centos-9 content provider fails pulling ubi9 "unrecognized signature format, starting with binary 0x3c"

Bug #1999131 reported by Marios Andreou on 2022-12-08

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Critical	Unassigned	tripleo antelope-1

Bug Description

At [1][2] the tripleo-ci-centos-9-content-provider fails while building base container, specifically fails to pull ubi9 with trace like:

  Stderr: 'Error: creating build container: copying system image from manifest list: Source image rejected: parsing signature https://access.redhat.com/webassets/docker/content/sigstore/ubi9@sha256=0fdb00a34006ec99b33d3092c2de015cd0badce50720daed0ea5d05ad7849b55/signature-7: unrecognized signature format, starting with binary 0x3c\n'
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder Traceback (most recent call last):
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib/python3.9/site-packages/tripleo_common/image/builder/buildah.py", line 141, in _generate_container
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder self.build(container_name,
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 333, in wrapped_f
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder return self(f, *args, **kw)
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder do = self.iter(retry_state=retry_state)
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 372, in iter
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder raise retry_exc.reraise()
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 189, in reraise
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder raise self.last_attempt.result()
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder return self.__get_result()
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder raise self._exception
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder result = fn(*args, **kwargs)
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib/python3.9/site-packages/tripleo_common/image/builder/buildah.py", line 191, in build
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder process.execute(
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib/python3.9/site-packages/tripleo_common/utils/process.py", line 53, in execute
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder result = processutils.execute(*cmd, **kwargs)
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder File "/usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py", line 438, in execute
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder raise ProcessExecutionError(exit_code=_returncode,
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder Command: sudo buildah bud --net=host --volume /etc/pki/rpm-gpg:/etc/pki/rpm-gpg:z --volume /etc/yum.repos.d:/etc/distro.repos.d:z --volume /etc/pki/rpm-gpg:/etc/pki/rpm-gpg:z --volume /etc/dnf/vars:/etc/dnf/vars:z --volume /opt/gating_repo:/opt/gating_repo:z --volume /etc/yum.repos.d:/etc/distro.repos.d:z --format docker --tls-verify=False --logfile /home/zuul/container-builds/de9e1bb3-0e2a-4d84-a32f-f794bb4f5032/base/base-build.log -t 127.0.0.1:5001/tripleomastercentos9/openstack-base:5e9ef1b0c3488e63e4c3e8561bd39ace /home/zuul/container-builds/de9e1bb3-0e2a-4d84-a32f-f794bb4f5032/base
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder Exit code: 125
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder Stdout: ''
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder Stderr: 'Error: creating build container: copying system image from manifest list: Source image rejected: parsing signature https://access.redhat.com/webassets/docker/content/sigstore/ubi9@sha256=0fdb00a34006ec99b33d3092c2de015cd0badce50720daed0ea5d05ad7849b55/signature-7: unrecognized signature format, starting with binary 0x3c\n'
  2022-12-08 08:02:28.876 68385 ERROR tripleo_common.image.builder.buildah.BuildahBuilder [00m

In job history [3] currently we see only check but this job runs in the gate so gate blocker \o/
This doesn't seem to affect c8 content provider only c9.

[1] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a44/866449/3/check/tripleo-ci-centos-9-content-provider/a44f94a/logs/undercloud/home/zuul/container_image_build.log
[2] https://fcd003cab3aab07c2e4f-3105dcd3f53910544ad4af58357c5c48.ssl.cf5.rackcdn.com/845180/4/check/tripleo-ci-centos-9-content-provider/b884aff/logs/undercloud/home/zuul/container_image_build.log
[3] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider

Tags:

Revision history for this message

Marios Andreou (marios-b) wrote on 2022-12-08 (last edit on 2022-12-08):

also hitting zed there [1]

2022-12-08 08:02:34.959 62361 ERROR tripleoclient.v2.tripleo_container_image.Build Stderr: 'Error: creating build container: copying system image from manifest list: Source image rejected: parsing signature https://access.redhat.com/webassets/docker/content/sigstore/ubi9@sha256=0fdb00a34006ec99b33d3092c2de015cd0badce50720daed0ea5d05ad7849b55/signature-7: unrecognized signature format, starting with binary 0x3c\n'