tripleo-ci-centos-8-content-provider failed to pull contaimers from quay.ceph.io having below traceback :
2021-05-10 09:00:10.329814 | primary | TASK [container-build : List down missing containers] **************************
2021-05-10 09:00:10.329982 | primary | Monday 10 May 2021 09:00:10 +0000 (0:00:13.204) 0:55:32.166 ************
2021-05-10 09:00:10.438524 | primary | ok: [undercloud] => (item={'msg': 'non-zero return code', 'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64'], 'stdout': '', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64...\n manifest unknown: manifest unknown\nError: 1 error occurred:\n\t* Error initializing source docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64: Error reading manifest v6.0.0-stable-6.0-pacific-centos-8-x86_64 in trunk.registry.rdoproject.org/ceph/daemon: manifest unknown: manifest unknown', 'rc': 125, 'start': '2021-05-10 08:59:57.597731', 'end': '2021-05-10 08:59:58.507648', 'delta': '0:00:00.909917', 'changed': True, 'failed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64...', ' manifest unknown: manifest unknown', 'Error: 1 error occurred:', '\t* Error initializing source docker://trunk.registry.rdoproject.org/ceph/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64: Error reading manifest v6.0.0-stable-6.0-pacific-centos-8-x86_64 in trunk.registry.rdoproject.org/ceph/daemon: manifest unknown: manifest unknown'], 'item': 'quay.ceph.io/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64', 'ansible_loop_var': 'item'})
2021-05-10 09:00:10.487931 | primary | skipping: [undercloud] => (item={'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/prometheus:v2.7.2'], 'stdout': '2a68492b2466243c7c73692f575e04c091feebfeae94bb164c4cc811bed773b0', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/prometheus:v2.7.2...\nGetting image source signatures\nCopying blob sha256:75c520aff8e8d376ed1b5d18fd8b8df5bd2e4a02a0154ceab9af9d5dfdea5f01\nCopying blob sha256:6d73410cc7fa96bdea537862c4d160d29d252949b71ab67f000d0e7a7c27022c\nCopying blob sha256:494d1a723657236b2a19ba8773d2058afc39579f2d5f760d9841e22746d84461\nCopying blob sha256:57c14dd66db0390dbf6da578421c077f6de8e88edd0815b4caa94607ba5f4c09\nCopying blob sha256:5f8f01ea2931bc86dba46415aca1f80f73c1ee67f18d34f4376ca1d365811bbd\nCopying blob sha256:65b1a901dcccc7c92e3b43591444a08b0b6ec104758a301c04a02dfb05500c67\nCopying blob sha256:c0fd0ac7829e8ad3f4d3e1a6bab933b2253440566b121e2cd2d5a790c9d3d09b\nCopying blob sha256:89c95bc388e90cb3fceb3a44f5f852533602db9ff54e9795e50326cfd96344a4\nCopying blob sha256:e51130820cb7f52e561a2f05abd624b0793bcc36accf3b2e5f5d2abdcb4fe6f2\nCopying config sha256:2a68492b2466243c7c73692f575e04c091feebfeae94bb164c4cc811bed773b0\nWriting manifest to image destination\nStoring signatures', 'rc': 0, 'start': '2021-05-10 08:59:58.969999', 'end': '2021-05-10 09:00:03.444637', 'delta': '0:00:04.474638', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/prometheus:v2.7.2', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['2a68492b2466243c7c73692f575e04c091feebfeae94bb164c4cc811bed773b0'], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/prometheus:v2.7.2...', 'Getting image source signatures', 'Copying blob sha256:75c520aff8e8d376ed1b5d18fd8b8df5bd2e4a02a0154ceab9af9d5dfdea5f01', 'Copying blob sha256:6d73410cc7fa96bdea537862c4d160d29d252949b71ab67f000d0e7a7c27022c', 'Copying blob sha256:494d1a723657236b2a19ba8773d2058afc39579f2d5f760d9841e22746d84461', 'Copying blob sha256:57c14dd66db0390dbf6da578421c077f6de8e88edd0815b4caa94607ba5f4c09', 'Copying blob sha256:5f8f01ea2931bc86dba46415aca1f80f73c1ee67f18d34f4376ca1d365811bbd', 'Copying blob sha256:65b1a901dcccc7c92e3b43591444a08b0b6ec104758a301c04a02dfb05500c67', 'Copying blob sha256:c0fd0ac7829e8ad3f4d3e1a6bab933b2253440566b121e2cd2d5a790c9d3d09b', 'Copying blob sha256:89c95bc388e90cb3fceb3a44f5f852533602db9ff54e9795e50326cfd96344a4', 'Copying blob sha256:e51130820cb7f52e561a2f05abd624b0793bcc36accf3b2e5f5d2abdcb4fe6f2', 'Copying config sha256:2a68492b2466243c7c73692f575e04c091feebfeae94bb164c4cc811bed773b0', 'Writing manifest to image destination', 'Storing signatures'], 'failed': False, 'item': 'quay.ceph.io/prometheus/prometheus:v2.7.2', 'ansible_loop_var': 'item'})
2021-05-10 09:00:10.538197 | primary | skipping: [undercloud] => (item={'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/alertmanager:v0.16.2'], 'stdout': 'a1c1bd95a2dd2b45df13eb39a56e25c2f1b791d7ffde6fb1fe9cddbedc99bd01', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/alertmanager:v0.16.2...\nGetting image source signatures\nCopying blob sha256:4ee3d6d19c59903413e1e6b55b29e2dd2268831c41ac5489511b39c1a4347065\nCopying blob sha256:ed2709e725224b326016c74f92927541229642f5bdbb7a38980aa88d291fc075\nCopying blob sha256:697743189b6d255069caf6c455be10c7f8cae8076c6f94d224ae15cd41420e87\nCopying blob sha256:a94ecdcbbebe73bf02cee692ccc517fb51e9d08d6fc4dc09017fbf139b5cfba3\nCopying blob sha256:85bad536b567cf559bf80da1bab180dabfc8d96cdd3d69efaf6a61a786d7e229\nCopying blob sha256:9ed97210a3829d5624e829e1ede7c794568b670dd88942b873337fe5fd806b76\nCopying config sha256:a1c1bd95a2dd2b45df13eb39a56e25c2f1b791d7ffde6fb1fe9cddbedc99bd01\nWriting manifest to image destination\nStoring signatures', 'rc': 0, 'start': '2021-05-10 09:00:03.903793', 'end': '2021-05-10 09:00:06.849636', 'delta': '0:00:02.945843', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/alertmanager:v0.16.2', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['a1c1bd95a2dd2b45df13eb39a56e25c2f1b791d7ffde6fb1fe9cddbedc99bd01'], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/alertmanager:v0.16.2...', 'Getting image source signatures', 'Copying blob sha256:4ee3d6d19c59903413e1e6b55b29e2dd2268831c41ac5489511b39c1a4347065', 'Copying blob sha256:ed2709e725224b326016c74f92927541229642f5bdbb7a38980aa88d291fc075', 'Copying blob sha256:697743189b6d255069caf6c455be10c7f8cae8076c6f94d224ae15cd41420e87', 'Copying blob sha256:a94ecdcbbebe73bf02cee692ccc517fb51e9d08d6fc4dc09017fbf139b5cfba3', 'Copying blob sha256:85bad536b567cf559bf80da1bab180dabfc8d96cdd3d69efaf6a61a786d7e229', 'Copying blob sha256:9ed97210a3829d5624e829e1ede7c794568b670dd88942b873337fe5fd806b76', 'Copying config sha256:a1c1bd95a2dd2b45df13eb39a56e25c2f1b791d7ffde6fb1fe9cddbedc99bd01', 'Writing manifest to image destination', 'Storing signatures'], 'failed': False, 'item': 'quay.ceph.io/prometheus/alertmanager:v0.16.2', 'ansible_loop_var': 'item'})
2021-05-10 09:00:10.588579 | primary | skipping: [undercloud] => (item={'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/node-exporter:v0.17.0'], 'stdout': 'b3e7f67a1480979f7c2068c14e4f0d3da6243631b9def8490ed68236a8311f11', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/node-exporter:v0.17.0...\nGetting image source signatures\nCopying blob sha256:0de338cf42586fa81850a9f1610429364d1a7f348e8db87786be48663b25a60d\nCopying blob sha256:d764f7880123ec20906e97bf9830c1fdc0f741549d88b3145ae8ac8da0017b2e\nCopying blob sha256:f508012419d8ed023913de91bf25eef8192444e7624e71cb87c5ce6d7323d497\nCopying config sha256:b3e7f67a1480979f7c2068c14e4f0d3da6243631b9def8490ed68236a8311f11\nWriting manifest to image destination\nStoring signatures', 'rc': 0, 'start': '2021-05-10 09:00:07.296284', 'end': '2021-05-10 09:00:09.330567', 'delta': '0:00:02.034283', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/node-exporter:v0.17.0', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': ['b3e7f67a1480979f7c2068c14e4f0d3da6243631b9def8490ed68236a8311f11'], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/node-exporter:v0.17.0...', 'Getting image source signatures', 'Copying blob sha256:0de338cf42586fa81850a9f1610429364d1a7f348e8db87786be48663b25a60d', 'Copying blob sha256:d764f7880123ec20906e97bf9830c1fdc0f741549d88b3145ae8ac8da0017b2e', 'Copying blob sha256:f508012419d8ed023913de91bf25eef8192444e7624e71cb87c5ce6d7323d497', 'Copying config sha256:b3e7f67a1480979f7c2068c14e4f0d3da6243631b9def8490ed68236a8311f11', 'Writing manifest to image destination', 'Storing signatures'], 'failed': False, 'item': 'quay.ceph.io/prometheus/node-exporter:v0.17.0', 'ansible_loop_var': 'item'})
2021-05-10 09:00:10.650911 | primary | ok: [undercloud] => (item={'msg': 'non-zero return code', 'cmd': ['podman', 'pull', 'docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4'], 'stdout': '', 'stderr': 'Trying to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4...\n manifest unknown: manifest unknown\nError: 1 error occurred:\n\t* Error initializing source docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: Error reading manifest 6.7.4 in trunk.registry.rdoproject.org/ceph/grafana: manifest unknown: manifest unknown', 'rc': 125, 'start': '2021-05-10 09:00:09.787072', 'end': '2021-05-10 09:00:10.282573', 'delta': '0:00:00.495501', 'changed': True, 'failed': True, 'invocation': {'module_args': {'_raw_params': 'podman pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4', 'warn': False, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['Trying to pull docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4...', ' manifest unknown: manifest unknown', 'Error: 1 error occurred:', '\t* Error initializing source docker://trunk.registry.rdoproject.org/ceph/grafana:6.7.4: Error reading manifest 6.7.4 in trunk.registry.rdoproject.org/ceph/grafana: manifest unknown: manifest unknown'], 'item': 'quay.ceph.io/app-sre/grafana:6.7.4', 'ansible_loop_var': 'item'})
References :
https://33807c6af0e20453b14c-c0af318d07ef9bf7ba9a60c77add6086.ssl.cf5.rackcdn.com/790377/1/check/tripleo-ci-centos-8-content-provider/2e666e0/job-output.txt
ceph related containers have been pushed into rdo.. however we still have a problem.
2021-05-10 14:45:06.018167 | primary | 127.0.0. 1:5001/ tripleomaster/ openstack- base 22fd7520a99b928 db16ffaa8eaee9b 63 f3a164bc5eab 40 minutes ago 561 MB access. redhat. com/ubi8 latest 613e5da7a934 3 weeks ago 213 MB rdoproject. org/ceph/ daemon v6.0.0- stable- 6.0-pacific- centos- 8-x86_64 14fee0875498 5 weeks ago 1.17 GB tripleoci/ registry 2 2d4f4b5309b1 10 months ago 26.8 MB rdoproject. org/ceph/ grafana 6.7.4 be4c69a1aae8 11 months ago 237 MB rdoproject. org/ceph/ alertmanager v0.16.2 a1c1bd95a2dd 2 years ago 43.9 MB rdoproject. org/ceph/ prometheus v2.7.2 2a68492b2466 2 years ago 103 MB rdoproject. org/ceph/ node-exporter v0.17.0 b3e7f67a1480 2 years ago 22.4 MB OVERLAY_ FS_REDIRECT_ DIR enabled" extfs, projectQuotaSup ported= false, useNativeDiff= false, usingMetacopy=true" containers/ registries. conf\"" containers/ registries. conf.d/ 000-shortnames. conf\"" containers/ registries. conf.d/ rhel-shortnames .conf\" " containers/ registries. conf\", assuming it's not blocked"
2021-05-10 14:45:06.018193 | primary | registry.
2021-05-10 14:45:06.018221 | primary | trunk.registry.
2021-05-10 14:45:06.018249 | primary | quay.io/
2021-05-10 14:45:06.018265 | primary | trunk.registry.
2021-05-10 14:45:06.018278 | primary | trunk.registry.
2021-05-10 14:45:06.018291 | primary | trunk.registry.
2021-05-10 14:45:06.018303 | primary | trunk.registry.
2021-05-10 14:45:06.018315 | primary |
2021-05-10 14:45:06.018328 | primary |
2021-05-10 14:45:06.018341 | primary | STDERR:
2021-05-10 14:45:06.018353 | primary |
2021-05-10 14:45:06.018365 | primary | level=debug msg="[graphdriver] trying provided driver \"overlay\""
2021-05-10 14:45:06.018378 | primary | level=debug msg="cached value indicated that overlay is supported"
2021-05-10 14:45:06.018390 | primary | level=debug msg="cached value indicated that metacopy is being used"
2021-05-10 14:45:06.018402 | primary | level=debug msg="cached value indicated that native-diff is not being used"
2021-05-10 14:45:06.018415 | primary | level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_
2021-05-10 14:45:06.018428 | primary | level=debug msg="backingFs=
2021-05-10 14:45:06.018440 | primary | level=debug msg="Loading registries configuration \"/etc/
2021-05-10 14:45:06.018453 | primary | level=debug msg="Loading registries configuration \"/etc/
2021-05-10 14:45:06.018466 | primary | level=debug msg="Loading registries configuration \"/etc/
2021-05-10 14:45:06.018479 | primary | level=debug msg="registry \"127.0.0.1:5001\" is not listed in registries configuration \"/etc/
2021-05-10 14:45:06.018491 | primary ...