[master] "error": "'config'", "msg": "Error running container image prepare: 'config'" failing on standalone jobs sc001 and sc004

[Bhagyashri Shewale]

After fixing this issue https://bugs.launchpad.net/tripleo/+bug/1946045/ Fix: https://review.opendev.org/c/openstack/puppet-tripleo/+/812457 we are getting this issue on standalone jobs

Error logs:

2021-10-05 10:53:24.744888 | fa163e6c-3a73-4bd6-e1dc-0000000014ee | TIMING | Build disk list for cephadm | undercloud | 0:03:45.832299 | 0.03s
2021-10-05 10:53:24.761010 | fa163e6c-3a73-4bd6-e1dc-0000000014f5 | TASK | Container image prepare
2021-10-05 11:04:58.145153 | fa163e6c-3a73-4bd6-e1dc-0000000014f5 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "'config'", "msg": "Error running container image prepare: 'config'", "params": {}, "success": false}
2021-10-05 11:04:58.150607 | fa163e6c-3a73-4bd6-e1dc-0000000014f5 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:15:19.237993 | 693.39s

PLAY RECAP *********************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
standalone : ok=246 changed=138 unreachable=0 failed=0 skipped=82 rescued=0 ignored=0
undercloud : ok=24 changed=7 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0
2021-10-05 11:04:58.186592 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-10-05 11:04:58.187146 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 366 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2021-10-05 11:04:58.187699 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:15:19.275113 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[1]: https://logserver.rdoproject.org/73/34373/7/check/periodic-tripleo-ci-centos-8-scenario001-standalone-master/498e7d7/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
[2]: https://logserver.rdoproject.org/73/34373/7/check/periodic-tripleo-ci-centos-8-scenario004-standalone-master/20c89bd/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[Rabi Mishra (rabi)]

I see the following errors in the container image prepare logs. Some issue with quay.ceph.io maybe?

2021-10-05 11:02:04,986 59824 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/prometheus/node-exporter:v0.17.0] Failed uploading the target image

2021-10-05 11:02:12,747 59825 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/prometheus/prometheus:v2.7.2] Failed uploading the target image

[David Vallee Delisle (valleedelisle)]

culprit: https://review.opendev.org/c/openstack/tripleo-common/+/810312

prometheus has bad metadata [1]

[1]
~~~
{
  "schemaVersion": 1,
  "name": "prometheus/prometheus",
  "tag": "v2.7.2",
  "architecture": "amd64",
  "fsLayers": [
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:e51130820cb7f52e561a2f05abd624b0793bcc36accf3b2e5f5d2abdcb4fe6f2"
    },
    {
      "blobSum": "sha256:89c95bc388e90cb3fceb3a44f5f852533602db9ff54e9795e50326cfd96344a4"
    },
    {
      "blobSum": "sha256:c0fd0ac7829e8ad3f4d3e1a6bab933b2253440566b121e2cd2d5a790c9d3d09b"
    },
    {
      "blobSum": "sha256:5f8f01ea2931bc86dba46415aca1f80f73c1ee67f18d34f4376ca1d365811bbd"
    },
    {
      "blobSum": "sha256:75c520aff8e8d376ed1b5d18fd8b8df5bd2e4a02a0154ceab9af9d5dfdea5f01"
    },
    {
      "blobSum": "sha256:494d1a723657236b2a19ba8773d2058afc39579f2d5f760d9841e22746d84461"
    },
    {
      "blobSum": "sha256:6d73410cc7fa96bdea537862c4d160d29d252949b71ab67f000d0e7a7c27022c"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:65b1a901dcccc7c92e3b43591444a08b0b6ec104758a301c04a02dfb05500c67"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:57c14dd66db0390dbf6da578421c077f6de8e88edd0815b4caa94607ba5f4c09"
    }
  ],
  "history": [
    {
      "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"nobody\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"9090/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"--config.file=/etc/prometheus/prometheus.yml\",\"--storage.tsdb.path=/prometheus\",\"--web.console.libraries=/usr/share/prometheus/console_libraries\",\"--web.console.templates=/usr/share/prometheus/consoles\"],\"ArgsEscaped\":true,\"Image\":\"sha256:6e68bb226f2692d11d46dcf82cc48adb169102b9d6f9ddae150142decc022fc6\",\"Volumes\":{\"/prometheus\":{}},\"WorkingDir\":\"/prometheus\",\"Entrypoint\":[\"/bin/prometheus\"],\"OnBuild\":[],\"Labels\":{\"maintainer\":\"The Prometheus Authors \\<email address hidden>\\u003e\"}},\"container\":\"7ba176a39a6f88d95ae1a986c7ebdc2a9a6762ad815bb60930474de6243e9836\",\"container_config\":{\"Hostname\":\"7ba176a39a6f\",\"Domainname\":\"\...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-common/+/812441

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/812506

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/812441
Committed: https://opendev.org/openstack/tripleo-common/commit/b2f76f8514baefb57e10c9a5562457b651ec29af
Submitter: "Zuul (22348)"
Branch: master

commit b2f76f8514baefb57e10c9a5562457b651ec29af
Author: Alex Schultz <email address hidden>
Date: Tue Oct 5 13:47:52 2021 +0000

    Revert "Allow for OCI manifests"

    This reverts commit e252328bf5a1134bfb236bfc84ce9c7b4597b5a7.

    Reason for revert: We're hitting some invalid metadata from some ceph
    containers.

    Change-Id: I6c5f449f96f212d9d030b0e4a83d4628faf089d5
    Closes-Bug: #1946091

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/812506
Committed: https://opendev.org/openstack/tripleo-ci/commit/41b936fd719002258ed55e07c29e0ac731056fc5
Submitter: "Zuul (22348)"
Branch: master

commit 41b936fd719002258ed55e07c29e0ac731056fc5
Author: Alex Schultz <email address hidden>
Date: Tue Oct 5 07:54:37 2021 -0600

    Add scenario001/004 coverage to image uploader changes

    Because we fetch some external containers, we need to check them when we
    make changes to the container image uploader bits.

    Depends-On: https://review.opendev.org/c/openstack/tripleo-common/+/812441
    Change-Id: I85166ca8932398dd230157d2faa6cc3c4d948df4
    Related-Bug: #1946091

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 16.2.0 release.