master standalone jobs consistently failing with "Not found image"

[Pooja Jadhav]

master standalone jobs consistently failing with "Not found image" while container image prepare with below traceback :

2021-10-19 06:38:33.980283 | fa163e7a-ab91-53d0-5eb6-0000000000a5 | TIMING | External deployment step 1 | undercloud | 0:03:38.688475 | 0.03s
2021-10-19 06:38:34.050716 | fa163e7a-ab91-53d0-5eb6-0000000000a6 | TIMING | include_tasks | undercloud | 0:03:38.758877 | 0.05s
2021-10-19 06:38:34.070047 | b0adab24-d2bc-4156-b311-435d58154d53 | INCLUDED | /home/zuul/tripleo-deploy/standalone-ansible-6mgz5snm/external_deploy_steps_tasks_step1.yaml | undercloud
2021-10-19 06:38:34.085007 | fa163e7a-ab91-53d0-5eb6-000000000fb2 | TASK | Container image prepare
2021-10-19 06:39:03.593311 | fa163e7a-ab91-53d0-5eb6-000000000fb2 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: https://trunk.registry.rdoproject.org/v2/tripleomaster/openstack-placement-api/manifests/810ca4bd5363b9331560829aac3a9665", "msg": "Error running container image prepare: Not found image: https://trunk.registry.rdoproject.org/v2/tripleomaster/openstack-placement-api/manifests/810ca4bd5363b9331560829aac3a9665", "params": {}, "success": false}
2021-10-19 06:39:03.597398 | fa163e7a-ab91-53d0-5eb6-000000000fb2 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:04:08.305539 | 29.51s

Detailed logs :

https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-common-master/3f6de84/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-common-master/bed5375/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-common-master/a6edeef/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

https://logserver.rdoproject.org/07/814407/5/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/fb85cf7/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

[Jiri Podivin (jpodivin)]

Does it bear any relation to the [0] or is it a separate issue?

[0]https://bugs.launchpad.net/tripleo/+bug/1946659

[Pooja Jadhav (poojajadhav22)]

that's different issue specific to upgrade jobs and that happens when there is promotion in middle of running job. current one is a generic issue, as all images older than 3 days are getting pruned, as there is issue finding whitelisted tag.

[Pooja Jadhav (poojajadhav22)]

Here is the updates:
Master current-tripleo containers got prunned from RDO registry due to letsencrypt cert validation issue (https://review.rdoproject.org/r/c/rdo-infra/rdo-infra-playbooks/+/36326 will fix it).

rebuilding and pushing the
master containers via this patch https://review.rdoproject.org/r/c/rdo-jobs/+/36327

Once the job finishes, Please recheck the affected patch.