Investigation result:-

* Minion pulls container from undercloud.
* Undercloud was previously uploading containers in its registry with tags that also have modify_append_tag suffix like "-updated-20200709032646" 
* In the failed jobs undercloud is uploading container without modify_append_tag suffix.


Minion is trying to pull container with tag (which is suffixed with modify_append_tag) but because that tag container don't exists - minion failed to pull containers from undercloud.


expected:-
=========

e961461fc3de7c29860ddacfcf697aaf-updated-20200709032646"

~~~
2020-07-09 04:17:19.547221 | bc764e20-0952-652d-4f3e-0000000006d8 |      FATAL | Pre-fetch all the containers | minion | item=192.168.24.1:8787/tripleomaster/centos-binary-heat-engine:e961461fc3de7c29860ddacfcf697aaf-updated-20200709032646 
~~~

Actual which undercloud is pushing now:-
======================================

e961461fc3de7c29860ddacfcf697aaf


Good job:-

https://73c3fc2c634b9a95041f-b66311f00e65e72370f624798f3cdac4.ssl.cf1.rackcdn.com/739863/1/gate/tripleo-ci-centos-8-containers-undercloud-minion/6a3a719/logs/undercloud/var/log/extra/podman/podman_allinfo.log

~~~
+ podman images
REPOSITORY                                                                TAG                                                       IMAGE ID       CREATED          SIZE
192.168.24.1:8787/tripleomaster/centos-binary-iscsid                      0a66a7cdb816ac8940c9ab32130babe7-updated-20200708125801   dedf259390fd   34 minutes ago   449 MB
192.168.24.1:8787/tripleomaster/centos-binary-neutron-server              0a66a7cdb816ac8940c9ab32130babe7-updated-20200708125801   7db3da394906   34 minutes ago   1.02 GB
192.168.24.1:8787/tripleomaster/centos-binary-swift-account               0a66a7cdb816ac8940c9ab32130babe7-updated-20200708125801   8910460e8390   34 minutes ago   760 MB
192.168.24.1:8787/tripleomaster/centos-binary-swift-container             0a66a7cdb816ac8940c9ab32130babe7-updated-20200708125801   03896ed30a5d   34 minutes ago   760 MB
192.168.24.1:8787/tripleomaster/centos-binary-keystone                    0a66a7cdb816ac8940c9ab32130babe7-updated-20200708125801   0768b3f4c2f5   34 minutes ago   805 MB
192.168.24.1:8787/tripleomaster/centos-binary-cron                        0a66a7cdb816ac8940c9ab32130babe7-updated-20200708125801   92cec8dec81e   35 minutes ago   441 MB
~~~

Current affected job:-

https://16a586043fd88ddd8ac7-0aa6565d1e0c06a8a7c8c1f2b9ada9fc.ssl.cf5.rackcdn.com/724734/22/check/tripleo-ci-centos-8-containers-undercloud-minion/3fe43e7/logs/undercloud/var/log/extra/podman/podman_allinfo.log
~~~
+ podman images
REPOSITORY                                                                TAG                                IMAGE ID       CREATED        SIZE
192.168.24.1:8787/tripleomaster/centos-binary-heat-api                    e961461fc3de7c29860ddacfcf697aaf   8749c2baf2b6   17 hours ago   735 MB
192.168.24.1:8787/tripleomaster/centos-binary-heat-engine                 e961461fc3de7c29860ddacfcf697aaf   5e8f7551788f   17 hours ago   735 MB
192.168.24.1:8787/tripleomaster/centos-binary-neutron-l3-agent            e961461fc3de7c29860ddacfcf697aaf   49394c6d04d6   17 hours ago   915 MB
192.168.24.1:8787/tripleomaster/centos-binary-neutron-dhcp-agent          e961461fc3de7c29860ddacfcf697aaf   bc0f1f980f77   17 hours ago   915 MB
~~~