"podman pull failed" errors from Puppet are reported as warnings instead of errors

[Tal Liron]

If the container image tag requested is not available at Docker Hub then the pull will fail. But unfortunately the failure will not lead to the overcloud deployment failing immediately. Instead it seems the Ansible playbook continues marching onward and confusing failures happen later as certain services are simply not available.

Here is a section of the log:

2019-06-14 18:37:51,414 p=348 u=mistral | TASK [Debug output for task: Run container-puppet tasks (generate config) during step 1] ***
2019-06-14 18:37:51,414 p=348 u=mistral | task path: /var/lib/mistral/lab/common_deploy_steps_tasks.yaml:451
2019-06-14 18:37:51,414 p=348 u=mistral | Friday 14 June 2019 18:37:51 -0500 (0:02:17.459) 0:08:22.205 ***********
2019-06-14 18:37:51,498 p=348 u=mistral | ok: [lab-computehci-0] => {
    "failed_when_result": false,
    "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
        "2019-06-14 23:35:34,333 INFO: 49820 -- Running container-puppet",

...

        "2019-06-14 23:35:35,829 INFO: 15747 -- Pulling image: docker.io/tripleostein/centos-binary-nova-api:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4",
        "2019-06-14 23:35:36,098 INFO: 15746 -- Pulling image: docker.io/tripleostein/centos-binary-redis:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4",
        "2019-06-14 23:35:36,165 INFO: 15743 -- Pulling image: docker.io/tripleostein/centos-binary-nova-placement-api:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4",
        "2019-06-14 23:35:36,268 INFO: 15748 -- Pulling image: docker.io/tripleostein/centos-binary-glance-api:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4",
        "2019-06-14 23:35:36,543 INFO: 15744 -- Pulling image: docker.io/tripleostein/centos-binary-heat-api:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4",
        "2019-06-14 23:36:08,648 WARNING: 15746 -- podman pull failed: Trying to pull docker://docker.io/tripleostein/centos-binary-redis:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4...Getting image source signatures",
        "Copying blob sha256:b49a3f363d5ea65a3408faed174571cce9d7352726324a482e7409b1660885d9",
        "Copying blob sha256:4d9c950caa9e0c4943ac19f8ad5ff7379d704b333aee2520ff32548e33f287c8",
        "Copying blob sha256:85f5e64afb45672dcca6542d721ae1c9549fb3367c634cd601d6f332b51fbe95",
        "Copying blob sha256:f9d2c058b06e767cf087fb7c76d19bc81fde2405ab84f51ee753f1f62110be11",
        "Copying blob sha256:c0452c1cc48e55aad7a2a0825e0cdd8254e53fc94533cb7e96f31fc316a9fb6b",
        "Copying blob sha256:8ba884070f611d31cb2c42eddb691319dc9facf5e0ec67672fcfa135181ab3df",
        "Copying blob sha256:18369c1c18fadd1774ba2f970fa096989810e387bed1b92f935d92025e3bc151",
        "Copying blob sha256:e58ce482730708e4c03672c4e8adc3ce58709e6f2554e23ff78282229c2e92c3",
        "Copying blob sha256:bc56159ccf2d443f95a7a009fd07fd82dbaafdecd83392fa91fd8ee1a89aceef",
        "Copying blob sha256:5fdc0f51bbbf255107d6b47fac9a1a68a9cd2461878f6a635b7e0add56bc8292",
        "Copying blob sha256:b1ce70caa9fa6e07ac534d97532a721e445e34cea76fc7db12fb96aed2d86e86",
        "Copying blob sha256:e1634191d25948e42cc859d4faa1461389ae48681f8de9eb6b79e25f10a34bbb",
        "Copying blob sha256:130afd2eb0d59dfe0916fb1770d3df19ddd88b3b1244f931c6f84f22b0b85781",
        "Copying blob sha256:231fa32c040c6759391097e6440581b590694d47fddc77bb33edeb3dce4e0bbd",
        "Copying blob sha256:ba8fa835f4aefb1f9cff272bd4027ce0805a2d23030ef117358e515efbd21cb5",
        "Copying blob sha256:9cc46ae7908d4164c9b670a549147c0556ae36fea3fafa07b67db30b81a3a593",
        "Copying blob sha256:5ee844e60cc3fc6f81293c3e4b33804de35f0b67a0d5a41765b879b7d0ee8148",
        "Copying blob sha256:4e271dd684c2207424d5182a930350af3251f66d9226d487e547c635fd931eac",
        "Copying blob sha256:c91a729d164ec409a4922735c2bfaf3bae28317954a98ad82d5e448dc897d472",
        "Copying blob sha256:2447d1a1f96e7176b4b3495c4ccdd4a4cbf82aa87e0d950260145c1871c7963f",
        "Copying blob sha256:17d8e312ca632a3e5518e1c037d199f9e0431fc26ff3b24c3c0dc63522e09152",
        "Copying blob sha256:d6a416374c4d8298bb19d933066a4d180c3b456a850e6d3fb638004552fde68b",
        "time=\"2019-06-14T23:36:03Z\" level=error msg=\"Error pulling image ref //tripleostein/centos-binary-redis:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4: Error reading blob sha256:33aaa14b457bc6cecc7fb28d541b6647eb912b3dce4d0be47fe861d7d7686b0c: Get https://registry-1.docker.io/v2/tripleostein/centos-binary-redis/blobs/sha256:33aaa14b457bc6cecc7fb28d541b6647eb912b3dce4d0be47fe861d7d7686b0c: dial tcp: lookup registry-1.docker.io on 1.0.0.1:53: read udp 10.0.2.2:44554->1.0.0.1:53: i/o timeout\" ",
        "Failed",
        "(0x559f60aa4420,0xc420908440)",
        "Error: error pulling image \"docker.io/tripleostein/centos-binary-redis:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4\": Invalid image name \"docker.io/tripleostein/centos-binary-redis:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4\", unknown transport \"docker.io/tripleostein/centos-binary-redis\"",
        "",
        "2019-06-14 23:36:08,648 WARNING: 15746 -- retrying pulling image: docker.io/tripleostein/centos-binary-redis:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4",
        "2019-06-14 23:36:15,843 WARNING: 15747 -- podman pull failed: Trying to pull docker://docker.io/tripleostein/centos-binary-nova-api:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4...Getting image source signatures",
        "Copying blob sha256:d0dfdb02ffe5aa99de9d907b701c11dcc7030d75e726eaf37e9c5046aa48f6ca",
        "Copying blob sha256:ed7451d9d8302d34baef8bae71b7d80f4129512f70ad006715334dcf626330dc",
        "Copying blob sha256:7bd930c51a4bc8d74d38a52ec70d7d1b48f254d2ad6f617e5662ed248bb01fbd",
        "Copying blob sha256:33aaa14b457bc6cecc7fb28d541b6647eb912b3dce4d0be47fe861d7d7686b0c",
        "Copying blob sha256:3456378890daf507162e383acee27b5aef73fe0f8a0e1ec2fa88460929101f58",
        "Copying blob sha256:aa74d13ad05ba7d535401a63d9e10f693724d5d53a0dc1192a2d5b51baf5e4e6",
        "Copying blob sha256:5d685ea2c425ff3431db6c07391af944b0d697fbd5cca015b7ac89a2c23d9237",
        "Copying blob sha256:c82b41b8e2e7154157f61cc4e187e45bcf773780568b12840b20e73313cd5f68",
        "Copying blob sha256:3b02eccbccefba5ece7d03d06b8be3c66f4ba983cc24d3f232408168cb5467ca",
        "Copying blob sha256:90d9b7d00c4a1b4a2763fcb34a5565b1f1c28ad91053fd85c27e4a6c76f107d9",
        "time=\"2019-06-14T23:36:12Z\" level=error msg=\"Error pulling image ref //tripleostein/centos-binary-nova-api:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4: Error reading blob sha256:2bbe780ec80db42ec14845088869c92f5304479c29cd36f70650d82ad6d46d77: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/2b/2bbe780ec80db42ec14845088869c92f5304479c29cd36f70650d82ad6d46d77/data?verify=1560558350-sfoVtZN33VHc22ZeVEMKuXcYFX0%3D: dial tcp: lookup production.cloudflare.docker.com on 1.0.0.1:53: read udp 10.0.2.2:52204->1.0.0.1:53: i/o timeout\" ",
        "(0x555f28b44420,0xc42096a9e0)",
        "Error: error pulling image \"docker.io/tripleostein/centos-binary-nova-api:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4\": Invalid image name \"docker.io/tripleostein/centos-binary-nova-api:d45bbddf5063604b241bc8f14e39d90cae03de8b_24ce03e4\", unknown transport \"docker.io/tripleostein/centos-binary-nova-api\"",

...