Molecule jobs failing for stable/ussuri with: Unable to gather info for 'molecule_local/ubi8': Error: error inspecting object: unable to find a name and tag match for molecule_local/ubi8 in repotags: no such image

[Jiri Podivin]

The tripleo-validations-centos-8-molecule-* jobs on the stable/ussuri branch of the tripleo-validations repo are failing with:

"
Unable to gather info for 'molecule_local/ubi8': Error: error inspecting object: unable to find a name and tag match for molecule_local/ubi8 in repotags: no such image
"

The issue appears to affect at least the last two patches on the branch:
- c1ec034b6ab39833e973947d17194a6ccfdc39a7 (HEAD as of writing)
- e8892681e2c71fff4951f81967fffc6eab9cafab

All tripleo-validations-centos-8-molecule-* on the branch appear to be affected.

Trace:
------
ansible_loop_var: item
      item:
        ansible_loop_var: item
        changed: true
        checksum: 50fbdeb471a84d57f9159d288cce3ab910d2c2fc
        dest: /home/zuul/.cache/molecule/tls_everywhere/default/Dockerfile_ubi8_ubi_init
        diff: []
        failed: false
        gid: 1000
        group: zuul
        i: 0
        invocation:
          module_args:
            _original_basename: Dockerfile
            attributes: null
            backup: false
            checksum: 50fbdeb471a84d57f9159d288cce3ab910d2c2fc
            content: null
            delimiter: null
            dest: /home/zuul/.cache/molecule/tls_everywhere/default/Dockerfile_ubi8_ubi_init
            directory_mode: null
            follow: false
            force: true
            group: null
            local_follow: null
            mode: '0600'
            owner: null
            regexp: null
            remote_src: null
            selevel: null
            serole: null
            setype: null
            seuser: null
            src: /home/zuul/.ansible/tmp/ansible-tmp-1627459400.3900034-8200491419186/source
            unsafe_writes: null
            validate: null
        item:
          dockerfile: ../../../../.config/molecule/Dockerfile
          environment:
            http_proxy: '{{ lookup(''env'', ''http_proxy'') }}'
            https_proxy: '{{ lookup(''env'', ''https_proxy'') }}'
          hostname: ubi8
          image: ubi8/ubi-init
          name: ubi8
          pkg_extras: python*-setuptools python*-pyyaml krb5-workstation
          privileged: true
          registry:
            url: registry.access.redhat.com
          ulimits:

          - host
          volumes:
          - /etc/ci/mirror_info.sh:/etc/ci/mirror_info.sh:ro
        md5sum: 696b29e1c218a4b9916084317c64fdaa
        mode: '0600'
        owner: zuul
        secontext: unconfined_u:object_r:cache_home_t:s0
        size: 1235
        src: /home/zuul/.ansible/tmp/ansible-tmp-1627459400.3900034-8200491419186/source
        state: file
        uid: 1000
      msg: |-
        Unable to gather info for 'molecule_local/ubi8': Error: error inspecting object: unable to find a name and tag match for molecule_local/ubi8 in repotags: no such image

Logs:
-----
https://zuul.opendev.org/t/openstack/build/7715bd86ce074d62a4592c762569617a
https://zuul.opendev.org/t/openstack/build/9a68e24e4d23426db6a49e013500fc73
https://zuul.opendev.org/t/openstack/build/3093e3f00abf4e21bfd47e54cd52fa41

[wes hayutin (weshayutin)]

example:
https://zuul.openstack.org/builds?job_name=tripleo-validations-centos-8-molecule-ceph&branch=stable%2Fussuri

https://zuul.openstack.org/builds?job_name=tripleo-validations-centos-8-molecule-compute_tsx&branch=stable%2Fussuri

[Alex Schultz (alex-schultz)]

It looks like podman build is not being invoked. I compared my local execution (which works) vs a CI job

working:
ug 19 12:01:46 multi-2 ansible-ansible.legacy.copy[45600]: Invoked with src=/home/stack/.ansible/tmp/ansible-tmp-1629388905.9265203-45582-49374290168461/source dest=/home/stack/.cache/molecule/check_kernel_version/default/Dockerfile_ubi8_ubi_init mode=0600 follow=False _original_basename=Dockerfile che
cksum=a1079a43f2cef4461332eb246954538d9c8150f2 backup=False force=True unsafe_writes=False content=NOT_LOGGING_PARAMETER validate=None directory_mode=None remote_src=None local_follow=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None
Aug 19 12:01:48 multi-2 ansible-podman_image_info[45618]: Invoked with name=['molecule_local/ubi8'] executable=podman
Aug 19 12:01:48 multi-2 systemd[22317]: Started podman-45619.scope.
Aug 19 12:01:49 multi-2 ansible-ansible.legacy.command[45671]: Invoked with _raw_params=podman build -f /home/stack/.cache/molecule/check_kernel_version/default/Dockerfile_ubi8_ubi_init -t molecule_local/ubi8/ubi-init
                                                                _uses_shell=False warn=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Aug 19 12:01:49 multi-2 systemd[22317]: Started podman-45672.scope.
Aug 19 12:02:36 multi-2 sssd_kcm[42845]: Shutting down (status = 0)
Aug 19 12:02:36 multi-2 systemd[1]: sssd-kcm.service: Succeeded.
Aug 19 12:04:00 multi-2 ansible-async_wrapper.py[49641]: Invoked with 237660152745 7200 /home/stack/.ansible/tmp/ansible-tmp-1629389040.3452756-49629-182397066003313/AnsiballZ_command.py _
Aug 19 12:04:00 multi-2 ansible-async_wrapper.py[49644]: Starting module and watcher
Aug 19 12:04:00 multi-2 ansible-async_wrapper.py[49644]: Start watching 49645 (7200)
Aug 19 12:04:00 multi-2 ansible-async_wrapper.py[49645]: Start module (49645)
Aug 19 12:04:00 multi-2 ansible-async_wrapper.py[49641]: Return async_wrapper task started.
Aug 19 12:04:01 multi-2 ansible-ansible.legacy.command[49646]: Invoked with _raw_params=podman run -d --name "ubi8" --privileged=True --volume /etc/ci/mirror_info.sh:/etc/ci/mirror_info.sh:ro --ulimit=host --hostname=ubi8 molecule_local/ubi8/ubi-init bash -c "while true; do sleep 10000; done"
                                                                _uses_shell=False warn=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Aug 19 12:04:01 multi-2 systemd[22317]: Started podman-49649.scope.
Aug 19 12:04:01 multi-2 kernel: tun: Universal TUN/TAP device driver, 1.6
Aug 19 12:04:02 multi-2 ansible-async_wrapper.py[49645]: Module complete (49645)
Aug 19 12:04:02 multi-2 ansible-ansible.legacy.async_status[49724]: Invoked with jid=237660152745.49641 mode=status _async_dir=/home/stack/.ansible_async
Aug 19 12:04:04 multi-2 systemd[22317]: Started podman-49782.scope.
Aug 19 12:04:04 multi-2 systemd[22317]: Started podman-49850.scope.
Aug 19 12:04:05 multi-2 systemd[22317]: Started podman-49967.scope.
Aug 19 12:04:05 multi-2 ansible-async_wrapper.py[4964...

[Alex Schultz (alex-schultz)]

might be an ansible 2.8 problem as ussuri is using 2.8 and my local reproducer was using 4.0

2021-08-19 14:48:39.503568 | TASK [Display test-python virtualenv package versions]
2021-08-19 14:48:40.629043 | centos-8 | ansi2html==1.6.0
2021-08-19 14:48:40.629210 | centos-8 | ansible==2.8.0
2021-08-19 14:48:40.629235 | centos-8 | ansible-lint==5.1.2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-validations/+/805290

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-validations/+/805290
Committed: https://opendev.org/openstack/tripleo-validations/commit/712125d562de6e360e60a4202873f4d449d5b54f
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 712125d562de6e360e60a4202873f4d449d5b54f
Author: Alex Schultz <email address hidden>
Date: Thu Aug 19 13:28:31 2021 -0600

    [ussuri-only] Stop requiring ansible 2.8

    It appears 2.8 doesn't work with the molecule podman jobs.
    Closes-Bug: #1938266

    Change-Id: I6ee5d51537fd54ece436b64d6d7137516ccc0882

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-validations ussuri-eol release.