Sysinv Startup - Download & Push Images Routine Intermittently Fails

[Joshua Reed]

Brief Description
-----------------
When the sysinv conductor starts up, one of its tasks is to kick off an ansible playbook which obtains static images required
for normal operation to be available on the system. In some cases, the local registry credentials stored in keyring are not
able to be retrieved. Sometimes the function returns None and others it works. This may be to due system load at the time
sysinv starts up, but the exact reason behind this behavior is unknown. What is known is that the number of times the software
accesses the credentials is equal to the number of images. The solution should seek to minimize that to the fewest accesses
possible.

Severity
--------
Major

Steps to Reproduce
------------------
Perform an upgrade from stx7 to stx8. After installing stx8, startup sysinv. Observe the sysinv.log for an error where the
system is unable to download an image.

Expected Behavior
------------------
At sysinv startup, if there are images to download then it should download all of them and push to local registry.

Actual Behavior
----------------
At sysinv startup, all images are downloaded. The software pushes some to the local registry but fails at other times.

Reproducibility
---------------
Intermittent

System Configuration
--------------------
Seen on SX / DX - But possible in all configurations.

Branch/Pull Time/Commit
-----------------------
master

Last Pass
---------
Unknown / Intermittent

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Sanity

Workaround
----------
Must manually pull, tag and push missing images before upgrade activation.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/898847

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/898847
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/86f739f0eb886d51842538c287e132e34c302863
Submitter: "Zuul (22348)"
Branch: master

commit 86f739f0eb886d51842538c287e132e34c302863
Author: Joshua Reed <email address hidden>
Date: Thu Oct 19 10:37:24 2023 -0700

    Reduce keyring access to once per playbook run.

    The push-docker-images role uses a python script to
    download, tag, and push images to the local registry.
    There is a multi-threaded function which does this for
    as many images as are needed. For each job, the local
    registry credentials are accessed. Sometimes, for whatever
    reason, the get_local_registry_auth function returns
    None, a None type is passed to the docker client and
    the routine fails.

    This modification minimizes this to a single credential
    access. Credentials are passed to each job as an input.

    Test Plan:

    PASS:
    1. Install AIO-SX at lower STX version.
    2. Initiate upgrade to higher STX version.
    3. Before running system upgrade ansible playbook, update
       download_images.py with code in this review.
    4. Run system upgrade playbook.
    5. Observe no errors downloading or pushing images to local
       registry in sysinv.log, user.log or platform.log

    Closes-Bug: 2039875
    Change-Id: I1ec1dc88f72b38bdb0561ffb25f4e253441b094c
    Signed-off-by: Joshua Reed <email address hidden>

[Ghada Khalil (gkhalil)]

Note: The above commit was reverted via https://review.opendev.org/c/starlingx/ansible-playbooks/+/900672