Subcloud prestage post upgrade/reinstall using prestaged data fails

[Tee Ngo]

Brief Description
-----------------
Subcloud prestage would fail if the following conditions are met a) it was previously upgraded or reinstalled using prestaged data and b) the prestage image list contains any stx-monitor images that start with docker.elastic.co

Severity
--------
Major

Steps to Reproduce
------------------
1. Bring up a DC system with 1 simplex subcloud
2. Apply stx-monitor app on both system controller and subcloud
2. Upgrade the system controller
3. Upload the prestage image list containing docker.elastic.co/beats/filebeat:7.9.3
3. Prestage the simplex subcloud for upgrade
4. Upgrade the subcloud
5. Prestage the subcloud using dcmanager subcloud prestage command

A simpler way to reproduce the issue is as-follows
1. Bring up a DC system with 1 simplex subcloud
2. On the subcloud, manually pull image docker.elastic.co/beats/filebeat:7.9.3 the tag the image as follows:
docker image tag docker.elastic.co/beats/filebeat:7.9.3 registry.local:9001/docker.elastic.co/beats/filebeat:7.9.3

docker image tag docker.elastic.co/beats/filebeat:7.9.3 registry.local:9001/docker.io/docker.elastic.co/beats/filebeat:7.9.3

then push both to the local registry
docker image push registry.local:9001/docker.elastic.co/beats/filebeat:7.9.3
docker image push registry.local:9001/docker.io/docker.elastic.co/beats/filebeat:7.9.3

3. Prestage the subcloud using dcmanager subcloud prestage command

Expected Behavior
------------------
Subcloud prestage succeeds

Actual Behavior
----------------
Subcloud prestage failed during generate image bundle step due to
Error: No such image: docker.io/docker.elastic.co/beats/filebeat:7.9.3

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Distributed Cloud

Branch/Pull Time/Commit
-----------------------
March 31st, 2022 master build

Last Pass
---------
First time testing this case

Timestamp/Logs
--------------
The issue is readily reproducible

Test Activity
-------------
Feature Testing

Workaround
----------
Manually remove docker.io/docker.elastic.co/beats/filebeat:7.9.3 if exists from the subcloud local registry before performing prestage.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/837453

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/837453
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/8f726b78345bbdfd557b9d270d4384f1029b9b37
Submitter: "Zuul (22348)"
Branch: master

commit 8f726b78345bbdfd557b9d270d4384f1029b9b37
Author: Tee Ngo <email address hidden>
Date: Mon Apr 11 23:31:15 2022 -0400

    Fix docker.elastic.co images import issue

    A code flaw in importing non *.io images from archive files into the
    local registry caused subcloud prestage to fail when the following
    conditions are met:
      a. The subcloud was upgraded/re-installed previously using prestaged
         data.
      b. At least one of the prestaged images was a docker.elastic.co
         image (e.g. docker.elastic.co/beats/filebeat:7.9.3).

    The solution is to fix the root cause and to also handle subclouds
    that are already in such state.

    Test Plan:
      - Verify that a DC system with at least one simplex
        subcloud is deployed successfully.
      - Verify that the stx-monitor app is applied successfully
        on both the system controller and the subcloud.
      - Prestage the simplex subcloud that has stx-monitor app applied,
        delete the subcloud, redeploy the subcloud using prestaged data,
        and finally apply stx-monitor app on the subcloud. Verify that
        subcloud prestage post redeployment completes successfully.

    Closes-Bug: 1968649
    Change-Id: I9ff4f18d7ce45941d475596f2357245c3c0851f7
    Signed-off-by: Tee Ngo <email address hidden>