Need to handle empty ghcr.io registry in prestage images playbook

Bug #1961826 reported by Tee Ngo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Tee Ngo

Bug Description

Brief Description
-----------------
Prestage images playbook does not handle the case where the system controller is running stx6 while the subcloud is running stx5 (i.e. ghcr.io registry is defined but is empty).

Severity
--------
Major

Steps to Reproduce
------------------
Set up a DC system where the system controller is running stx6 and the subcloud(s) is running stx5
Apply the dev patch containing the subcloud prestage feature
Run subcloud prestage command on the system controller to prestage

Expected Behavior
------------------
Subcloud prestage completes successfully

Actual Behavior
----------------
Prestage images failed due to mishandling of empty ghcr.io registry

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Distributed Cloud

Branch/Pull Time/Commit
-----------------------
http://mirror.starlingx.cengn.ca/mirror/starlingx/release/6.0.0/centos/flock/outputs/iso/

Last Pass
---------
This test passed when the system controller was also running stx5 load. It fails when the system controller runs stx6 load where the support for ghcr.io registry was first introduced.

Timestamp/Logs
--------------
    500 Server Error: Internal Server Error ("manifest unknown: manifest unknown")
    Image registry.central:9001/k8s.gcr.io/kube-apiserver:v1.19.13 not found on local registry, attempt to download...
    Image download succeeded: registry.central:9001/k8s.gcr.io/kube-apiserver:v1.19.13
    500 Server Error: Internal Server Error ("manifest unknown: manifest unknown")
    Image registry.central:9001/docker.io/helm/tiller:v2.16.9 not found on local registry, attempt to download...
     HARD FAIL - Image download failed: registry.central:9001/docker.io/helm/tiller:v2.16.9 404 Client Error: Not Found ("manifest for registry.central:9001/docker.io/helm/tiller:v2.16.9 not found")
    500 Server Error: Internal Server Error ("manifest unknown: manifest unknown")
    Image registry.central:9001/docker.io/k8snetworkplumbingwg/multus-cni:v3.7.1 not found on local registry, attempt to download...
     HARD FAIL - Image download failed: registry.central:9001/docker.io/k8snetworkplumbingwg/multus-cni:v3.7.1 404 Client Error: Not Found ("manifest for registry.central:9001/docker.io/k8snetworkplumbingwg/multus-cni:v3.7.1 not found")
    Local download flag: no
    Traceback (most recent call last):
      File "/tmp/.ansible-sysadmin/tmp/ansible-tmp-1645550248.67-16555307709979/download_images.py", line 333, in <module>
        raise Exception("Failed to download images %s" % failed_downloads)
    Exception: Failed to download images [u'registry.central:9001/docker.io/helm/tiller:v2.16.9', u'registry.central:9001/docker.io/k8snetworkplumbingwg/multus-cni:v3.7.1']
  stdout_lines: <omitted>

Test Activity
-------------
Feature Testing

Workaround
----------
None

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (master)
Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (master)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/830515
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/7fe29d2400578e9ad8df9d8369c6ff8ed5d8b61b
Submitter: "Zuul (22348)"
Branch: master

commit 7fe29d2400578e9ad8df9d8369c6ff8ed5d8b61b
Author: Tee Ngo <email address hidden>
Date: Tue Feb 22 13:50:47 2022 -0500

    Handle empty ghcr.io registry

    Update the prestage_images.yml playbook to handle the case where
    ghcr.io registry is defined but is empty.

    Test Plan:
      - Run the prestage_images.yml playbook in a simulated environment
        as described in the launchpad and verify that ghcr.io related
        images are downloaded and included in the archive bundles.

    Closes-Bug: 1961826
    Change-Id: Ic5555ebd146a049a096d7edbc9ed6e3bebf2b649
    Signed-off-by: Tee Ngo <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
tags: added: stx.7.0 stx.config
Changed in starlingx:
importance: Undecided → Medium
assignee: nobody → Tee Ngo (teewrs)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.