Distributed Cloud Ipv6: subcloud customer app Failed to download image from registry.central

[Peng Peng]

Brief Description
-----------------
Apply customer app, like hello-kitty, on system controller. Make sure customer app docker image is under docker.io. Try to apply same app in subcloud, but failed by "Failed to download image from registry.central"

Severity
--------
Major

Steps to Reproduce
------------------
as description

TC-name:

Expected Behavior
------------------

Actual Behavior
----------------

Reproducibility
---------------
Reproducible

System Configuration
--------------------
DC
IPv6

Lab-name: DC

Branch/Pull Time/Commit
-----------------------
2019-11-21_20-00-00

Last Pass
---------
unknow

Timestamp/Logs
--------------

System controller:
[sysadmin@controller-1 custom_apps(keystone_admin)]$ sudo docker images | grep hello
Password:
gwaines/hellokitty v1.0 3f3a386d794e 7 weeks ago 148MB
tis-lab-registry.cumulus.wrs.com:9001/wrcp-staging/docker.io/gwaines/hellokitty v1.0 3f3a386d794e 7 weeks ago 148MB
[sysadmin@controller-1 custom_apps(keystone_admin)]$ ls /var/lib/docker-distribution/docker/registry/v2/repositories/docker.io/
starlingx

subcloud6:
[sysadmin@controller-0 ~(keystone_admin)]$ tailf /var/log/sysinv.log
sysinv 2019-11-27 19:29:25.048 85461 INFO sysinv.conductor.kube_app [-] Image docker.io/gwaines/hellokitty:v1.0 download started from public/private registry
sysinv 2019-11-27 19:29:25.407 85461 ERROR sysinv.conductor.kube_app [-] Image docker.io/gwaines/hellokitty:v1.0 download failed from public/private registry: 404 Client Error: Not Found ("manifest for registry.central:9001/docker.io/gwaines/hellokitty:v1.0 not found"): NotFound: 404 Client Error: Not Found ("manifest for registry.central:9001/docker.io/gwaines/hellokitty:v1.0 not found")
sysinv 2019-11-27 19:29:25.408 85461 INFO sysinv.conductor.kube_app [-] Failed to download image: docker.io/gwaines/hellokitty:v1.0

Test Activity
-------------
Regression Testing

[Peng Peng (ppeng)]

Subcloud6 collect logs

[Angie Wang (angiewang)]

Currently, we are not supporting custom apps to pull images from local registry, so images won't exist on system controller's registry. In the case that subcloud configured with registry.central, it will fail for custom apps due to images do not exist in central registry.

The fix for it is to support pulling images from local registry for custom app.
There are existing methods from class AppImageParser in sysinv that be used for support pulling images from local registry for system apps, those can be leveraged for custom apps as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/696631

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696631
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=284139cfd0d51277b00e3d10992c10d8d0081626
Submitter: Zuul
Branch: master

commit 284139cfd0d51277b00e3d10992c10d8d0081626
Author: Angie Wang <email address hidden>
Date: Thu Nov 28 17:28:39 2019 -0500

    Support pulling images from local registry for custom apps

    Currently, custom apps are not pulling images from local registry,
    this commit updates to support it.

    Tested on AIO-DX:
    - application-upload and apply hello-kitty, verified that
      images were pulled from local registry
    - application-upload and apply system apps

    Change-Id: I44346ed7c13dba43a78c0c354464883061c8fada
    Closes-Bug: 1854226
    Signed-off-by: Angie Wang <email address hidden>

[Ghada Khalil (gkhalil)]

stx.3.0 / high priority as this affects custom app deployment on Distributed Cloud

[Ghada Khalil (gkhalil)]

The next step is to cherrypick to the r/stx.3.0 branch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.3.0
Review: https://review.opendev.org/696937

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696937
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=f6f3bd63ef64739957bdcad13152f7de7293382d
Submitter: Zuul
Branch: r/stx.3.0

commit f6f3bd63ef64739957bdcad13152f7de7293382d
Author: Angie Wang <email address hidden>
Date: Thu Nov 28 17:28:39 2019 -0500

    Support pulling images from local registry for custom apps

    Currently, custom apps are not pulling images from local registry,
    this commit updates to support it.

    Tested on AIO-DX:
    - application-upload and apply hello-kitty, verified that
      images were pulled from local registry
    - application-upload and apply system apps

    Change-Id: I44346ed7c13dba43a78c0c354464883061c8fada
    Closes-Bug: 1854226
    Signed-off-by: Angie Wang <email address hidden>
    (cherry picked from commit 284139cfd0d51277b00e3d10992c10d8d0081626)

[Peng Peng (ppeng)]

Verified on

BUILD_ID="2019-12-13_19-03-42"