StarlingX

cannot apply a chart with local registry

Bug #1843915 reported by Jose Perez Carranza
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
High
Al Bailey

Bug Description

Brief Description
-----------------
Cannot deploy an application by helm chart upload which is pointing to an image on local registry
Severity
--------
Provide the severity of the defect.
Major: System/Feature is usable but degraded

Steps to Reproduce
------------------
1. Have a SXT deploymen up an running, with stx-openstack application correctly applied
2. Copy `hello-kitty.tgz` file (attached) to the active controller
3. Modify the `values.yaml` file adding the <REGISTRY IP> on:
    - charts/hello-kitty/values.yaml
    - charts/hello-kitty.tgz/hello-kitty/values.yaml

   as follows:

    images:
      pullPolicy: IfNotPresent
      tags:
        hellokitty: <REGISTRY_IP>/gwaines/hellokitty:v1.0

3. on active controller do
   $ . /etc/platform/openrc
   $ system application-upload -n hello-kitty -v 1.0 /home/sysadmin/hello-kitty.tgz
     # wait until is uploaded correctly
   $ system application-apply hello-kitty

Expected Behavior
------------------
Application should be applied correctly.

Actual Behavior
----------------
Application cannot be applied due cannot download images

NOTE: Note: If we do a `sudo docker pull : <REGISTRY_IP>/gwaines/hellokitty:v1.0` image is pulled correctly, also this works correctly when the image is downloaded from the public registry.

======== Logs ==============================================================================

2019-09-12 13:26:40.036 328907 INFO sysinv.conductor.kube_app [-] Application hello-kitty (1.0) upload completed.
2019-09-12 13:26:41.306 330854 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-12 13:26:41.381 330854 INFO sysinv.api.controllers.v1.host [-] Provisioned storage node(s) []
2019-09-12 13:26:43.448 330855 INFO sysinv.openstack.common.rpc.common [-] Connected to AMQP server on 10.10.62.2:5672
2019-09-12 13:26:43.457 328907 INFO sysinv.conductor.kube_app [-] Register the initial abort status of app hello-kitty
2019-09-12 13:26:43.586 328907 INFO sysinv.conductor.kube_app [-] Application hello-kitty (1.0) apply started.
2019-09-12 13:26:43.618 328907 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-09-12 13:26:43.682 328907 INFO sysinv.helm.manifest_base [req-202eb504-da89-4ce2-848c-18a0ea84f154 admin admin] Delete manifest file /opt/platform/armada/19.08/hello-kitty/1.0/hello-kitty-manifest-del.yaml generated
2019-09-12 13:26:43.683 328907 INFO sysinv.conductor.kube_app [-] Application overrides generated.
2019-09-12 13:26:43.784 328907 INFO sysinv.conductor.kube_app [-] Image 192.168.90.60/gwaines/hellokitty:v1.0 download started from public/private registry
2019-09-12 13:26:43.785 328907 ERROR sysinv.conductor.kube_app [-] Image 192.168.90.60/gwaines/hellokitty:v1.0 download failed from public/private registry: 'module' object has no attribute 'SERVICE_PARAM_NAME_DOCKER_DOCKER_REGISTRY'
2019-09-12 13:26:43.786 328907 ERROR sysinv.conductor.kube_app [-] Deployment of application hello-kitty (1.0) failed: failed to download one or more image(s).
2019-09-12 13:26:43.786 328907 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-09-12 13:26:43.786 328907 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1812, in perform_app_apply
2019-09-12 13:26:43.786 328907 TRACE sysinv.conductor.kube_app self._download_images(app)
2019-09-12 13:26:43.786 328907 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 763, in _download_images
2019-09-12 13:26:43.786 328907 TRACE sysinv.conductor.kube_app reason=reason)
2019-09-12 13:26:43.786 328907 TRACE sysinv.conductor.kube_app KubeAppApplyFailure: Deployment of application hello-kitty (1.0) failed: failed to download one or more image(s).
2019-09-12 13:26:43.786 328907 TRACE sysinv.conductor.kube_app
2019-09-12 13:26:44.020 328907 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.
2019-09-12 13:26:44.020 328907 INFO sysinv.conductor.kube_app [-] Deregister the abort status of app hello-kitty
2019-09-12 13:26:44.021 328907 ERROR sysinv.openstack.common.rpc.amqp [req-202eb504-da89-4ce2-848c-18a0ea84f154 admin admin] Exception during message handling
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10172, in perform_app_apply
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp app_applied = self._app.perform_app_apply(rpc_app, mode)
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1812, in perform_app_apply
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp self._download_images(app)
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 763, in _download_images
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp reason=reason)
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp KubeAppApplyFailure: Deployment of application hello-kitty (1.0) failed: failed to download one or more image(s).
2019-09-12 13:26:44.021 328907 TRACE sysinv.openstack.common.rpc.amqp

=========================================================================================

Reproducibility
---------------
<Reproducible/Intermittent/Seen once>
100%

System Configuration
--------------------
Multi-node system - Local Storage

Branch/Pull Time/Commit
-----------------------

Last Pass
---------
First Time tested

Timestamp/Logs
--------------
File hello-kitty.tgz attached

Test Activity
-------------
Pytest automated suite enablement

See original description
Revision history for this message
Jose Perez Carranza (jgperezc) wrote :
Revision history for this message
Ghada Khalil (gkhalil) wrote :

@Jose, You need to have an image secret in the application spec to be able to access the private registry. You haven't attached the yaml file, so we're suspecting you don't have an image secret defined.

Changed in starlingx:
status: New → Incomplete
Revision history for this message
Jose Perez Carranza (jgperezc) wrote :

Hi Ghada

What kind of YAML file is needed?

On the attached hello-kitty.tgz #comment1 [1] is the values.yaml and Chart.yaml files, are you are referring to those ones or another one is needed?

https://bugs.launchpad.net/starlingx/+bug/1843915/comments/1

Revision history for this message
Mingyuan Qi (myqi) wrote :

The SERVICE_PARAM_NAME_DOCKER_DOCKER_REGISTRY issue should be fixed in this commit https://review.opendev.org/#/c/676213/ a month ago. Could you confirm the iso version? there is no SERVICE_PARAM_NAME_DOCKER_DOCKER_REGISTRY in latest code any more.

Revision history for this message
Jose Perez Carranza (jgperezc) wrote :
Download full text (5.7 KiB)

Tried on the ISO with 2.0 and still having the same issue

###
### StarlingX
### Release 19.08
###

OS="centos"
SW_VERSION="19.08"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="r/stx.2.0"

JOB="STX_BUILD_2.0"
<email address hidden>"
BUILD_NUMBER="40"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-08-26 23:30:00 +0000"

----------------------------
----------------------------

2019-09-23 04:46:49.342 3079810 ERROR sysinv.conductor.kube_app [-] Image 192.168.100.60/gwaines/hellokitty:v1.0 download failed from public/private registry: 'module' object has no attribute 'SERVICE_PARAM_NAME_DOCKER_DOCKER_REGISTRY'
2019-09-23 04:46:49.342 3079810 INFO sysinv.conductor.kube_app [-] Failed to download image: 192.168.100.60/gwaines/hellokitty:v1.0
2019-09-23 04:46:49.342 3079810 INFO sysinv.conductor.kube_app [-] Retry docker images download for application hello-kitty after 30 seconds
2019-09-23 04:47:19.343 3079810 INFO sysinv.conductor.kube_app [-] Image 192.168.100.60/gwaines/hellokitty:v1.0 download started from public/private registry
2019-09-23 04:47:19.344 3079810 ERROR sysinv.conductor.kube_app [-] Image 192.168.100.60/gwaines/hellokitty:v1.0 download failed from public/private registry: 'module' object has no attribute 'SERVICE_PARAM_NAME_DOCKER_DOCKER_REGISTRY'
2019-09-23 04:47:19.344 3079810 INFO sysinv.conductor.kube_app [-] Failed to download image: 192.168.100.60/gwaines/hellokitty:v1.0
2019-09-23 04:47:19.344 3079810 INFO sysinv.conductor.kube_app [-] Retry docker images download for application hello-kitty after 30 seconds
2019-09-23 04:47:22.965 3080667 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_start_2019-09-23-04-47-22 patch
2019-09-23 04:47:22.966 3080667 INFO sysinv.api.controllers.v1.host [-] controller-0 ihost_patch_end. No changes from mtce/1.0.
2019-09-23 04:47:49.357 3079810 INFO sysinv.conductor.kube_app [-] Image 192.168.100.60/gwaines/hellokitty:v1.0 download started from public/private registry
2019-09-23 04:47:49.357 3079810 ERROR sysinv.conductor.kube_app [-] Image 192.168.100.60/gwaines/hellokitty:v1.0 download failed from public/private registry: 'module' object has no attribute 'SERVICE_PARAM_NAME_DOCKER_DOCKER_REGISTRY'
2019-09-23 04:47:49.357 3079810 INFO sysinv.conductor.kube_app [-] Failed to download image: 192.168.100.60/gwaines/hellokitty:v1.0
2019-09-23 04:47:49.358 3079810 ERROR sysinv.conductor.kube_app [-] Deployment of application hello-kitty (1.0) failed: failed to download one or more image(s).
2019-09-23 04:47:49.358 3079810 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-09-23 04:47:49.358 3079810 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1835, in perform_app_apply
2019-09-23 04:47:49.358 3079810 TRACE sysinv.conductor.kube_app self._download_images(app)
2019-09-23 04:47:49.358 3079810 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 775, in _download_images
2019-09-23 04:47:49.358 3079810 TRACE sysinv.conductor.kube_app reason=constants.APP_PROGRESS_IMAGES_DOWNLOAD_FAILED)
2019-09-23 04...

Read more...

Revision history for this message
Jose Perez Carranza (jgperezc) wrote :

just tested this on recent build on master and is working correctly, so patch should be back-ported to r/2.0 branch.

Tested on:
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.09"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190922T230000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="259"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-09-22 23:00:00 +0000"

Revision history for this message
Al Bailey (albailey1974) wrote :

Cherry pick is here.
https://review.opendev.org/#/c/676213/

Revision history for this message
Al Bailey (albailey1974) wrote :

I pointed to the original commit.
Cherry Pick is https://review.opendev.org/#/c/684415/

Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → High
status: Incomplete → In Progress
assignee: nobody → Al Bailey (albailey1974)
tags: added: stx.2.0 stx.config stx.containers
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marked as stx.2.0 gating / high priority as this impacts application deployment with local registry. The plan is to include this fix in the next stx.2.0 mtce release.

As per above, this issue was introduced on Aug 2 and only fixed in master. The fix should have been cherry-picked to the r/stx.2.0 branch at the time.

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to config (r/stx.2.0)

Reviewed: https://review.opendev.org/684415
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=3560d5b9fdd78837f639e0b0fe8fbe4b676a33eb
Submitter: Zuul
Branch: r/stx.2.0

commit 3560d5b9fdd78837f639e0b0fe8fbe4b676a33eb
Author: Angie Wang <email address hidden>
Date: Tue Aug 13 10:42:44 2019 -0400

    Support to pull container images from authenticated registries

    Fix the constants spelling error which was introduced in commit
    https://review.opendev.org/#/c/673100/.

    Co-Authored-By: Wayne Chan <email address hidden>

    Change-Id: If00e668c977c540dcd98caaf1207c49ef20897b0
    Story: 2006274
    Task: 35998
    Closes-Bug: 1843915
    Signed-off-by: Angie Wang <email address hidden>
    (cherry picked from commit 27d3d1af099251ddfba6d8ad56ecdcee418aa37e)

Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marking as Fix Released ...fix is already in master and has now been merged in r/stx.2.0

tags: added: in-r-stx20
Changed in starlingx:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.