stx-monitor images are pulled from the public registry when a private registry is configured

[Chris Winnicki]

Brief Description
-----------------
stx-monitor images are pulled from the public registry when a private registry is configured

Severity
--------
Provide the severity of the defect.
Major

Steps to Reproduce
------------------
Install a AIO-DX system using example localhost.yml (below)

Expected Behavior
------------------
If a private registry is configured during the bootstrap
all images should be pulled from the private registry

Actual Behavior
----------------
During:
system application-apply stx-monitor
Images are pulled from the public registry even though a private registry is configured.

Example of localhost.yml

###############################################################################
[sysadmin@controller-1 ~(keystone_admin)]$ cat localhost.yml
system_mode: duplex
dns_servers:
  - 2620:10a:a001:a103::2

management_subnet: face::/64
management_multicast_subnet: ff05::1b:0/124
cluster_host_subnet: feed:beef::/64
cluster_pod_subnet: dead:beef::/64
cluster_service_subnet: fd04::/112

external_oam_subnet: 2620:10a:a001:a103::6:0/64
external_oam_gateway_address: 2620:10a:a001:a103::6:0
external_oam_floating_address: 2620:10A:A001:A103::11
external_oam_node_0_address: 2620:10A:A001:A103::8
external_oam_node_1_address: 2620:10A:A001:A103::9

admin_password: Secret2019pass^
ansible_become_pass: Secret2019pass^
pxeboot_subnet: 192.168.202.0/24
no_log: false

docker_registries:
    k8s.gcr.io:
        url: tis-lab-docker-registry.cumulus.wrs.com/625619392498.dkr.ecr.us-west-2.amazonaws.com/k8s.gcr.io
    gcr.io:
        url: tis-lab-docker-registry.cumulus.wrs.com/625619392498.dkr.ecr.us-west-2.amazonaws.com/gcr.io
    quay.io:
        url: tis-lab-docker-registry.cumulus.wrs.com/625619392498.dkr.ecr.us-west-2.amazonaws.com/quay.io
    docker.io:
        url: tis-lab-docker-registry.cumulus.wrs.com/625619392498.dkr.ecr.us-west-2.amazonaws.com/docker.io
    docker.elastic.co:
        url: tis-lab-docker-registry.cumulus.wrs.com/625619392498.dkr.ecr.us-west-2.amazonaws.com/docker.elastic.co
    defaults:
        type: docker
        username: username
        password: userpassword

ssl_ca_cert: /home/sysadmin/cumulus-docker-registry-ca-cert.pem

docker_no_proxy:
  - registry.local
  - tis-lab-docker-registry.cumulus.wrs.com
###############################################################################

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
AIO-DX
Wind River Lab: cgcs-r430-3-4

Branch/Pull Time/Commit
-----------------------
2019-11-02_08-39-54

Last Pass
---------
Not known

Timestamp/Logs
--------------
Attached

Test Activity
-------------
Feature Testing

Other info:
Note the paths to: "Image" on one of the pods in monitor namespace:

[sysadmin@controller-1 ~(keystone_admin)]$ kubectl describe pod mon-filebeat-kw5hd -n monitor
Name: mon-filebeat-kw5hd
Namespace: monitor
Priority: 0
Node: controller-1/face::3
Start Time: Mon, 04 Nov 2019 17:03:06 +0000
Labels: app=filebeat
              controller-revision-hash=5d485d658b
              pod-template-generation=1
              release=mon-filebeat
Annotations: checksum/secret: ee1fc16c6db82048868f6bb32c8d088ccac28fc76d96d9475e992d46330ad259
              cni.projectcalico.org/podIP: dead:beef::a4ce:fec1:5423:e30c/128
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "chain",
                    "ips": [
                        "dead:beef::a4ce:fec1:5423:e30c"
                    ],
                    "default": true,
                    "dns": {}
                }]
Status: Running
IP: dead:beef::a4ce:fec1:5423:e30c
IPs:
  IP: dead:beef::a4ce:fec1:5423:e30c
Controlled By: DaemonSet/mon-filebeat
Init Containers:
  setup-script:
    Container ID: docker://0b78c127aeb815c255b179b30ea80f80f96f5fe88cd2a75638befe91672d942b
    Image: docker.elastic.co/beats/filebeat-oss:7.4.0
    Image ID: docker-pullable://docker.elastic.co/beats/filebeat-oss@sha256:ba7b786c8372ed18b58bea4c9c2e6192997bc3251b4c9d42eb1e2a60e7bb02d8
    Port: <none>
    Host Port: <none>
    Command:
      /bin/bash
      -c
      /usr/share/filebeat/setup-script.sh
    State: Terminated
      Reason: Completed
      Exit Code: 0
      Started: Mon, 04 Nov 2019 17:03:13 +0000
      Finished: Mon, 04 Nov 2019 17:03:44 +0000
    Ready: True
    Restart Count: 0
    Environment:
      POD_NAMESPACE: monitor (v1:metadata.namespace)
      NODE_NAME: (v1:spec.nodeName)
      OUTPUT_ELASTICSEARCH_ENABLED: false
      OUTPUT_ELASTICSEARCH_HOSTS: [http://mon-elasticsearch-client:9200]
      OUTPUT_ELASTICSEARCH_ILM.PATTERN: 000001
      OUTPUT_ELASTICSEARCH_INDEX: ${INDEX_NAME}-%{+yyyy.MM.dd}
      SYSTEM_NAME_FOR_INDEX: -yow-cgcs-r430-3-4
      INDEX_PATTERN: filebeat-%{[agent.version]}-yow-cgcs-r430-3-4-*
      INDEX_NAME: filebeat-%{[agent.version]}-yow-cgcs-r430-3-4
    Mounts:
      /usr/share/filebeat/filebeat.yml from filebeat-config (ro,path="filebeat.yml")
      /usr/share/filebeat/setup-script.sh from setupscript (rw,path="setup-script.sh")
      /var/run/secrets/kubernetes.io/serviceaccount from mon-filebeat-token-dvb5b (ro)
Containers:
  filebeat:
    Container ID: docker://4d327e5c779530ccbf1dfb12081ddceed6d532ca251f158c16504a768d67d6ae
    Image: docker.elastic.co/beats/filebeat-oss:7.4.0
    Image ID: docker-pullable://docker.elastic.co/beats/filebeat-oss@sha256:ba7b786c8372ed18b58bea4c9c2e6192997bc3251b4c9d42eb1e2a60e7bb02d8
    Port: 5066/TCP
    Host Port: 0/TCP
    Args:
      -e
    State: Running
      Started: Mon, 04 Nov 2019 17:03:44 +0000
    Ready: True
    Restart Count: 0
    Limits:
      cpu: 80m
      memory: 256Mi
    Requests:
      cpu: 40m
      memory: 256Mi
    Environment:
      POD_NAMESPACE: monitor (v1:metadata.namespace)
      NODE_NAME: (v1:spec.nodeName)
      OUTPUT_ELASTICSEARCH_ENABLED: false
      OUTPUT_ELASTICSEARCH_HOSTS: [http://mon-elasticsearch-client:9200]
      OUTPUT_ELASTICSEARCH_ILM.PATTERN: 000001
      OUTPUT_ELASTICSEARCH_INDEX: ${INDEX_NAME}-%{+yyyy.MM.dd}
      SYSTEM_NAME_FOR_INDEX: -yow-cgcs-r430-3-4
      INDEX_PATTERN: filebeat-%{[agent.version]}-yow-cgcs-r430-3-4-*
      INDEX_NAME: filebeat-%{[agent.version]}-yow-cgcs-r430-3-4
    Mounts:
      /usr/share/filebeat/data from data (rw)
      /usr/share/filebeat/filebeat.yml from filebeat-config (ro,path="filebeat.yml")
      /usr/share/filebeat/setup-script.sh from setupscript (rw,path="setup-script.sh")
      /var/lib/docker/containers from varlibdockercontainers (ro)
      /var/log from varlog (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from mon-filebeat-token-dvb5b (ro)
  mon-filebeat-prometheus-exporter:
    Container ID: docker://003edf10f73925024bcb1242ca9e35cba872f5489aa35e466dd19628d76258f5
    Image: trustpilot/beat-exporter:0.1.1
    Image ID: docker-pullable://trustpilot/beat-exporter@sha256:78640014debdeed14867b4dbd8d081e38df37f438e35994624458c80c7681eb7
    Port: 9479/TCP
    Host Port: 0/TCP
    State: Running
      Started: Mon, 04 Nov 2019 17:03:46 +0000
    Ready: True
    Restart Count: 0
    Environment: <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from mon-filebeat-token-dvb5b (ro)
Conditions:
  Type Status
  Initialized True
  Ready True
  ContainersReady True
  PodScheduled True
Volumes:
  varlog:
    Type: HostPath (bare host directory volume)
    Path: /var/log
    HostPathType:
  varlibdockercontainers:
    Type: HostPath (bare host directory volume)
    Path: /var/lib/docker/containers
    HostPathType:
  filebeat-config:
    Type: Secret (a volume populated by a Secret)
    SecretName: mon-filebeat
    Optional: false
  data:
    Type: HostPath (bare host directory volume)
    Path: /var/lib/filebeat
    HostPathType: DirectoryOrCreate
  setupscript:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: mon-filebeat
    Optional: false
  mon-filebeat-token-dvb5b:
    Type: Secret (a volume populated by a Secret)
    SecretName: mon-filebeat-token-dvb5b
    Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
                 services=disabled:NoExecute
Events: <none>

[John Kung (john-kung)]

The image pull is from local image registry on subsequent apply on controller-0; however on the other controller and compute, the image pull is from public repo.

The Elastic charts, primarily ServiceAccount v1, were updated as per attached diff (elastic_service_account_1.diff) - attempt to pull from local registry though encounters:
  Warning Failed 99s (x3 over 2m15s) kubelet, controller-1 Failed to pull image "registry.local:9001/docker.elastic.co/elasticsearch/elasticsearch-oss:7.4.0": rpc error: code = Unknown desc = Error response from daemon: Get https://registry.local:9001/v2/docker.elastic.co/elasticsearch/elasticsearch-oss/manifests/7.4.0: unauthorized: authentication required

In order to pull from the local registry, further investigation into the following areas is required:
- armada manifest needs to be updated to pull from local registry
- elastic charts need to be able to handle pull with secrets for the local registry

[John Kung (john-kung)]

With the updated diff elastic_service_account_2.diff, the authentication issue on pull from local registry noted in comment above (https://bugs.launchpad.net/starlingx/+bug/1851294/comments/3) is resolved.

A corresponding config-repo sysinv change is required to setup the download images.

[Ghada Khalil (gkhalil)]

stx.3.0 / medium priority - seems to affect a specific registry configuration

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693918

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693922

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693918
Committed: https://git.openstack.org/cgit/starlingx/monitor-armada-app/commit/?id=352430351cce0c4bf75b75ba181f12c209fbf23a
Submitter: Zuul
Branch: master

commit 352430351cce0c4bf75b75ba181f12c209fbf23a
Author: John Kung <email address hidden>
Date: Tue Nov 12 15:10:41 2019 -0500

    Update elastic charts to enable docker images from local repository

    Update manifest with imagePullSecrets or serviceaccount in order to
    allow stx-monitor to pull images from local docker image repository.

    The armada manifest has registry.local included until sysinv is updated
    to be able to handle image overrides with the various elastic formats.

    Verified that 'kubectl describe pods -n monitor' Image: are
    pulled from registry.local

    Verified elastic-curator and distributed cloud configuration with
    stx-monitor.

    Change-Id: Ia63e593bef96ea011cffbcca49ccf5a3a0f2f4d7
    Partial-Bug: 1851294
    Signed-off-by: John Kung <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693922
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=31d10b72e7e4d3e2cbfe7ed38df3097602539e38
Submitter: Zuul
Branch: master

commit 31d10b72e7e4d3e2cbfe7ed38df3097602539e38
Author: Angie Wang <email address hidden>
Date: Tue Nov 12 14:51:32 2019 -0500

    Update sysinv to handle various image formats

    This commit updates sysinv to handle more image formats that
    specified in application manifest(ie..stx-monitor app) in
    order to get the full list of required download images.

    A follow-up commit will be made to update armada manifest with
    the correct format of image reference with registry.local prefix
    and support user-overrides for various image formats.

    Change-Id: I22858226724bd95b0fcbb45fb43f5577859fb467
    Partial-Bug: 1851294
    Depends-On: https://review.opendev.org/#/c/693918/
    Signed-off-by: Angie Wang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/696123

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/696124

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696123
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=d1feebccd97b85b9d2f09da793c318e095f7b878
Submitter: Zuul
Branch: master

commit d1feebccd97b85b9d2f09da793c318e095f7b878
Author: Angie Wang <email address hidden>
Date: Sun Nov 24 23:16:21 2019 -0500

    Support to handle various image formats for application

    Add AppImageParser to enhance the process for the various
    image formats for application in order to get a list of
    required download images and correctly update manifest file
    and override files with local registry prefix.

    Tested:
     - application upload and apply (platform-integ-apps,
       stx-openstack, stx-monitor, hello-kitty)
     - helm-override-update to override images and reapply
     - verified that images download list generated correctly
       and images were pulled from local registry on all nodes

    Change-Id: I57a3c3b1d81d9d8344eff2afa629c9d69629c49c
    Partial-Bug: 1851294
    Signed-off-by: Angie Wang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696124
Committed: https://git.openstack.org/cgit/starlingx/monitor-armada-app/commit/?id=3a0c00ee399cd739976b53d8288ae731674c926e
Submitter: Zuul
Branch: master

commit 3a0c00ee399cd739976b53d8288ae731674c926e
Author: Angie Wang <email address hidden>
Date: Sun Nov 24 23:18:41 2019 -0500

    Update stx-monitor manifest

    The commit https://review.opendev.org/#/c/696123/ has
    updated to handle various image formats for application
    to support pulling images from local registry.

    This commit updates the manifest to remove the unnecessary
    image references and specify "null" for the images that
    are not required for the app.

    Change-Id: I6517ba3cb7173c70b8c00807dcb58f73ec426c62
    Depends-On: https://review.opendev.org/#/c/696123/
    Closes-Bug: 1851294
    Signed-off-by: Angie Wang <email address hidden>

[Peng Peng (ppeng)]

Issue fixed.

[Ghada Khalil (gkhalil)]

Angie/Frank, This LP is marked as gating for stx.3.0. Please cherry-pick the code changes to the stx.3.0 branch if applicable or add a note explaining why it shouldn't be cherry-picked.