Bootstrap of controller-0 failing due to missing tag in gcr.io registry

[OpenInfra]

Provisioned StarlingX release 5, standard with dedicated storage.

Bootstrap of controller-0 failing as 'gcr.io/kubernetes-helm/tiller:v2.16.1' is no longer available.

fatal: [localhost]: FAILED! => {
    "changed": true,
    "msg": "non-zero return code",
    "rc": 1,
    "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1629451316.05-110850422460737/download_images.py\", line 144, in <module>\n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['gcr.io/kubernetes-helm/tiller:v2.16.1']\n",
    "stderr_lines": [
        "Traceback (most recent call last):",
        " File \"/tmp/.ansible-root/tmp/ansible-tmp-1629451316.05-110850422460737/download_images.py\", line 144, in <module>",
        " raise Exception(\"Failed to download images %s\" % failed_downloads)",
        "Exception: Failed to download images ['gcr.io/kubernetes-helm/tiller:v2.16.1']"
    ],

[OpenInfra (openinfra)]

Changed updated the tag in [1], [2] files with tiller:v2.16.10.

[1] /usr/share/ansible/stx-ansible/playbooks/roles/common/load-images-information/vars/k8s-v1.18.1/system-images.yml

[2] /usr/share/ansible/stx-ansible/playbooks/roles/common/armada-helm/templates/armada-overrides.yaml.j2

[OpenInfra (openinfra)]

Bootstrap complted as expected after the above mentioned tag update (tiller:v2.16.10.)

TASK [bootstrap/bringup-bootstrap-applications : Mark the bootstrap as completed] ****************************************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************
localhost : ok=314 changed=168 unreachable=0 failed=0

[Bob Church (rchurch)]

The fix for this is to cherry-pick the following commits from master to r/stx.5.0:
 - https://opendev.org/starlingx/ansible-playbooks/commit/fcde313c6b8c91ae76eb71f2963c7928db6cbec2
 - https://opendev.org/starlingx/platform-armada-app/commit/7d30242557401bfdb0be5f75bd29ce05708d250f

[Ghada Khalil (gkhalil)]

screening: stx.5.0 / critical - The tagged version of tiller in stx.5.0 has disappeared from upstream as per the email thread to the mailing list: http://lists.starlingx.io/pipermail/starlingx-discuss/2021-August/011927.html

This will need to be corrected in the r/stx.5.0 release branch and a mtce release will need to be built.

[OpenInfra (openinfra)]

I found a typo in stx-platform-helm/stx-platform-helm/helm-charts/rbd-provisioner/templates/deployment.yaml

diff --git a/stx-platform-helm/stx-platform-helm/helm-charts/rbd-provisioner/templates/deployment.yaml b/stx-platform-helm/stx-platform-helm/helm-charts/rbd-provisioner/templates/deployment.yaml
index 5bda622..718aef7 100644
--- a/stx-platform-helm/stx-platform-helm/helm-charts/rbd-provisioner/templates/deployment.yaml
+++ b/stx-platform-helm/stx-platform-helm/helm-charts/rbd-provisioner/templates/deployment.yaml
@@ -50,7 +50,7 @@ spec:
 {{- if .Values.global.tolerations }}
       tolerations:
 {{ .Values.global.tolerations | toYaml | trim | indent 8 }}
-{{- end}}
+{{- end }}
 {{- if .Values.global.resources }}
       resources:
 {{ .Values.global.resources | toYaml | trim | indent 8 }}

[Ghada Khalil (gkhalil)]

Code has merged in the r/stx.5.0 branch:
https://review.opendev.org/c/starlingx/platform-armada-app/+/805812
https://review.opendev.org/c/starlingx/ansible-playbooks/+/805813

Next step is to build an stx.5.0 RC build for sanity

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.5.0
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/807101

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/806912
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/5ac4f922f1494512579bcafcf44be45f4a0f4085
Submitter: "Zuul (22348)"
Branch: master

commit 5ac4f922f1494512579bcafcf44be45f4a0f4085
Author: Daniel Safta <email address hidden>
Date: Wed Sep 1 11:12:02 2021 +0000

    Updated tiller image source

    The source of the tiller image changed
    from gcr.io/kubernetes-helm/tiller:v2.16.9
    to ghcr.io/helm/tiller:v2.16.9 .

    This commit brings the required fixes
    to update the image source.

    Closes-bug: 1940696
    Signed-off-by: Daniel Safta <email address hidden>
    Change-Id: Icd5f3d8895957815a50c0c13e9490cd550e000da

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/807101
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/8574a1093904d32b634686136f49181cdd0c8af6
Submitter: "Zuul (22348)"
Branch: r/stx.5.0

commit 8574a1093904d32b634686136f49181cdd0c8af6
Author: Daniel Safta <email address hidden>
Date: Wed Sep 1 11:12:02 2021 +0000

    Updated tiller image source

    The source of the tiller image changed
    from gcr.io/kubernetes-helm/tiller:v2.16.9
    to ghcr.io/helm/tiller:v2.16.9 .

    This commit brings the required fixes
    to update the image source.

    Closes-bug: 1940696
    Signed-off-by: Daniel Safta <email address hidden>
    Change-Id: Icd5f3d8895957815a50c0c13e9490cd550e000da

[OpenInfra (openinfra)]

tag v2.16.9 is not available. at ghcr.io/helm/tiller

I have tested the above mentioned fix at comment #9 by manually change in release 5.0 image but bootstrap still fail due to missing tag of v2.16.9.

[OpenInfra (openinfra)]

I have tested the above mentioned fix at comment #9 by manually change in release 5.0 image but bootstrap still fail due to missing tag of v2.16.9.

[OpenInfra (openinfra)]

Please ignore my previous comments #10 and #11.
Sorry for the inconvenience .