Bootstrap of controller-0 failing due to missing tag in gcr.io registry

Bug #1940696 reported by OpenInfra
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Critical
Mihnea Saracin

Bug Description

Provisioned StarlingX release 5, standard with dedicated storage.

Bootstrap of controller-0 failing as 'gcr.io/kubernetes-helm/tiller:v2.16.1' is no longer available.

fatal: [localhost]: FAILED! => {
    "changed": true,
    "msg": "non-zero return code",
    "rc": 1,
    "stderr": "Traceback (most recent call last):\n File \"/tmp/.ansible-root/tmp/ansible-tmp-1629451316.05-110850422460737/download_images.py\", line 144, in <module>\n raise Exception(\"Failed to download images %s\" % failed_downloads)\nException: Failed to download images ['gcr.io/kubernetes-helm/tiller:v2.16.1']\n",
    "stderr_lines": [
        "Traceback (most recent call last):",
        " File \"/tmp/.ansible-root/tmp/ansible-tmp-1629451316.05-110850422460737/download_images.py\", line 144, in <module>",
        " raise Exception(\"Failed to download images %s\" % failed_downloads)",
        "Exception: Failed to download images ['gcr.io/kubernetes-helm/tiller:v2.16.1']"
    ],

Revision history for this message
OpenInfra (openinfra) wrote :

Changed updated the tag in [1], [2] files with tiller:v2.16.10.

[1] /usr/share/ansible/stx-ansible/playbooks/roles/common/load-images-information/vars/k8s-v1.18.1/system-images.yml

[2] /usr/share/ansible/stx-ansible/playbooks/roles/common/armada-helm/templates/armada-overrides.yaml.j2

description: updated
Revision history for this message
OpenInfra (openinfra) wrote :

Bootstrap complted as expected after the above mentioned tag update (tiller:v2.16.10.)

TASK [bootstrap/bringup-bootstrap-applications : Mark the bootstrap as completed] ****************************************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************
localhost : ok=314 changed=168 unreachable=0 failed=0

Revision history for this message
Bob Church (rchurch) wrote :
Ghada Khalil (gkhalil)
tags: added: stx.containers
Revision history for this message
Ghada Khalil (gkhalil) wrote :

screening: stx.5.0 / critical - The tagged version of tiller in stx.5.0 has disappeared from upstream as per the email thread to the mailing list: http://lists.starlingx.io/pipermail/starlingx-discuss/2021-August/011927.html

This will need to be corrected in the r/stx.5.0 release branch and a mtce release will need to be built.

Changed in starlingx:
importance: Undecided → Critical
status: New → Triaged
tags: added: stx.5.0
Changed in starlingx:
assignee: nobody → Mihnea Saracin (msaracin)
Revision history for this message
OpenInfra (openinfra) wrote :

I found a typo in stx-platform-helm/stx-platform-helm/helm-charts/rbd-provisioner/templates/deployment.yaml

diff --git a/stx-platform-helm/stx-platform-helm/helm-charts/rbd-provisioner/templates/deployment.yaml b/stx-platform-helm/stx-platform-helm/helm-charts/rbd-provisioner/templates/deployment.yaml
index 5bda622..718aef7 100644
--- a/stx-platform-helm/stx-platform-helm/helm-charts/rbd-provisioner/templates/deployment.yaml
+++ b/stx-platform-helm/stx-platform-helm/helm-charts/rbd-provisioner/templates/deployment.yaml
@@ -50,7 +50,7 @@ spec:
 {{- if .Values.global.tolerations }}
       tolerations:
 {{ .Values.global.tolerations | toYaml | trim | indent 8 }}
-{{- end}}
+{{- end }}
 {{- if .Values.global.resources }}
       resources:
 {{ .Values.global.resources | toYaml | trim | indent 8 }}

Ghada Khalil (gkhalil)
Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Code has merged in the r/stx.5.0 branch:
https://review.opendev.org/c/starlingx/platform-armada-app/+/805812
https://review.opendev.org/c/starlingx/ansible-playbooks/+/805813

Next step is to build an stx.5.0 RC build for sanity

Changed in starlingx:
status: In Progress → Fix Committed
tags: added: in-r-stx50
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (r/stx.5.0)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (master)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/806912
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/5ac4f922f1494512579bcafcf44be45f4a0f4085
Submitter: "Zuul (22348)"
Branch: master

commit 5ac4f922f1494512579bcafcf44be45f4a0f4085
Author: Daniel Safta <email address hidden>
Date: Wed Sep 1 11:12:02 2021 +0000

    Updated tiller image source

    The source of the tiller image changed
    from gcr.io/kubernetes-helm/tiller:v2.16.9
    to ghcr.io/helm/tiller:v2.16.9 .

    This commit brings the required fixes
    to update the image source.

    Closes-bug: 1940696
    Signed-off-by: Daniel Safta <email address hidden>
    Change-Id: Icd5f3d8895957815a50c0c13e9490cd550e000da

Changed in starlingx:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (r/stx.5.0)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/807101
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/8574a1093904d32b634686136f49181cdd0c8af6
Submitter: "Zuul (22348)"
Branch: r/stx.5.0

commit 8574a1093904d32b634686136f49181cdd0c8af6
Author: Daniel Safta <email address hidden>
Date: Wed Sep 1 11:12:02 2021 +0000

    Updated tiller image source

    The source of the tiller image changed
    from gcr.io/kubernetes-helm/tiller:v2.16.9
    to ghcr.io/helm/tiller:v2.16.9 .

    This commit brings the required fixes
    to update the image source.

    Closes-bug: 1940696
    Signed-off-by: Daniel Safta <email address hidden>
    Change-Id: Icd5f3d8895957815a50c0c13e9490cd550e000da

Revision history for this message
OpenInfra (openinfra) wrote (last edit ):

tag v2.16.9 is not available. at ghcr.io/helm/tiller

I have tested the above mentioned fix at comment #9 by manually change in release 5.0 image but bootstrap still fail due to missing tag of v2.16.9.

Revision history for this message
OpenInfra (openinfra) wrote :

I have tested the above mentioned fix at comment #9 by manually change in release 5.0 image but bootstrap still fail due to missing tag of v2.16.9.

Revision history for this message
OpenInfra (openinfra) wrote :

Please ignore my previous comments #10 and #11.
Sorry for the inconvenience .

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers