Ansible bootstrap fails - https://kubernetes-charts.storage.googleapis.com cannot be reached

[Cristopher Lemus]

Brief Description
-----------------
On environments with Proxy or Local registry, ansible bootstrap fails because it cannot reach https://kubernetes-charts.storage.googleapis.com

Severity
--------
Critical

Steps to Reproduce
------------------
Run ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml

Expected Behavior
------------------
Playbook should complete without errors, using either proxy configuration provided in localhost.yml, or mirror (local) registry images.

Actual Behavior
----------------
Playbook fails on task TASK [bootstrap/bringup-essential-services : Initialize Helm (local host client only)]

Reproducibility
---------------
100%

System Configuration
--------------------
All configs, virtual and baremetal, either using local registry or proxy.

Branch/Pull Time/Commit
-----------------------
BUILD_ID="20191003T013000Z"

Last Pass
---------
Image from Oct/2nd didn't have this issue.

Timestamp/Logs
--------------

Errors using local registry: http://paste.openstack.org/show/780903/
Errors using proxy: http://paste.openstack.org/show/780904/
Logs (collect) attached.

Test Activity
-------------
Sanity

[Cristopher Lemus (cjlemusc)]

tar with collect from a system using proxy.

[Al Bailey (albailey1974)]

This error in "helm", "init", "--client-only"
is added by this commit:
https://opendev.org/starlingx/ansible-playbooks/commit/2b049882fb796280b11468ecd98adc17b40861cf

That command may need to be re-examined if it cannot work in an environment that is unable to access an outside network.

[Ghada Khalil (gkhalil)]

stx.3.0 gating / critical as it blocks sanity. Issue was introduced by recent work related to upversioning K8s

[Al Bailey (albailey1974)]

We have a workaround for this issue in the build system
https://opendev.org/starlingx/openstack-armada-app/src/branch/master/openstack-helm/centos/openstack-helm.spec#L77

I will see if the same can be done in armada.

[Al Bailey (albailey1974)]

local testing shows that by adding "--skip-refresh" to the "helm init --client-only" command, this does not trigger an error in an air-gap system without networking access, such as what was being used by this sanity test.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/686729

[Al Bailey (albailey1974)]

Investigating if the tiller upgrade-prep changes would also encounter this problem.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/686748

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686729
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=9347e2263d11ad6ecf8ea21f1cf7597078b5ab04
Submitter: Zuul
Branch: master

commit 9347e2263d11ad6ecf8ea21f1cf7597078b5ab04
Author: Al Bailey <email address hidden>
Date: Fri Oct 4 09:05:28 2019 -0500

    Fix ansible bootstrap failure on systems without network access

    The helm init --client-only attempts to refresh its repo at:
    https://kubernetes-charts.storage.googleapis.com

    Passing the --skip-refresh argument will allow helm initialization
    to pass. This is needed for systems that do not have networking,
    or dns setup, and for air-gap installations.

    Change-Id: Ib5208985599cc51239be450a212a204b9d7ee120
    Closes-Bug: 1846540
    Signed-off-by: Al Bailey <email address hidden>

[Cristopher Lemus (cjlemusc)]

Hello Al,

Just to confirm. With BUILD_ID="20191006T230000Z", both, proxy and local (mirror) registry installations completed without any issues on all configs.

Thanks a lot for fixing it.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Al Bailey (<email address hidden>) on branch: master
Review: https://review.opendev.org/686748
Reason: Kristine's commit https://review.opendev.org/#/c/687683/ includes the skip-refresh so this change is no longer needed.