Release osh-kube-system-ingress failed during stx-openstack apply

[Nicolae Jascanu]

Brief Description
-----------------
ON MONOLITHIC - Application stx-openstack apply failed with message: Release osh-kube-system-ingress failed

Severity
--------
Critical:

Steps to Reproduce
------------------
Following the steps for installing StarlingX it fails at Provision when applying stx-openstack

Expected Behavior
------------------
Apply stx-openstack shall pass

Actual Behavior
----------------
Apply stx-openstack shall fails

Reproducibility
---------------
Reproducible on Baremetal SIMPLEX, STANDARD, STANDARD EXTERNAL STORAGE

System Configuration
--------------------
Baremetal SIMPLEX, STANDARD, STANDARD EXTERNAL STORAGE

OS="centos"
SW_VERSION="20.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20200507T080012Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="533"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2020-05-07 08:00:12 +0000"

Branch/Pull Time/Commit
-----------------------
ISO=http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/monolithic/20200507T080012Z/outputs/iso/

HELM_CHART= http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/20200505T204039Z/outputs/helm-charts/

Timestamp/Logs
--------------
Please see attached:
 - sysinv.log
 - ansible.log
 - stx-openstack-apply_2020-05-08-03-58-38.log

Test Activity
-------------
Sanity

Workaround
----------

[Nicolae Jascanu (njascanu-intel)]

SIMPLEX: The Sanity check on Layered build http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200508T061331Z/outputs/iso/

FAILED with the same error.

The log files are uploaded at: https://files.starlingx.kube.cengn.ca/launchpad/1877548

[Nicolae Jascanu (njascanu-intel)]

DUPLEX - I'm seeing the same error.

[Ghada Khalil (gkhalil)]

For reference, a similar issue was reported a few months ago (although the error logs are not exactly the same). See https://bugs.launchpad.net/starlingx/+bug/1827877

[Austin Sun (sunausti)]

Hi Nic:
This issue was already fix by [1], you are using monolithic build , to verify it , you can use monolithic build helm-chart[3], but not [2].
[1] https://review.opendev.org/#/c/724385/1
[2] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/containers/20200505T204039Z/outputs/helm-charts/stx-openstack-1.0-19-centos-stable-latest.tgz
[3] http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/monolithic/20200505T080012Z/outputs/tarballs/openstack-armada-app-v3.0.0.rc0-19.tgz

[Austin Sun (sunausti)]

just realize [1] was not a normal application format , need build team check where can get monolithic helm chart.

[1]http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/monolithic/20200505T080012Z/outputs/tarballs/openstack-armada-app-v3.0.0.rc0-19.tgz

[Frank Miller (sensfan22)]

I have confirmed this is an issue with the incorrect helm chart being used by the build system. Austin's analysis is correct that this commit is not being included in the generated helm-charts:
https://review.opendev.org/#/c/724385

Looking at the stx-openstack.yaml in the helm-charts-stx-openstack-centos-stable-latest.tgz tarball we see:
data:
  chart_name: ingress
  dependencies:
  - helm-toolkit
  install:
    no_hooks: false
  namespace: kube-system
  release: kube-system-ingress

But Sabeel's commit changed this to release: openstack-ingress

The reason for the build output to not include Sabeel's change is unclear at this time. The theory is that the layered build system builds this package but because the version did not change it is not able to properly distinguish a previously built package with the same version from the newly built package. Scott will need to confirm this theory. As a workaround the package is being upversioned to see if this addresses the issue and results in the right helm chart package in the build output directory. Even if this works a more robust solution will be required in the build system.

On a side note, now that we have cutover to the build layering method, the monolithic method still runs nightly but does not generate docker images or helm charts. As such when using a monolithic build the helm charts picked up by sanity are from the build layer method - this is not correct and needs to be addressed separately.

[Ghada Khalil (gkhalil)]

An additional commit just merged to bump up the version# for the openstack application:
https://review.opendev.org/#/c/726909/

Based on the explanation above, this should trigger the building of the helm charts properly.

[Ghada Khalil (gkhalil)]

As per emails on the stx-discuss mailing list, there was another issue with the CENGN build as well.

The CENGN flock build was skipping the stx-openstack helm charts due to a script issue. This has been corrected by Don Penney.

The charts are available at:
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/latest_build/outputs/helm-charts/

This was addressed in the 2020-05-12 CENGN build.

[Ghada Khalil (gkhalil)]

A green sanity was confirmed by Nicolae Jascanu

http://lists.starlingx.io/pipermail/starlingx-discuss/2020-May/008603.html