Prior to running any additional user applications (including the stx-openstack
application), you will want to make sure that the platform application has been
applied [4] to ensure that persistent volume claims will be serviced. Other than
this check, no other additional changes are required from an automation
perspective to launch the stx-openstack application.
2) The stx-openstack apply aborts because the secret created by the platform-integ-apps didn't occur yet
- Update for #1 will avoid this
3) platform-integ-apps overrides are being overwritten for the helm toolkit when the stx-openstack upload/apply occurs. This seems to cause the abort of platform-integ-apps
- We need to land https://review.opendev.org/#/c/660498/ to isolate the app overrides. The difference is shown here as the toolkit is present in both helm repos as they are required by both apps
# Audit task triggers creation/upload of platform-integ-apps
2019-05-21 07:13:20.082 99992 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Creating...
2019-05-21 07:13:21.428 99992 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Uploading...
2019-05-21 07:13:21.430 99992 INFO sysinv.conductor.kube_app [-] Application (platform-integ-apps) upload started.
2019-05-21 07:13:23.633 99992 INFO sysinv.conductor.kube_app [-] Manifest file /manifests/platform-integ-apps-manifest.yaml was successfully validated.
2019-05-21 07:13:24.178 99992 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps will load charts to chart repo stx-platform
2019-05-21 07:13:27.362 99992 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-05-21 07:13:28.054 99992 INFO sysinv.conductor.kube_app [-] Application (platform-integ-apps) upload completed.
# Audit task triggers application apply of platform-integ-apps
2019-05-21 07:14:21.632 99992 INFO sysinv.conductor.manager [-] Platform managed application platform-integ-apps: Applying...
2019-05-21 07:14:21.634 99992 INFO sysinv.conductor.kube_app [-] Application (platform-integ-apps) apply started.
2019-05-21 07:14:21.861 99992 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-05-21 07:14:22.017 99992 INFO sysinv.conductor.kube_app [-] Application overrides generated.
# stx-openstack is uploaded
2019-05-21 07:15:46.083 99992 INFO sysinv.conductor.kube_app [-] Application (stx-openstack) upload started.
2019-05-21 07:16:01.119 99992 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-05-21 07:16:10.900 99992 INFO sysinv.conductor.kube_app [-] Application (stx-openstack) upload completed.
# platform-integ-apps docker images are downloaded and the Armada apply starts
2019-05-21 07:16:13.471 99992 INFO sysinv.conductor.kube_app [-] All docker images for application platform-integ-apps were successfully downloaded in 111 seconds
2019-05-21 07:16:13.480 99992 INFO sysinv.conductor.kube_app [-] Armada apply command = /bin/bash -c 'armada apply --debug /manifests/platform-integ-apps-manifest.yaml --values /overrides/helm-toolkit-helm-toolkit.yaml --values /overrides/kube-system-rbd-provisioner.yaml --values /overrides/kube-system-ceph-pools-audit.yaml | tee platform-integ-apps-apply.log'
# stx-openstack apply starts prior to the completion of platform-integ-apps
2019-05-21 07:16:14.305 99992 INFO sysinv.conductor.kube_app [-] Application (stx-openstack) apply started.
2019-05-21 07:16:14.839 99992 INFO sysinv.conductor.kube_app [-] Starting progress monitoring thread for app platform-integ-apps
# The secret, created by the successful apply of the platform-integ-apps is not available (yet) so the copy fails and the stx-openstack apply is aborted
2019-05-21 07:16:15.288 99992 ERROR sysinv.common.kubernetes [req-e227bf77-d019-49ee-a8cf-184efa10e7d5 admin admin] Failed to copy Secret ceph-pool-kube-rbd from Namespace kube-system to Namespace openstack: (404)
2019-05-21 07:16:15.310 99992 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.
# The platform-integ-apps apply aborts complaining about an invalid helm-toolkit-helm-toolkit.yaml. Possibly related to stx-openstack apply changing the contents of this file that was created by the platform-integ-apps apply
2019-05-21 07:16:15.374 52 ERROR armada.cli [-] Caught internal exception: armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/helm-toolkit-helm-toolkit.yaml is not a valid override file.
2019-05-21 07:16:15.374 52 ERROR armada.cli PermissionError: [Errno 13] Permission denied: '/overrides/helm-toolkit-helm-toolkit.yaml'
2019-05-21 07:16:15.374 52 ERROR armada.cli armada.exceptions.override_exceptions.InvalidOverrideFileException: /overrides/helm-toolkit-helm-toolkit.yaml is not a valid override file.
2019-05-21 07:16:15.488 99992 ERROR sysinv.conductor.kube_app [-] Received a false positive response from Docker/Armada. Failed to apply application manifest /manifests/platform-integ-apps-manifest.yaml: 2019-05-21 07:16:14.278 52 DEBUG armada.handlers.document [-] Resolving reference /manifests/platform-integ-apps-manifest.yaml. resolve_reference /usr/local/lib/python3.6/dist-packages/armada/handlers/document.py:49[00m
2019-05-21 07:16:15.489 99992 INFO sysinv.conductor.kube_app [-] Exiting progress monitoring thread for app platform-integ-apps
2019-05-21 07:16:15.494 99992 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.
Issues observed here:
1) The stx-openstack app is being applied prior to the completion of the platform-integ-apps lists.starlingx .io/pipermail/ starlingx- discuss/ 2019-May/ 004447. html
- Per: http://
Prior to running any additional user applications (including the stx-openstack
application), you will want to make sure that the platform application has been
applied [4] to ensure that persistent volume claims will be serviced. Other than
this check, no other additional changes are required from an automation
perspective to launch the stx-openstack application.
- https:/ /opendev. org/starlingx/ config/ commit/ 4758cdfbd864826 d46e6e06571d406 93dd040b14 will prevent this apply if attempted too soon
2) The stx-openstack apply aborts because the secret created by the platform-integ-apps didn't occur yet
- Update for #1 will avoid this
3) platform-integ-apps overrides are being overwritten for the helm toolkit when the stx-openstack upload/apply occurs. This seems to cause the abort of platform-integ-apps /review. opendev. org/#/c/ 660498/ to isolate the app overrides. The difference is shown here as the toolkit is present in both helm repos as they are required by both apps
- We need to land https:/
[wrsroot@ controller- 0 19.05(keystone_ admin)] $ diff helm-toolkit- helm-toolkit. yaml ~/openstack- save/helm- toolkit- helm-toolkit. yaml controller: 8080/helm_ charts/ stx-platform/ helm-toolkit- 0.1.0.tgz controller: 8080/helm_ charts/ starlingx/ helm-toolkit- 0.1.0.tgz
3c3
< location: http://
---
> location: http://
In summary, we have a sequencing issue here which can no longer happen based on the inter_app dependency code that I added in https:/ /opendev. org/starlingx/ config/ commit/ 4758cdfbd864826 d46e6e06571d406 93dd040b14
Timeline: ------- ------- ------- ------- ------- ---
-------
# Ceph client is accessable
2019-05-21 07:13:06.530 98217 INFO ceph_client [-] Request params: url=https:/ /controller- 0:5001/ request? wait=1, json={'prefix': 'fsid', 'format': 'text'} c644-4855- b5f9-eaeb0328ff 73\n', u'outs': u'', u'command': u'fsid format=text'}], u'is_finished': True, u'id': u'140310473308432'}
2019-05-21 07:13:06.546 98217 INFO ceph_client [-] Result: {u'waiting': [], u'has_failed': False, u'state': u'success', u'is_waiting': False, u'running': [], u'failed': [], u'finished': [{u'outb': u'326ed215-
# Audit task triggers creation/upload of platform-integ-apps
2019-05-21 07:13:20.082 99992 INFO sysinv. conductor. manager [-] Platform managed application platform- integ-apps: Creating... conductor. manager [-] Platform managed application platform- integ-apps: Uploading... conductor. kube_app [-] Application (platform- integ-apps) upload started. conductor. kube_app [-] Manifest file /manifests/ platform- integ-apps- manifest. yaml was successfully validated. conductor. kube_app [-] Application platform-integ-apps will load charts to chart repo stx-platform conductor. kube_app [-] Generating application overrides... conductor. kube_app [-] Application (platform- integ-apps) upload completed.
2019-05-21 07:13:21.428 99992 INFO sysinv.
2019-05-21 07:13:21.430 99992 INFO sysinv.
2019-05-21 07:13:23.633 99992 INFO sysinv.
2019-05-21 07:13:24.178 99992 INFO sysinv.
2019-05-21 07:13:27.362 99992 INFO sysinv.
2019-05-21 07:13:28.054 99992 INFO sysinv.
# Audit task triggers application apply of platform-integ-apps
2019-05-21 07:14:21.632 99992 INFO sysinv. conductor. manager [-] Platform managed application platform- integ-apps: Applying... conductor. kube_app [-] Application (platform- integ-apps) apply started. conductor. kube_app [-] Generating application overrides... conductor. kube_app [-] Application overrides generated.
2019-05-21 07:14:21.634 99992 INFO sysinv.
2019-05-21 07:14:21.861 99992 INFO sysinv.
2019-05-21 07:14:22.017 99992 INFO sysinv.
# stx-openstack is uploaded
2019-05-21 07:15:46.083 99992 INFO sysinv. conductor. kube_app [-] Application (stx-openstack) upload started. conductor. kube_app [-] Generating application overrides... conductor. kube_app [-] Application (stx-openstack) upload completed.
2019-05-21 07:16:01.119 99992 INFO sysinv.
2019-05-21 07:16:10.900 99992 INFO sysinv.
# platform-integ-apps docker images are downloaded and the Armada apply starts
2019-05-21 07:16:13.471 99992 INFO sysinv. conductor. kube_app [-] All docker images for application platform-integ-apps were successfully downloaded in 111 seconds conductor. kube_app [-] Armada apply command = /bin/bash -c 'armada apply --debug /manifests/ platform- integ-apps- manifest. yaml --values /overrides/ helm-toolkit- helm-toolkit. yaml --values /overrides/ kube-system- rbd-provisioner .yaml --values /overrides/ kube-system- ceph-pools- audit.yaml | tee platform- integ-apps- apply.log'
2019-05-21 07:16:13.480 99992 INFO sysinv.
# stx-openstack apply starts prior to the completion of platform-integ-apps
2019-05-21 07:16:14.305 99992 INFO sysinv. conductor. kube_app [-] Application (stx-openstack) apply started. conductor. kube_app [-] Starting progress monitoring thread for app platform-integ-apps
2019-05-21 07:16:14.839 99992 INFO sysinv.
# The secret, created by the successful apply of the platform-integ-apps is not available (yet) so the copy fails and the stx-openstack apply is aborted
2019-05-21 07:16:15.288 99992 ERROR sysinv. common. kubernetes [req-e227bf77- d019-49ee- a8cf-184efa10e7 d5 admin admin] Failed to copy Secret ceph-pool-kube-rbd from Namespace kube-system to Namespace openstack: (404) conductor. kube_app [-] Application apply aborted!.
2019-05-21 07:16:15.310 99992 ERROR sysinv.
# The platform-integ-apps apply aborts complaining about an invalid helm-toolkit- helm-toolkit. yaml. Possibly related to stx-openstack apply changing the contents of this file that was created by the platform-integ-apps apply
2019-05-21 07:16:15.374 52 ERROR armada.cli [-] Caught internal exception: armada. exceptions. override_ exceptions. InvalidOverride FileException: /overrides/ helm-toolkit- helm-toolkit. yaml is not a valid override file. helm-toolkit- helm-toolkit. yaml' exceptions. override_ exceptions. InvalidOverride FileException: /overrides/ helm-toolkit- helm-toolkit. yaml is not a valid override file. conductor. kube_app [-] Received a false positive response from Docker/Armada. Failed to apply application manifest /manifests/ platform- integ-apps- manifest. yaml: 2019-05-21 07:16:14.278 52 DEBUG armada. handlers. document [-] Resolving reference /manifests/ platform- integ-apps- manifest. yaml. resolve_reference /usr/local/ lib/python3. 6/dist- packages/ armada/ handlers/ document. py:49[00m conductor. kube_app [-] Exiting progress monitoring thread for app platform-integ-apps conductor. kube_app [-] Application apply aborted!.
2019-05-21 07:16:15.374 52 ERROR armada.cli PermissionError: [Errno 13] Permission denied: '/overrides/
2019-05-21 07:16:15.374 52 ERROR armada.cli armada.
2019-05-21 07:16:15.488 99992 ERROR sysinv.
2019-05-21 07:16:15.489 99992 INFO sysinv.
2019-05-21 07:16:15.494 99992 ERROR sysinv.