Brief Description
-----------------
Application upload for oidc-auth-app and platform-integ-apps failed after fresh install.
Severity
--------
Major
Steps to Reproduce
------------------
After fresh install, upload oidc-auth-app and platform-integ-apps
Expected Behavior
------------------
Application upload to work
Actual Behavior
----------------
Application upload failed.
Reproducibility
---------------
Seen once
System Configuration
--------------------
one node system
Branch/Pull Time/Commit
-----------------------
2020-05-07
Last Pass
---------
2020-05-06
Timestamp/Logs
--------------
2020-05-08 11:59:29.061
sysinv 2020-05-08 11:59:29.061 99806 ERROR sysinv.conductor.kube_app [-] Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.: KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.
2020-05-08 11:59:29.061 99806 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2020-05-08 11:59:29.061 99806 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1928, in perform_app_upload
2020-05-08 11:59:29.061 99806 ERROR sysinv.conductor.kube_app reason="Failed to validate application manifest.")
2020-05-08 11:59:29.061 99806 ERROR sysinv.conductor.kube_app KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.
2020-05-08 11:59:29.061 99806 ERROR sysinv.conductor.kube_app
sysinv 2020-05-08 11:59:29.223 99806 INFO sysinv.api.controllers.v1.rest_api [-] Response={u'pd': {}}
sysinv 2020-05-08 11:59:29.224 99806 INFO sysinv.conductor.manager [-] Platform managed application oidc-auth-apps: Creating...
sysinv 2020-05-08 11:59:29.226 99806 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.: KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.
sysinv 2020-05-08 11:59:29.277 99806 INFO sysinv.common.utils [-] Checksum file is included and validated.
sysinv 2020-05-08 11:59:29.279 99806 INFO sysinv.api.controllers.v1.kube_app [-] No patch required for application oidc-auth-apps (1.0-0).
sysinv 2020-05-08 11:59:29.349 99806 INFO sysinv.conductor.manager [-] Platform managed application oidc-auth-apps: Uploading...
sysinv 2020-05-08 11:59:29.720 99806 INFO sysinv.conductor.kube_app [-] Application oidc-auth-apps (1.0-0) upload started.
sysinv 2020-05-08 11:59:29.776 99806 INFO sysinv.conductor.manager [-] Setting config target of host 'controller-0' to '0972a620-0ea7-48ff-8287-f40d2db54298'.
sysinv 2020-05-08 11:59:29.803 99806 WARNING sysinv.conductor.manager [-] controller-0: iconfig out of date: target 0972a620-0ea7-48ff-8287-f40d2db54298, applied c3b3baa2-ba7c-4fb4-8088-456d2f3ba5fc
sysinv 2020-05-08 11:59:29.804 99806 WARNING sysinv.conductor.manager [-] SYS_I Raise system config alarm: host controller-0 config applied: c3b3baa2-ba7c-4fb4-8088-456d2f3ba5fc vs. target: 0972a620-0ea7-48ff-8287-f40d2db54298.
sysinv 2020-05-08 11:59:29.812 99806 INFO sysinv.conductor.kube_app [-] Restarting Armada service...
sysinv 2020-05-08 11:59:29.826 99806 INFO sysinv.conductor.manager [-] _config_update_hosts config_uuid=0972a620-0ea7-48ff-8287-f40d2db54298
sysinv 2020-05-08 11:59:29.827 99806 INFO sysinv.conductor.manager [-] applying runtime manifest config_uuid=0972a620-0ea7-48ff-8287-f40d2db54298, classes: ['openstack::keystone::endpoint::runtime', 'platform::firewall::runtime']
sysinv 2020-05-08 11:59:29.842 99806 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: 0972a620-0ea7-48ff-8287-f40d2db54298
sysinv 2020-05-08 11:59:29.954 99806 INFO sysinv.conductor.kube_app [-] Starting Armada service...
sysinv 2020-05-08 11:59:29.954 99806 INFO sysinv.conductor.kube_app [-] kube_config=/opt/platform/armada/20.04/admin.conf, manifests_dir=/opt/platform/armada/20.04, overrides_dir=/opt/platform/helm/20.04, logs_dir=/var/log/armada.
sysinv 2020-05-08 11:59:29.975 99806 ERROR sysinv.conductor.kube_app [-] Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.: KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.
2020-05-08 11:59:29.975 99806 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2020-05-08 11:59:29.975 99806 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1928, in perform_app_upload
2020-05-08 11:59:29.975 99806 ERROR sysinv.conductor.kube_app reason="Failed to validate application manifest.")
2020-05-08 11:59:29.975 99806 ERROR sysinv.conductor.kube_app KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.
2020-05-08 11:59:29.975 99806 ERROR sysinv.conductor.kube_app
sysinv 2020-05-08 11:59:30.153 99806 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.: KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.
Test Activity
-------------
Regression Testing
System was unlocked. Platform managed application try to upload. The armada container attempts to start but fails. It is in an exited status but appears to be running after the reboot and we have a port conflict: bind: address already in use: unknown" auto-apply integ-apps; system application-delete oidc-auth-apps
The workaround for this is to run the following, then the apps will auto-upload/
- docker rm 4435db3ceb9f
- system application-delete platform-
$ sudo docker ps -a local:9001/ quay.io/ airshipit/ armada: 8a1638098f88d92 bf799ef4934abe5 69789b885e- ubuntu_ bionic "./entrypoint.sh ser…" 3 hours ago Exited (128) 3 hours ago armada_service
Password:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4435db3ceb9f registry.
root 88294 1 0 11:55 ? 00:00:00 containerd-shim -namespace moby -workdir /var/lib/ docker/ io.containerd. runtime. v1.linux/ moby/4435db3ceb 9f5e72676dfb60c a6285751a69f530 6dad457a64c6c23 a412eac64 -address /var/run/ containerd/ containerd. sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/ docker/ runtime- runc
2020-05- 08T11:49: 20.231 + Host Info +------ ------- ------- ------- ------- ----+ 08T11:49: 20.231 | action : unlock 08T11:49: 20.231 | personality: controller 08T11:49: 20.231 | hostname : controller-0 08T11:49: 20.231 | task : none 08T11:49: 20.231 | info : none 08T11:49: 20.231 | ip : fd01:14::3 08T11:49: 20.231 | mac : 48:df:37:22:c5:f0 08T11:49: 20.231 | uuid : 25fed003- 05ff-44c6- ac1e-88d67a8cc8 08 08T11:49: 20.232 | adminState: locked 08T11:49: 20.232 | operState: disabled 08T11:49: 20.232 | availStatus: online 08T11:49: 20.232 | bm ip : none 08T11:49: 20.232 | bm un : none 08T11:49: 20.232 | bm type : none 08T11:49: 20.232 | subFunction: controller,worker 08T11:49: 20.232 | operState: disabled 08T11:49: 20.232 | availStatus: online 08T11:49: 20.232 +------ ------+ ------- ------- ------- ------- ------- ---+
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05-
2020-05- 08T11:49: 54.404 subcloud7 containerd[116275]: info time="2020- 05-08T11: 49:54.404297821 Z" level=info msg="shim reaped" id=4435db3ceb9f 5e72676dfb60ca6 285751a69f5306d ad457a64c6c23a4 12eac64 08T11:49: 54.433 subcloud7 systemd[1]: info Unmounted /var/lib/ docker/ containers/ 4435db3ceb9f5e7 2676dfb60ca6285 751a69f5306dad4 57a64c6c23a412e ac64/mounts/ shm.
2020-05-
reboot system boot 3.10.0-1127.el7. Fri May 8 11:52 - 14:44 (02:51)
2020-05- 08T11:55: 55.709 controller-0 containerd[1962]: info time="2020- 05-08T11: 55:55.709515678 Z" level=info msg="shim reaped" id=4435db3ceb9f 5e72676dfb60ca6 285751a69f5306d ad457a64c6c23a4 12eac64 08T11:55: 56.452 controller-0 dockerd[88042]: info time="2020- 05-08T11: 55:56.451978430 Z" level=error msg="4435db3ceb 9f5e72676dfb60c a6285751a69f530 6dad457a64c6c23 a412eac64 cleanup: failed to delete container from containerd: no such container" 08T11:55: 56.452 controller-0...
2020-05-
2020-05-