I took a look at the collect logs. The fix for this looks to be present as is https://review.opendev.org/#/c/689438/ which was later reverted due to causing some ceph related issues. I'm not sure if the follow is related to that change or some other like the k8s upgrade to 1.16.2.
This problem does appear not to be related to this LP change. All the application overrides look to be correct and are being applied based on the updated apply pre-reqs.
What I’m observing is the following pattern on application apply
- sysinv fires off the manifest apply
- tiller requests all the current releases
- an API log WARNING shows up in horizon.log for the configmap requests that tiller is looking for related to the releases.
- tiller fails to get the release configmaps and fails the apply.
Apply #1:
2019-10-22 12:30:40.297 111637 INFO sysinv.conductor.kube_app [-] Armada apply command = /bin/bash -c 'set -o pipefail; armada apply --enable-chart-cleanup --debug /manifests/platform-integ-apps/1.0-8/platform-integ-apps-manifest.yaml --values /overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml --values /overrides/platform-integ-apps/1.0-8/kube-system-ceph-pools-audit.yaml --values /overrides/platform-integ-apps/1.0-8/helm-toolkit-helm-toolkit.yaml --tiller-host tiller-deploy.kube-system.svc.cluster.local | tee /logs/platform-integ-apps-apply.log'
2019-10-22 12:30:41.120893971Z: [storage] 2019/10/22 12:30:41 listing all releases with filter
2019-10-22 12:30:41.836 [WARNING] django.request: Not Found: /api/v1/namespaces/kube-system/configmaps
2019-10-22 12:30:41.83790798Z: [storage/driver] 2019/10/22 12:30:41 list: failed to list: the server could not find the requested resource (get configmaps)
2019-10-22 12:30:42.215 111637 ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/platform-integ-apps/1.0-8/platform-integ-apps-manifest.yaml. See /var/log/armada/platform-integ-apps-apply.log for details.
2019-10-22 12:30:42.484 111637 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.
Apply #2:
2019-10-22 13:08:49.308 111637 INFO sysinv.conductor.kube_app [-] Armada apply command = /bin/bash -c 'set -o pipefail; armada apply --enable-chart-cleanup --debug /manifests/platform-integ-apps/1.0-8/platform-integ-apps-manifest.yaml --values /overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml --values /overrides/platform-integ-apps/1.0-8/kube-system-ceph-pools-audit.yaml --values /overrides/platform-integ-apps/1.0-8/helm-toolkit-helm-toolkit.yaml --tiller-host tiller-deploy.kube-system.svc.cluster.local | tee /logs/platform-integ-apps-apply.log'
2019-10-22 13:08:50.067400757Z: [storage] 2019/10/22 13:08:50 listing all releases with filter
2019-10-22 13:08:50.139 [WARNING] django.request: Not Found: /api/v1/namespaces/kube-system/configmaps
2019-10-22 13:08:50.140220136Z: [storage/driver] 2019/10/22 13:08:50 list: failed to list: the server could not find the requested resource (get configmaps)
2019-10-22 13:08:51.125 111637 ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/platform-integ-apps/1.0-8/platform-integ-apps-manifest.yaml. See /var/log/armada/platform-integ-apps-apply.log for details.
2019-10-22 13:08:51.362 111637 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.
Apply #3:
2019-10-22 13:13:53.446 111637 INFO sysinv.conductor.kube_app [-] Armada apply command = /bin/bash -c 'set -o pipefail; armada apply --enable-chart-cleanup --debug /manifests/platform-integ-apps/1.0-8/platform-integ-apps-manifest.yaml --values /overrides/platform-integ-apps/1.0-8/kube-system-rbd-provisioner.yaml --values /overrides/platform-integ-apps/1.0-8/kube-system-ceph-pools-audit.yaml --values /overrides/platform-integ-apps/1.0-8/helm-toolkit-helm-toolkit.yaml --tiller-host tiller-deploy.kube-system.svc.cluster.local | tee /logs/platform-integ-apps-apply.log'
2019-10-22 13:13:54.21153814Z: [storage] 2019/10/22 13:13:54 listing all releases with filter
2019-10-22 13:13:54.277 [WARNING] django.request: Not Found: /api/v1/namespaces/kube-system/configmaps
2019-10-22 13:13:54.279302854Z: [storage/driver] 2019/10/22 13:13:54 list: failed to list: the server could not find the requested resource (get configmaps)
2019-10-22 13:13:55.270 111637 ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/platform-integ-apps/1.0-8/platform-integ-apps-manifest.yaml. See /var/log/armada/platform-integ-apps-apply.log for details.
2019-10-22 13:13:55.441 111637 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.
I took a look at the collect logs. The fix for this looks to be present as is https:/ /review. opendev. org/#/c/ 689438/ which was later reverted due to causing some ceph related issues. I'm not sure if the follow is related to that change or some other like the k8s upgrade to 1.16.2.
This problem does appear not to be related to this LP change. All the application overrides look to be correct and are being applied based on the updated apply pre-reqs.
What I’m observing is the following pattern on application apply
- sysinv fires off the manifest apply
- tiller requests all the current releases
- an API log WARNING shows up in horizon.log for the configmap requests that tiller is looking for related to the releases.
- tiller fails to get the release configmaps and fails the apply.
Apply #1: conductor. kube_app [-] Armada apply command = /bin/bash -c 'set -o pipefail; armada apply --enable- chart-cleanup --debug /manifests/ platform- integ-apps/ 1.0-8/platform- integ-apps- manifest. yaml --values /overrides/ platform- integ-apps/ 1.0-8/kube- system- rbd-provisioner .yaml --values /overrides/ platform- integ-apps/ 1.0-8/kube- system- ceph-pools- audit.yaml --values /overrides/ platform- integ-apps/ 1.0-8/helm- toolkit- helm-toolkit. yaml --tiller-host tiller- deploy. kube-system. svc.cluster. local | tee /logs/platform- integ-apps- apply.log' 120893971Z: [storage] 2019/10/22 12:30:41 listing all releases with filter namespaces/ kube-system/ configmaps conductor. kube_app [-] Failed to apply application manifest /manifests/ platform- integ-apps/ 1.0-8/platform- integ-apps- manifest. yaml. See /var/log/ armada/ platform- integ-apps- apply.log for details. conductor. kube_app [-] Application apply aborted!.
2019-10-22 12:30:40.297 111637 INFO sysinv.
2019-10-22 12:30:41.
2019-10-22 12:30:41.836 [WARNING] django.request: Not Found: /api/v1/
2019-10-22 12:30:41.83790798Z: [storage/driver] 2019/10/22 12:30:41 list: failed to list: the server could not find the requested resource (get configmaps)
2019-10-22 12:30:42.215 111637 ERROR sysinv.
2019-10-22 12:30:42.484 111637 ERROR sysinv.
Apply #2: conductor. kube_app [-] Armada apply command = /bin/bash -c 'set -o pipefail; armada apply --enable- chart-cleanup --debug /manifests/ platform- integ-apps/ 1.0-8/platform- integ-apps- manifest. yaml --values /overrides/ platform- integ-apps/ 1.0-8/kube- system- rbd-provisioner .yaml --values /overrides/ platform- integ-apps/ 1.0-8/kube- system- ceph-pools- audit.yaml --values /overrides/ platform- integ-apps/ 1.0-8/helm- toolkit- helm-toolkit. yaml --tiller-host tiller- deploy. kube-system. svc.cluster. local | tee /logs/platform- integ-apps- apply.log' 067400757Z: [storage] 2019/10/22 13:08:50 listing all releases with filter namespaces/ kube-system/ configmaps 140220136Z: [storage/driver] 2019/10/22 13:08:50 list: failed to list: the server could not find the requested resource (get configmaps) conductor. kube_app [-] Failed to apply application manifest /manifests/ platform- integ-apps/ 1.0-8/platform- integ-apps- manifest. yaml. See /var/log/ armada/ platform- integ-apps- apply.log for details. conductor. kube_app [-] Application apply aborted!.
2019-10-22 13:08:49.308 111637 INFO sysinv.
2019-10-22 13:08:50.
2019-10-22 13:08:50.139 [WARNING] django.request: Not Found: /api/v1/
2019-10-22 13:08:50.
2019-10-22 13:08:51.125 111637 ERROR sysinv.
2019-10-22 13:08:51.362 111637 ERROR sysinv.
Apply #3: conductor. kube_app [-] Armada apply command = /bin/bash -c 'set -o pipefail; armada apply --enable- chart-cleanup --debug /manifests/ platform- integ-apps/ 1.0-8/platform- integ-apps- manifest. yaml --values /overrides/ platform- integ-apps/ 1.0-8/kube- system- rbd-provisioner .yaml --values /overrides/ platform- integ-apps/ 1.0-8/kube- system- ceph-pools- audit.yaml --values /overrides/ platform- integ-apps/ 1.0-8/helm- toolkit- helm-toolkit. yaml --tiller-host tiller- deploy. kube-system. svc.cluster. local | tee /logs/platform- integ-apps- apply.log' namespaces/ kube-system/ configmaps 279302854Z: [storage/driver] 2019/10/22 13:13:54 list: failed to list: the server could not find the requested resource (get configmaps) conductor. kube_app [-] Failed to apply application manifest /manifests/ platform- integ-apps/ 1.0-8/platform- integ-apps- manifest. yaml. See /var/log/ armada/ platform- integ-apps- apply.log for details. conductor. kube_app [-] Application apply aborted!.
2019-10-22 13:13:53.446 111637 INFO sysinv.
2019-10-22 13:13:54.21153814Z: [storage] 2019/10/22 13:13:54 listing all releases with filter
2019-10-22 13:13:54.277 [WARNING] django.request: Not Found: /api/v1/
2019-10-22 13:13:54.
2019-10-22 13:13:55.270 111637 ERROR sysinv.
2019-10-22 13:13:55.441 111637 ERROR sysinv.
Plus 7 additional applies with the same signature