StarlingX

Applications upload aborted by failed to validate application manifest

Bug #1879970 reported by Peng Peng on 2020-05-21

This bug report is a duplicate of: Bug #1877582: Application upload failed oidc-auth and platform-integ-apps. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	Medium	Paul-Ionut Vaduva

Bug Description

Brief Description
-----------------
During system initializing, both platform-integ-apps and oidc-auth-apps applications upload aborted. The sysinv.log shows "Failed to validate application manifest".

Severity
--------
Major

Steps to Reproduce
------------------
Upload application platform-integ-apps and oidc-auth-apps

Expected Behavior
------------------
Both application platform-integ-apps and oidc-auth-apps upload success

Actual Behavior
----------------
Both application platform-integ-apps and oidc-auth-apps upload abort.

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
Two node system

Lab-name: R430_3-4

Branch/Pull Time/Commit
-----------------------
2020-05-20_21-00-00

Last Pass
---------
2020-05-19_20-00-00

From sysinv.log:
sysinv 2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app [-] Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.: KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.

sysinv 2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app [-] Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.: KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.

Test Activity
-------------
installation

Tags:

Revision history for this message

Peng Peng (ppeng) wrote on 2020-05-21:

Log added.
https://files.starlingx.kube.cengn.ca/launchpad/1879970

Ghada Khalil (gkhalil) on 2020-06-03

tags:	added: stx.conta
tags:	added: stx.4.0 stx.containers removed: stx.conta
Changed in starlingx:
importance:	Undecided → Medium
status:	New → Triaged
assignee:	nobody → Paul-Ionut Vaduva (pvaduva)

Revision history for this message

Paul-Ionut Vaduva (pvaduva) wrote on 2020-06-09:

Download full text (6.5 KiB)

An assesment of what happened in the lab when platform-integ-apps and oidc-auth-apps failed to upload

At 09:12:29.137 platform-integ-apps failed to upload
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app Traceback (most recent call last): │
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1928, in perform_app_upload │
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app reason="Failed to validate application manifest.") │
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest. │
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app

At 09:12:29.978 oidc-auth-apps failed to upload
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app Traceback (most recent call last): │
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1928, in perform_app_upload │
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app reason="Failed to validate application manifest.") │
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest. │
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app

Around 2020-05-21T09:12:04
The cert-manager pod container started to throw this errors in a loop:

2020-05-21T09:12:04.317805242Z stderr F E0521 09:12:04.317715 1 dynamic_source.go:87] "msg"="Failed to generate initial serving certificate, retrying..." "er
ror"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input" "interval"=1000000000
2020-05-21T09:12:05.308809217Z stderr F I0521 09:12:05.308645 1 dynamic_source.go:171] "msg"="Generating new ECDSA private key"
2020-05-21T09:12:05.315177628Z stderr F I0521 09:12:05.315036 1 dynamic_source.go:186] "msg"="Signing new serving certificate"

An assesment of what happened in the lab when platform-integ-apps and oidc-auth-apps failed to upload

At 09:12:29.137 platform-integ-apps failed  to upload 
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app Traceback (most recent call last):                                                                                                      │
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1928, in perform_app_upload                              │
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app     reason="Failed to validate application manifest.")                                                                                  │
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.                │
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app

At 09:12:29.978 oidc-auth-apps failed to upload 
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app Traceback (most recent call last):                                                                                                      │
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1928, in perform_app_upload                              │
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app     reason="Failed to validate application manifest.")                                                                                  │
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.                     │
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app

Around 2020-05-21T09:12:04
The cert-manager pod container started to throw this errors in a loop:

2020-05-21T09:12:04.317805242Z stderr F E0521 09:12:04.317715       1 dynamic_source.go:87]  "msg"="Failed to generate initial serving certificate, retrying..." "er
ror"="failed verifying CA keypair: tls: failed to find any PEM data in certificate input"  "interval"=1000000000
2020-05-21T09:12:05.308809217Z stderr F I0521 09:12:05.308645       1 dynamic_source.go:171]  "msg"="Generating new ECDSA private key"
2020-05-21T09:12:05.315177628Z stderr F I0521 09:12:05.315036       1 dynamic_source.go:186]  "msg"="Signing new serving certificate"

Around 2020-05-21T09:12:09Z
Things started to go wrong with one of cert-manager application pod
2020-05-21T09:12:09Z cm-cert-manager-webhook-7d5c897795-tstjz   Pod     Readiness probe failed: HTTP probe failed with statuscode: 500  Unhealthy       Warning
2020-05-21T09:12:10Z calico-kube-controllers-5cd4695574-mtspd   Pod     Container image "registry.local:9001/quay.io/calico/kube-controllers:v3.12.0" already pres
ent on machine        Pulled  Normal
2020-05-21T09:12:10Z coredns-78d9fd7cb9-q5nxw   Pod     Readiness probe failed: HTTP probe failed with statuscode: 503  Unhealthy       Warning
2020-05-21T09:12:10Z calico-kube-controllers-5cd4695574-mtspd   Pod     Started container calico-kube-controllers       Started Normal
2020-05-21T09:12:10Z calico-kube-controllers-5cd4695574-mtspd   Pod     Created container calico-kube-controllers       Created Normal
2020-05-21T09:12:12Z calico-kube-controllers-5cd4695574-mtspd   Pod     Readiness probe failed: Failed to read status file status.json: open status.json: no such
file or directory
        Unhealthy       Warning
2020-05-21T09:12:21Z calico-kube-controllers-5cd4695574-mtspd   Pod     Back-off restarting failed container    BackOff Warning
2020-05-21T09:12:24Z kube-scheduler     Lease   controller-0_df95482f-1ac0-474a-a491-c503622f57d1 became leader LeaderElection  Normal
2020-05-21T09:12:24Z coredns-78d9fd7cb9-lsv8g   Pod     0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy exist
ing pods anti-affinity rules. FailedScheduling        Warning
2020-05-21T09:12:24Z kube-scheduler     Endpoints       controller-0_df95482f-1ac0-474a-a491-c503622f57d1 became leader LeaderElection  Normal
2020-05-21T09:12:32Z cm-cert-manager-cainjector-56b68989b5-8xrw6        Pod     Back-off restarting failed container    BackOff Warning
2020-05-21T09:12:33Z cert-manager-controller    ConfigMap       cm-cert-manager-7b8b94bf9f-v5cmx-external-cert-manager-controller became leader LeaderElection  No
rmal
2020-05-21T09:12:34Z platform-deployment-manager-0      Pod     Back-off restarting failed container    BackOff Warning

Some issues with disk capacity around 2020-05-21T09:38:34 (The times are ulterior to the application failure so my guess is that they can't be causal)

2020-05-21T09:38:34Z coredns-78d9fd7cb9-lsv8g   Pod     Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "07b16d
823331e1ac4b326b65c01737d4eb2ce258835c2a5d74b4899664d78fc6": Multus: [kube-system/coredns-78d9fd7cb9-lsv8g]: error adding container to network "chain": delegateAd
d: error invoking conflistAdd - "chain": conflistAdd: error in getting result from AddNetworkList: stat /var/lib/calico/nodename: no such file or directory: check
 that the calico/node container is running and has mounted /var/lib/calico/       FailedCreatePodSandBox  Warning
2020-05-21T09:38:37Z ic-nginx-ingress-controller-hbs8r  Pod     Successfully pulled image "registry.local:9001/quay.io/kubernetes-ingress-controller/nginx-ingress
-controller:0.23.0"   Pulled  Normal
2020-05-21T09:38:39Z ic-nginx-ingress-controller-hbs8r  Pod     Created container nginx-ingress-controller      Created Normal
2020-05-21T09:38:39Z ic-nginx-ingress-controller-hbs8r  Pod     Started container nginx-ingress-controller      Started Normal
2020-05-21T09:38:41Z ic-nginx-ingress-controller        ConfigMap       ConfigMap kube-system/ic-nginx-ingress-controller       CREATE  Normal
2020-05-21T09:38:48Z calico-node-dpfzj  Pod     Successfully pulled image "registry.local:9001/quay.io/calico/node:v3.12.0"     Pulled  Normal
2020-05-21T09:38:55Z controller-1       Node    Starting kubelet.       Starting        Normal
2020-05-21T09:38:55Z controller-1       Node    invalid capacity 0 on image filesystem  InvalidDiskCapacity     Warning
2020-05-21T09:38:55Z controller-1       Node    invalid capacity 0 on image filesystem  ImageGCFailed   Warning

My only conclusion until now is that cert-manager malfunction is causal to failing application upload.

Revision history for this message

Bob Church (rchurch) wrote on 2020-06-09:

Download full text (10.2 KiB)

Thu May 21 13:50:59 UTC 2020 : : docker container ps -a
--------------------------------------------------------------------
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ae7089822ac6 registry.local:9001/quay.io/airshipit/armada:8a1638098f88d92bf799ef4934abe569789b885e-ubuntu_bionic "./entrypoint.sh ser…" 5 hours ago Exited (127) 5 hours ago armada_service

So exited approx around May 21 08:50:59...

2020-05-21T08:59:57.143 + Host Info +--------------------------------------+
2020-05-21T08:59:57.143 | action : unlock
2020-05-21T08:59:57.143 | personality: controller
2020-05-21T08:59:57.143 | hostname : controller-0
2020-05-21T08:59:57.143 | task : none
2020-05-21T08:59:57.143 | info : none
2020-05-21T08:59:57.143 | ip : face::2
2020-05-21T08:59:57.143 | mac : 3c:fd:fe:25:d5:c0
2020-05-21T08:59:57.143 | uuid : b726aaf0-96aa-42d3-a668-a22e360f9691
2020-05-21T08:59:57.143 | adminState: locked
2020-05-21T08:59:57.143 | operState: disabled
2020-05-21T08:59:57.143 | availStatus: online
2020-05-21T08:59:57.143 | bm ip : none
2020-05-21T08:59:57.143 | bm un : none
2020-05-21T08:59:57.143 | bm type : none
2020-05-21T08:59:57.143 | subFunction: controller,worker
2020-05-21T08:59:57.143 | operState: disabled
2020-05-21T08:59:57.143 | availStatus: online
2020-05-21T08:59:57.143 +------------+--------------------------------------+

2020-05-21T09:00:30.637 localhost containerd[108819]: info time="2020-05-21T09:00:30.636987457Z" level=info msg="shim reaped" id=ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6
2020-05-21T09:00:30.646 localhost dockerd[108828]: info time="2020-05-21T09:00:30.645241723Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2020-05-21T09:00:30.663 localhost systemd[1]: info Unmounted /var/lib/docker/containers/ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6/mounts/shm.

2020-05-21T09:04:20.331 controller-0 containerd[2121]: info time="2020-05-21T09:04:20.331574760Z" level=info msg="shim reaped" id=ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6
2020-05-21T09:04:20.341 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.341878502Z" level=error msg="stream copy error: reading from a closed fifo"
2020-05-21T09:04:20.341 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.341873475Z" level=error msg="stream copy error: reading from a closed fifo"
2020-05-21T09:04:20.367 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.367669129Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container fr...

This looks like another variation of https://bugs.launchpad.net/starlingx/+bug/1877582. It’s not quite the same failure, but is basically an issue with the proper shutdown of the armada container on an unlock, leaves the container unable to be started/restarted after the reboot.
 
Thu May 21 13:50:59 UTC 2020 :  : docker container ps -a
--------------------------------------------------------------------
CONTAINER ID        IMAGE                                                                                                 COMMAND                  CREATED             STATUS                     PORTS               NAMES
ae7089822ac6        registry.local:9001/quay.io/airshipit/armada:8a1638098f88d92bf799ef4934abe569789b885e-ubuntu_bionic   "./entrypoint.sh ser…"   5 hours ago         Exited (127) 5 hours ago                       armada_service
 
So exited approx around May 21 08:50:59...
 
2020-05-21T08:59:57.143 + Host Info  +--------------------------------------+
2020-05-21T08:59:57.143 | action     : unlock
2020-05-21T08:59:57.143 | personality: controller
2020-05-21T08:59:57.143 | hostname   : controller-0
2020-05-21T08:59:57.143 | task       : none
2020-05-21T08:59:57.143 | info       : none
2020-05-21T08:59:57.143 | ip         : face::2
2020-05-21T08:59:57.143 | mac        : 3c:fd:fe:25:d5:c0
2020-05-21T08:59:57.143 | uuid       : b726aaf0-96aa-42d3-a668-a22e360f9691
2020-05-21T08:59:57.143 |  adminState: locked
2020-05-21T08:59:57.143 |   operState: disabled
2020-05-21T08:59:57.143 | availStatus: online
2020-05-21T08:59:57.143 | bm ip      : none
2020-05-21T08:59:57.143 | bm un      : none
2020-05-21T08:59:57.143 | bm type    : none
2020-05-21T08:59:57.143 | subFunction: controller,worker
2020-05-21T08:59:57.143 |   operState: disabled
2020-05-21T08:59:57.143 | availStatus: online
2020-05-21T08:59:57.143 +------------+--------------------------------------+
 
 
2020-05-21T09:00:30.637 localhost containerd[108819]: info time="2020-05-21T09:00:30.636987457Z" level=info msg="shim reaped" id=ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6
2020-05-21T09:00:30.646 localhost dockerd[108828]: info time="2020-05-21T09:00:30.645241723Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2020-05-21T09:00:30.663 localhost systemd[1]: info Unmounted /var/lib/docker/containers/ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6/mounts/shm.
 
2020-05-21T09:04:20.331 controller-0 containerd[2121]: info time="2020-05-21T09:04:20.331574760Z" level=info msg="shim reaped" id=ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6
2020-05-21T09:04:20.341 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.341878502Z" level=error msg="stream copy error: reading from a closed fifo"
2020-05-21T09:04:20.341 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.341873475Z" level=error msg="stream copy error: reading from a closed fifo"
2020-05-21T09:04:20.367 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.367669129Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container"
2020-05-21T09:04:20.367 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.367704982Z" level=error msg="Failed to start container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: OCI runtime create failed: container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"/opt/platform/armada/20.01/admin.conf\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay2/d890f67cbf0124877243d9bee6a0df56f469219171917dc61c7d0a17326a1669/merged\\\\\\\" at \\\\\\\"/var/lib/docker/overlay2/d890f67cbf0124877243d9bee6a0df56f469219171917dc61c7d0a17326a1669/merged/armada/.kube/config\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"
 
2020-05-21T09:09:03.829 controller-0 dockerd[90085]: info time="2020-05-21T09:09:03.829615064Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container"
2020-05-21T09:09:03.829 controller-0 dockerd[90085]: info time="2020-05-21T09:09:03.829637948Z" level=error msg="Failed to start container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: transport is closing: unavailable"
 
2020-05-21T09:09:03.829 controller-0 dockerd[90085]: info time="2020-05-21T09:09:03.829615064Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container"
2020-05-21T09:09:03.829 controller-0 dockerd[90085]: info time="2020-05-21T09:09:03.829637948Z" level=error msg="Failed to start container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: transport is closing: unavailable"
 
2020-05-21T09:09:04.464 controller-0 dockerd[91243]: info time="2020-05-21T09:09:04.464625905Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container"
2020-05-21T09:09:04.464 controller-0 dockerd[91243]: info time="2020-05-21T09:09:04.464656844Z" level=error msg="Failed to start container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: task ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 already exists: unknown"
 
 
2020-05-21 09:12:28.904 104305 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) upload started.
2020-05-21 09:12:29.021 104305 INFO sysinv.conductor.kube_app [-] Restarting Armada service...
2020-05-21T09:12:29.052 controller-0 dockerd[91243]: info time="2020-05-21T09:12:29.052585445Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container"
2020-05-21T09:12:29.125 controller-0 dockerd[91243]: info time="2020-05-21T09:12:29.125565034Z" level=error msg="Handler for POST /v1.35/containers/ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6/restart returned error: Cannot restart container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: task ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 already exists: unknown"
2020-05-21 09:12:29.126 104305 INFO sysinv.conductor.kube_app [-] Starting Armada service...
2020-05-21 09:12:29.127 104305 INFO sysinv.conductor.kube_app [-] kube_config=/opt/platform/armada/20.01/admin.conf, manifests_dir=/opt/platform/armada/20.01, overrides_dir=/opt/platform/helm/20.01, logs_dir=/var/log/armada.
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app [-] Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.: KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1928, in perform_app_upload
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app     reason="Failed to validate application manifest.")
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.
2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app
2020-05-21 09:12:29.298 104305 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.: KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.
 
2020-05-21 09:12:29.436 104305 INFO sysinv.conductor.manager [-] Platform managed application oidc-auth-apps: Uploading...
2020-05-21 09:12:29.790 104305 INFO sysinv.conductor.kube_app [-] Application oidc-auth-apps (1.0-0) upload started.
2020-05-21 09:12:29.870 104305 INFO sysinv.conductor.kube_app [-] Restarting Armada service...
2020-05-21T09:12:29.902 controller-0 dockerd[91243]: info time="2020-05-21T09:12:29.902655676Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container"
2020-05-21T09:12:29.959 controller-0 dockerd[91243]: info time="2020-05-21T09:12:29.959898459Z" level=error msg="Handler for POST /v1.35/containers/ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6/restart returned error: Cannot restart container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: task ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 already exists: unknown"
2020-05-21 09:12:29.962 104305 INFO sysinv.conductor.kube_app [-] Starting Armada service...
2020-05-21 09:12:29.963 104305 INFO sysinv.conductor.kube_app [-] kube_config=/opt/platform/armada/20.01/admin.conf, manifests_dir=/opt/platform/armada/20.01, overrides_dir=/opt/platform/helm/20.01, logs_dir=/var/log/armada.
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app [-] Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.: KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1928, in perform_app_upload
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app     reason="Failed to validate application manifest.")
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.
2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app
2020-05-21 09:12:30.163 104305 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.: KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.

Revision history for this message

Frank Miller (sensfan22) wrote on 2020-06-12:

The fix for LP 1877582 is expected to address this variant of issue as well.

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2020-06-19:

Duplicate LP has been addressed.
https://review.opendev.org/735374
Merged in stx master on 2020-06-15

Changed in starlingx:
status:	Triaged → Fix Released

Revision history for this message

Yang Liu (yliu12) wrote on 2020-06-24:

This issue has not been seen in recent sanity on same system (r430-3-4)

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1877582 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.