This looks like another variation of https://bugs.launchpad.net/starlingx/+bug/1877582. It’s not quite the same failure, but is basically an issue with the proper shutdown of the armada container on an unlock, leaves the container unable to be started/restarted after the reboot. Thu May 21 13:50:59 UTC 2020 : : docker container ps -a -------------------------------------------------------------------- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ae7089822ac6 registry.local:9001/quay.io/airshipit/armada:8a1638098f88d92bf799ef4934abe569789b885e-ubuntu_bionic "./entrypoint.sh ser…" 5 hours ago Exited (127) 5 hours ago armada_service So exited approx around May 21 08:50:59... 2020-05-21T08:59:57.143 + Host Info +--------------------------------------+ 2020-05-21T08:59:57.143 | action : unlock 2020-05-21T08:59:57.143 | personality: controller 2020-05-21T08:59:57.143 | hostname : controller-0 2020-05-21T08:59:57.143 | task : none 2020-05-21T08:59:57.143 | info : none 2020-05-21T08:59:57.143 | ip : face::2 2020-05-21T08:59:57.143 | mac : 3c:fd:fe:25:d5:c0 2020-05-21T08:59:57.143 | uuid : b726aaf0-96aa-42d3-a668-a22e360f9691 2020-05-21T08:59:57.143 | adminState: locked 2020-05-21T08:59:57.143 | operState: disabled 2020-05-21T08:59:57.143 | availStatus: online 2020-05-21T08:59:57.143 | bm ip : none 2020-05-21T08:59:57.143 | bm un : none 2020-05-21T08:59:57.143 | bm type : none 2020-05-21T08:59:57.143 | subFunction: controller,worker 2020-05-21T08:59:57.143 | operState: disabled 2020-05-21T08:59:57.143 | availStatus: online 2020-05-21T08:59:57.143 +------------+--------------------------------------+ 2020-05-21T09:00:30.637 localhost containerd[108819]: info time="2020-05-21T09:00:30.636987457Z" level=info msg="shim reaped" id=ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 2020-05-21T09:00:30.646 localhost dockerd[108828]: info time="2020-05-21T09:00:30.645241723Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete" 2020-05-21T09:00:30.663 localhost systemd[1]: info Unmounted /var/lib/docker/containers/ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6/mounts/shm. 2020-05-21T09:04:20.331 controller-0 containerd[2121]: info time="2020-05-21T09:04:20.331574760Z" level=info msg="shim reaped" id=ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 2020-05-21T09:04:20.341 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.341878502Z" level=error msg="stream copy error: reading from a closed fifo" 2020-05-21T09:04:20.341 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.341873475Z" level=error msg="stream copy error: reading from a closed fifo" 2020-05-21T09:04:20.367 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.367669129Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container" 2020-05-21T09:04:20.367 controller-0 dockerd[2130]: info time="2020-05-21T09:04:20.367704982Z" level=error msg="Failed to start container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: OCI runtime create failed: container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"/opt/platform/armada/20.01/admin.conf\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay2/d890f67cbf0124877243d9bee6a0df56f469219171917dc61c7d0a17326a1669/merged\\\\\\\" at \\\\\\\"/var/lib/docker/overlay2/d890f67cbf0124877243d9bee6a0df56f469219171917dc61c7d0a17326a1669/merged/armada/.kube/config\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type" 2020-05-21T09:09:03.829 controller-0 dockerd[90085]: info time="2020-05-21T09:09:03.829615064Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container" 2020-05-21T09:09:03.829 controller-0 dockerd[90085]: info time="2020-05-21T09:09:03.829637948Z" level=error msg="Failed to start container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: transport is closing: unavailable" 2020-05-21T09:09:03.829 controller-0 dockerd[90085]: info time="2020-05-21T09:09:03.829615064Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container" 2020-05-21T09:09:03.829 controller-0 dockerd[90085]: info time="2020-05-21T09:09:03.829637948Z" level=error msg="Failed to start container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: transport is closing: unavailable" 2020-05-21T09:09:04.464 controller-0 dockerd[91243]: info time="2020-05-21T09:09:04.464625905Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container" 2020-05-21T09:09:04.464 controller-0 dockerd[91243]: info time="2020-05-21T09:09:04.464656844Z" level=error msg="Failed to start container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: task ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 already exists: unknown" 2020-05-21 09:12:28.904 104305 INFO sysinv.conductor.kube_app [-] Application platform-integ-apps (1.0-8) upload started. 2020-05-21 09:12:29.021 104305 INFO sysinv.conductor.kube_app [-] Restarting Armada service... 2020-05-21T09:12:29.052 controller-0 dockerd[91243]: info time="2020-05-21T09:12:29.052585445Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container" 2020-05-21T09:12:29.125 controller-0 dockerd[91243]: info time="2020-05-21T09:12:29.125565034Z" level=error msg="Handler for POST /v1.35/containers/ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6/restart returned error: Cannot restart container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: task ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 already exists: unknown" 2020-05-21 09:12:29.126 104305 INFO sysinv.conductor.kube_app [-] Starting Armada service... 2020-05-21 09:12:29.127 104305 INFO sysinv.conductor.kube_app [-] kube_config=/opt/platform/armada/20.01/admin.conf, manifests_dir=/opt/platform/armada/20.01, overrides_dir=/opt/platform/helm/20.01, logs_dir=/var/log/armada. 2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app [-] Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest.: KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest. 2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1928, in perform_app_upload 2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app reason="Failed to validate application manifest.") 2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest. 2020-05-21 09:12:29.137 104305 ERROR sysinv.conductor.kube_app 2020-05-21 09:12:29.298 104305 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.: KubeAppUploadFailure: Upload of application platform-integ-apps (1.0-8) failed: Failed to validate application manifest. 2020-05-21 09:12:29.436 104305 INFO sysinv.conductor.manager [-] Platform managed application oidc-auth-apps: Uploading... 2020-05-21 09:12:29.790 104305 INFO sysinv.conductor.kube_app [-] Application oidc-auth-apps (1.0-0) upload started. 2020-05-21 09:12:29.870 104305 INFO sysinv.conductor.kube_app [-] Restarting Armada service... 2020-05-21T09:12:29.902 controller-0 dockerd[91243]: info time="2020-05-21T09:12:29.902655676Z" level=error msg="ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 cleanup: failed to delete container from containerd: no such container" 2020-05-21T09:12:29.959 controller-0 dockerd[91243]: info time="2020-05-21T09:12:29.959898459Z" level=error msg="Handler for POST /v1.35/containers/ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6/restart returned error: Cannot restart container ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6: task ae7089822ac60e810d9f348d57a31f03a64bf2c6a8932da82721d4e260f363e6 already exists: unknown" 2020-05-21 09:12:29.962 104305 INFO sysinv.conductor.kube_app [-] Starting Armada service... 2020-05-21 09:12:29.963 104305 INFO sysinv.conductor.kube_app [-] kube_config=/opt/platform/armada/20.01/admin.conf, manifests_dir=/opt/platform/armada/20.01, overrides_dir=/opt/platform/helm/20.01, logs_dir=/var/log/armada. 2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app [-] Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.: KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest. 2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app Traceback (most recent call last): 2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1928, in perform_app_upload 2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app reason="Failed to validate application manifest.") 2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest. 2020-05-21 09:12:29.978 104305 ERROR sysinv.conductor.kube_app 2020-05-21 09:12:30.163 104305 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.: KubeAppUploadFailure: Upload of application oidc-auth-apps (1.0-0) failed: Failed to validate application manifest.