Comment 9 for bug 1874858

Reproduction:
- The issue (pods stuck with Unknown status after AIO-SX reboot) is relatively easy to reproduce on our DC-4 lab (AIO-DX System Controller with 10 AIO-SX subclouds).
- The best environment to reproduce includes:
  - Following apps applied: cert-manager, nginx-ingress-controller, oidc-auth-apps, stx-monitor.
  - 30 PV ‘small’ test pods running.
- Doing a lock/unlock of an AIO-SX subcloud reproduces the issue about 1 out of 4 reboots.

Potential cause:
- On startup, the kubelet is trying to reattach all the volumes to the pods. For some reason, it fails to do this sometimes and the pods are then stuck in the unknown state. The kubelet logs show a “MountVolume.SetUp failed for volume” error when this happens.
- There are several k8s bug reports that reference the “MountVolume.SetUp failed for volume” error shown above. The most promising one is: https://github.com/kubernetes/kubernetes/issues/68211
- The pods that usually experience the failure are stx-oidc-client and some of the monitor pods (e.g. mon-metricbeat-metrics and mon-logstash). These pods use the “subPath” volume configuration referenced in the above issue.
- Another pointer to the above issue is that when a pod is stuck in the unknown state, listing the contents of the pods volumes under /var/lib/kubelet/<uuid>/volumes shows the contents of all the volumes, except the volume using the subPath.

Yesterday I tried to update the deployment for the stx-oidc-client pods to use one of the workarounds that avoids the subPath volume configuration (workarounds suggested in the bug report):
- Use “items” in the volume config instead of subPath.
- Use projected volumes instead of subPath.

After several hours of effort I was not able to get the updated deployment to work. The pod would always fail to run, with errors like this:
Error: failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"./stx-oidc-client\": stat ./stx-oidc-client: no such file or directory": unknown

I suspect an actual change to the image might be required.

My thinking is that if we remove the subPath config from stx-oidc-client and verify that the issue no longer happens with that pod, we’d at least be able to confirm the cause. We would then want to remove the subPath config from the stx-monitor pods (that could take some effort). Even then, we still may have cases where user pods are using subPath (assuming that is the cause), so we still may need to update the workaround we have in sysinv to kill pods in the NodeAffinity sate to also kill any pods stuck with Unknown status after a certain amount of time has expired after the reboot. The workaround is not ideal though because we’d need to wait a significant amount of time before killing them because some pods do go through the Unknown status as they come up - that is why it makes sense to avoid using the subPath so we can avoid the workaround as much as possible.