pod stuck in "ContainerCreating" status

[Peng Peng]

Brief Description
-----------------
After create client-pod by using client-pod1.yaml, client-pod status stuck at "ContainerCreating".

Severity
--------
Major

Steps to Reproduce
------------------
kubectl create -f client-pod1.yaml
kubectl get pod -o=wide -l=client=pod-to-pod

TC-name: test_pod_to_pod.py::TestPodtoPod::test_pod_to_pod_connection

Expected Behavior
------------------
pod in Running status

Actual Behavior
----------------
pod in ContainerCreating status

Reproducibility
---------------
intermittent

System Configuration
--------------------
Two node system

Lab-name: R430_1-2

Branch/Pull Time/Commit
-----------------------
2020-03-31_00-10-00

Last Pass
---------
same load different lab

Timestamp/Logs
--------------
[2020-04-01 08:34:03,121] 314 DEBUG MainThread ssh.send :: Send 'kubectl create -f client-pod1.yaml'
[2020-04-01 08:34:03,580] 436 DEBUG MainThread ssh.expect :: Output:
pod/client-pod1 created
controller-1:~$

2020-04-01 08:34:04,617] 314 DEBUG MainThread ssh.send :: Send 'kubectl get pod -o=wide -l=client=pod-to-pod'
[2020-04-01 08:34:04,883] 436 DEBUG MainThread ssh.expect :: Output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
client-pod1 0/1 ContainerCreating 0 1s <none> controller-0 <none> <none>
client-pod2 0/1 ContainerCreating 0 0s <none> controller-1 <none> <none>
controller-1:~$

[2020-04-01 08:38:02,796] 314 DEBUG MainThread ssh.send :: Send 'kubectl get pod server-pod-dep-69f6c4ffd7-xrb27 server-pod-dep-69f6c4ffd7-zk2qj client-pod1 -n=default -o=wide'
[2020-04-01 08:38:03,149] 436 DEBUG MainThread ssh.expect :: Output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
server-pod-dep-69f6c4ffd7-xrb27 0/1 ContainerCreating 0 4m <none> controller-0 <none> <none>
server-pod-dep-69f6c4ffd7-zk2qj 0/1 ContainerCreating 0 4m <none> controller-0 <none> <none>
client-pod1

cat client-pod1.yaml
apiVersion: v1
kind: Pod
metadata:
  labels: {client: pod-to-pod}
  name: client-pod1
  namespace: default
spec:
  containers:
  - args: [sleep 60m]
    command: [/bin/sh, -c]
    image: tis-lab-registry.cumulus.wrs.com:9001/pv-test
    imagePullPolicy: IfNotPresent
    name: client-container
  nodeSelector: {test: server}
  restartPolicy: Always

Test Activity
-------------
Sanity

[Peng Peng (ppeng)]

Lo added:
https://files.starlingx.kube.cengn.ca/launchpad/1870186

[Ghada Khalil (gkhalil)]

@Peng, please monitor for a re-occurrence. This sounds like it's a one-of failure.

[Peng Peng (ppeng)]

Issue was reproduced on
Lab: R430_1_2
Load: 2020-04-07_00-10-00

log upload at:https://files.starlingx.kube.cengn.ca/launchpad/1870186

 [2020-04-08 08:39:25,763] 314 DEBUG MainThread ssh.send :: Send 'kubectl get pod server-pod-dep-69f6c4ffd7-dgb29 server-pod-dep-69f6c4ffd7-jtbsp client-pod1 -n=default -o=wide'
[2020-04-08 08:39:26,074] 436 DEBUG MainThread ssh.expect :: Output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
server-pod-dep-69f6c4ffd7-dgb29 0/1 ContainerCreating 0 4m2s <none> controller-0 <none> <none>
server-pod-dep-69f6c4ffd7-jtbsp 0/1 ContainerCreating 0 4m2s <none> controller-0 <none> <none>
client-pod1 0/1 ContainerCreating 0 4m1s <none> controller-0 <none> <none>
controller-1:~$

[Ghada Khalil (gkhalil)]

stx.4.0 / medium priority - issue appears to be reproducible at least on the one lab; requires further investigation

[Peng Peng (ppeng)]

Issue was reproduced at
Lab: R430_1_2
Load: 2020-04-08_00-10-00

[Frank Miller (sensfan22)]

Re-tagging to stx.5.0 as this issue is very intermittent and not seen with the main STX application or platform pods.

[Ghada Khalil (gkhalil)]

Changing the priority to Low. This has not been since April 2020.

[Peng Peng (ppeng)]

This issue reproduced more frequently on R730-1.

load: 2020-11-12_20-00-07

New log added at
https://files.starlingx.kube.cengn.ca/launchpad/1870186