Pod stuck at terminating state after multiple create & delete

[Senthil Mukundakumar]

Brief Description
-----------------

Pod stuck at terminating state after creating and deleting pods multiple times.
No of pods in the system 300.

large-benchmark burstable-stress-large-6dc96d67bc-f5vcn 0/1 Terminating 3 14h
large-benchmark burstable-stress-large-6dc96d67bc-fkczb 0/1 Terminating 2 14h
large-benchmark burstable-stress-large-6dc96d67bc-fr7dk 0/1 Terminating 4 14h
large-benchmark burstable-stress-large-6dc96d67bc-jvqvv 0/1 Terminating 3 14h
large-benchmark burstable-stress-large-6dc96d67bc-kllr8 0/1 Terminating 1 14h
large-benchmark burstable-stress-large-6dc96d67bc-pvrl2 0/1 Terminating 4 14h
large-benchmark burstable-stress-large-6dc96d67bc-pzcb2 0/1 Terminating 1 14h
large-benchmark burstable-stress-large-6dc96d67bc-sgmhb 0/1 Terminating 4 14h
large-benchmark burstable-stress-large-6dc96d67bc-x7vf5 0/1 Terminating 5 14h
large-benchmark burstable-stress-large-6dc96d67bc-zm54w 0/1 Terminating 1 14h

Severity
--------
Low

Steps to Reproduce
------------------

1. Create & delete 300 pods multiple times.
Use benchmark test script to reproduce the issue
scripts/apps-up.sh -m large -w 5
scripts/apps-down.sh -m large
scripts/apps-up.sh -m large -w 10
scripts/apps-down.sh -m large
scripts/apps-up.sh -m large -w 15
scripts/apps-down.sh -m large
scripts/apps-up.sh -m large -w 5
scripts/apps-down.sh -m large
scripts/apps-up.sh -m large -w 20
scripts/apps-down.sh -m large

Expected Behavior
------------------
Pods should be terminated and deleted

Actual Behavior
----------------
Pods stuck at terminating state

Reproducibility
---------------
Intermittent

System Configuration
--------------------
Large sytem (2_4_20)WCP_35_60

Branch/Pull Time/Commit
-----------------------
2020-06-05_20-00-00

Timestamp/Logs
--------------
2020-06-13T03:51:09.814670004Z
https://files.starlingx.kube.cengn.ca/launchpad/1883379

Test Activity
-------------
System Test

[Ghada Khalil (gkhalil)]

As suggested by Yang, assigning to Nimalini to re-visit the test-case to determine if the # of pods are within reasonable engineering limits.