After intelfpga label is removed, intel-fpga-plugin pod stays Terminating status

[Difu Hu]

Brief Description
-----------------
After intelfpga label is removed from host, intel-fpga-plugin pod stays at
Terminating status

Severity
--------
Major

Steps to Reproduce
------------------
lab installation
system host-label-assign controller-0 intelfpga=enabled
wait for intel-fpga-plugin pod running (kubectl get pods -A)
system host-label-remove controller-0 intelfpga

Expected Behavior
------------------
intel-fpga-plugin pod is terminated successfully.

Actual Behavior
----------------
intel-fpga-plugin pod stays at Terminating status.

Reproducibility
---------------
Reproduced 3 times

System Configuration
--------------------
AIO-SIMPLEX, IPv6
Lab-name: wp_15

Branch/Pull Time/Commit
-----------------------
2020-04-20_20-00-00

Last Pass
---------
Not sure

Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get pods -A | grep fpga
kube-system intel-fpga-plugin-5b72m 1/1
     Terminating 0 18h

Test Activity
-------------
Regression Testing

[Difu Hu (difuhu)]

log
https://files.starlingx.kube.cengn.ca/launchpad/1874319

[Difu Hu (difuhu)]

After the issue happens, re-assign label will NOT bring intel-fpga-plugin
to Running. The pod keeps staying at Terminating.

[Ghada Khalil (gkhalil)]

low priority - expect this is unique to this particular device plugin pod