When /var/lib/kubelet is filled, "kubectl describe nodes" still shows NodeHasNoDiskPressure and no Taints

[Ramesh kumar Sivanandam]

Brief Description:
When alarm "100.104 File System threshold exceeded" is raised, "kubectl describe nodes" still shows "controller-0 status is now: NodeHasNoDiskPressure"

Severity:
Major

Steps to Reproduce:

df -h | grep /var/lib/kubelet
sudo dd if=/dev/zero of=/var/lib/kubelet/hd-fillup.zeros bs=1G count=9
fm alarm-list
kubectl describe nodes 2>&1 | grep "Pressure"

Expected Behavior:

Alarm is raised, an kubectl describe nodes shows "status is now: NodeHasDiskPressure"

Actual Behavior:

Alarm "100.104 File System threshold exceeded" is raised, but "kubectl describe nodes" still shows "controller-0 status is now: NodeHasNoDiskPressure"

Reproducibility:

reproducible

[sysadmin@controller-0 ~(keystone_admin)]$ df -h | grep /var/lib/kubelet

/dev/mapper/cgts--vg-kubelet--lv 9.8G 9.1G 234M 98% /var/lib/kubelet

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl describe nodes 2>&1 | grep Taint

Taints: <none>
[sysadmin@controller-0 ~(keystone_admin)]$ fm alarm-list

+-------+------------------------------------------------------------------+--------------------------------------+----------+---------------+
| Alarm | Reason Text | Entity ID | Severity | Time Stamp |
| ID | | | | |
+-------+------------------------------------------------------------------+--------------------------------------+----------+---------------+
| 100. | File System threshold exceeded ; threshold 90.00%, actual 92.37% | host=controller-0.filesystem= | critical | 2022-06-22T17 |
| 104 | | /var/lib/kubelet | | :53:48.776435 |

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl describe nodes 2>&1 | grep "Pressure"

  MemoryPressure False Wed, 22 Jun 2022 18:19:38 +0000 Tue, 21 Jun 2022 14:35:32 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
  DiskPressure False Wed, 22 Jun 2022 18:19:38 +0000 Tue, 21 Jun 2022 14:35:32 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
  PIDPressure False Wed, 22 Jun 2022 18:19:38 +0000 Tue, 21 Jun 2022 14:35:32 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
  Normal NodeHasNoDiskPressure 27h (x3 over 27h) kubelet Node controller-0 status is now: NodeHasNoDiskPressure
  Normal NodeHasNoDiskPressure 27h kubelet Node controller-0 status is now: NodeHasNoDiskPressure
  Normal NodeHasNoDiskPressure 27h (x7 over 27h) kubelet Node controller-0 status is now: NodeHasNoDiskPressure
  Normal NodeHasNoDiskPressure 16h (x7 over 16h) kubelet Node controller-0 status is now: NodeHasNoDiskPressure

Test Activity:
Regression Testing

[Ghada Khalil (gkhalil)]

Fixed by:
https://review.opendev.org/c/starlingx/ansible-playbooks/+/865524
https://review.opendev.org/c/starlingx/stx-puppet/+/865886