Low-latency worker node reboots when pods under heavy load

[Brent Rowsell]

Brief Description
-----------------
This is likely related to
https://bugs.launchpad.net/starlingx/+bug/1830296

I was running a pod with exclusive cpu's running a process in a busy loop. After a few mins it rebooted.

Severity
--------
Critical

Steps to Reproduce
------------------
See above

Expected Behavior
------------------
No reboot

Actual Behavior
----------------
Reboot

Reproducibility
---------------
100%

System Configuration
--------------------
Standard config

Branch/Pull Time/Commit
-----------------------
2019-05-22 17:57:16 -0400

Last Pass
---------
Don;t know

Timestamp/Logs
--------------
There were no useful logs. All we see is the loss of mgmt/cluster network heartbeat on the controller. It appears the worker simply stopped responding.

Test Activity
-------------
Other ]

https://bugs.launchpad.net/starlingx/+bug/1830296

[Ghada Khalil (gkhalil)]

Marking as release gating; container testing taking the node down

[Jim Gauld (jgauld)]

There are few changes to kubelet that require upstream fixes. Will open upstream issues to track. eg., prevent throttling of CFS shares for Guaranteed pods; ability to isolate linux platform from kubepods. Usage of 'isolcpus' has already been disabled for low-latency, so we no longer get tasks stuck on specific cores.

[Frank Miller (sensfan22)]

After discussion with Brent (containers TL), we agreed to re-gate this issue to stx.3.0 as the fixes required will need to be implemented in the upstream kubernetes package.

[Frank Miller (sensfan22)]

Lowered priority to medium as issue is only seen under very heavy load.

[Frank Miller (sensfan22)]

This issue has not yet been addressed in the kubernetes package. Removing the stx.3.0 tag.

[Ghada Khalil (gkhalil)]

Lowering the priority as this may require k8s upstream fixes. We will not hold up stx.4.0 for this given that the issue was reported a year ago and is present in previous releases.

[Ramaswamy Subramanian (rsubrama)]

No progress on this bug for more than 2 years. Candidate for closure.

If there is no update, this issue is targeted to be closed as 'Won't Fix' in 2 weeks.

[Ramaswamy Subramanian (rsubrama)]

Changing the status to 'Won't Fix' as there is no activity.