| 2019-11-06 22:03:06 |
Jim Gauld |
description |
Brief Description:
-----------------
On AIO configurations where kubelet has "--cpu-manager-policy=none", all K8S tasks are being re-affined back to platform cores. This behaviour makes sense on openstack-compute-node, but not in general for non-openstack. As a result, the application cores are completely unusable and platform is degraded as any apps get launched.
Severity:
--------
Major.
Steps to Reproduce:
------------------
Install AIO, the 'none' cpu manager policy is configured by default.
Optionally: set host-label kube-cpu-mgr-policy=none.
Expected behaviour:
------------------
Platform daemons are affined to platform cores.
K8S pods have cgroup cpuset that float on all cores.
Actual behaviour:
----------------
All tasks end up on the platform cores, i.e., share the same cpuset
The application cores are unused.
Running pod application benchmarks are severely cpu limited.
Eg., noticed when attempting 'fio' performance benchmarks for CEPH.
Reproducible:
100% reproducible
System configuration:
AIO-SX and AIO-DX.
Branch/Pull Time/Commit
-----------------------
BUILD_DATE="2019-10-31 10:38:49 -0400"
Last pass:
---------
Prior to introduction of configurable cpu-manager-policy with default of none.
Previous default was 'static'.
Timestamp/Logs:
--------------
no logs. Issue is not obvious by logs.
Test Activity:
-------------
Developer testing.
Pod performance benchmarking activity for CEPH hit same issue. |
Brief Description:
-----------------
On AIO configurations where kubelet has "--cpu-manager-policy=none", all K8S tasks are being re-affined back to platform cores. This behaviour makes sense on openstack-compute-node, but not in general for non-openstack. As a result, the application cores are completely unusable and platform is degraded as any apps get launched.
Severity:
--------
Major.
Steps to Reproduce:
------------------
Install AIO, the 'none' cpu manager policy is configured by default.
Optionally: set host-label kube-cpu-mgr-policy=none.
Expected behaviour:
------------------
Platform daemons are affined to platform cores.
K8S pods have cgroup cpuset that float on all cores.
Actual behaviour:
----------------
All tasks end up on the platform cores, i.e., share the same cpuset
The application cores are unused.
Running pod application benchmarks are severely cpu limited.
Eg., noticed when attempting 'fio' performance benchmarks for CEPH.
Reproducible:
100% reproducible
System configuration:
AIO-SX and AIO-DX.
Branch/Pull Time/Commit
-----------------------
BUILD_DATE="2019-10-31 10:38:49 -0400"
Last pass:
---------
Prior to introduction of configurable cpu-manager-policy with default of none. Previous default was 'static'.
Issue was introduced with: https://review.opendev.org/#/c/689610/
Timestamp/Logs:
--------------
no logs. Issue is not obvious by logs.
Test Activity:
-------------
Developer testing.
Pod performance benchmarking activity for CEPH hit same issue. |
|
