AIO K8S tasks affined to platform cores with cpu manager policy none

[Jim Gauld]

Brief Description:
-----------------
On AIO configurations where kubelet has "--cpu-manager-policy=none", all K8S tasks are being re-affined back to platform cores. This behaviour makes sense on openstack-compute-node, but not in general for non-openstack. As a result, the application cores are completely unusable and platform is degraded as any apps get launched.

Severity:
--------
Major.

Steps to Reproduce:
------------------
Install AIO, the 'none' cpu manager policy is configured by default.
Optionally: set host-label kube-cpu-mgr-policy=none.

Expected behaviour:
------------------
Platform daemons are affined to platform cores.
K8S pods have cgroup cpuset that float on all cores.

Actual behaviour:
----------------
All tasks end up on the platform cores, i.e., share the same cpuset
The application cores are unused.

Running pod application benchmarks are severely cpu limited.
Eg., noticed when attempting 'fio' performance benchmarks for CEPH.

Reproducible:
100% reproducible

System configuration:
AIO-SX and AIO-DX.

Branch/Pull Time/Commit
-----------------------
BUILD_DATE="2019-10-31 10:38:49 -0400"

Last pass:
---------
Prior to introduction of configurable cpu-manager-policy with default of none. Previous default was 'static'.
Issue was introduced with: https://review.opendev.org/#/c/689610/

Timestamp/Logs:
--------------
no logs. Issue is not obvious by logs.

Test Activity:
-------------
Developer testing.
Pod performance benchmarking activity for CEPH hit same issue.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693272
Committed: https://git.openstack.org/cgit/starlingx/utilities/commit/?id=01c7f51607431ef3e64df71ca2be2984aa8ef2d9
Submitter: Zuul
Branch: master

commit 01c7f51607431ef3e64df71ca2be2984aa8ef2d9
Author: Jim Gauld <email address hidden>
Date: Wed Nov 6 18:33:14 2019 -0500

    Correct K8S and docker affinity on AIO

    This updates K8S cpuset affinity on AIO when kubernetes cpu
    manager policy is configured to 'none' (i.e., the default setting)
    and openstack-compute-node is not configured.

    This update makes K8s pods float on all cores instead of being
    reaffined to platform cores. This is done since kubernetes cannot
    isolate kube-system platform versus application pods.

    This update affines docker uwsgi tasks to platform cores.

    Change-Id: Iee40f747025c9777f80a94fe96b7c90e91d017e6
    Closes-bug: 1851569
    Signed-off-by: Jim Gauld <email address hidden>