Large cpu discrepancy between the schedstat platform cores and cgroup cpuacct based measurements

[Tee Ngo]

Brief Description
-----------------
There's a large discrepancy between cpu Usage reading and cpu Platform reading on worker nodes in a Standard system. Here's a sample:

2019-10-28T01:50:58.550 compute-0 collectd[79959]: info platform cpu usage plugin Usage: 19.3% (avg per cpu); cpus: 1, Platform: 28.0% (Base: 17.3, k8s-system: 8.1, k8s-addon: 2.6)
2019-10-28T01:51:08.550 compute-0 collectd[79959]: info platform cpu usage plugin Usage: 21.7% (avg per cpu); cpus: 1, Platform: 33.2% (Base: 18.1, k8s-system: 8.5, k8s-addon: 6.7)
2019-10-28T01:51:18.550 compute-0 collectd[79959]: info platform cpu usage plugin Usage: 14.8% (avg per cpu); cpus: 1, Platform: 22.5% (Base: 11.5, k8s-system: 8.5, k8s-addon: 2.5)
2019-10-28T01:51:28.550 compute-0 collectd[79959]: info platform cpu usage plugin Usage: 12.7% (avg per cpu); cpus: 1, Platform: 22.0% (Base: 10.7, k8s-system: 7.4, k8s-addon: 3.9)
2019-10-28T01:51:38.550 compute-0 collectd[79959]: info platform cpu usage plugin Usage: 21.0% (avg per cpu); cpus: 1, Platform: 27.3% (Base: 17.6, k8s-system: 7.6, k8s-addon: 2.2)

Severity
--------
Major

Steps to Reproduce
------------------
Install Oct. 24th build in a standard lab
Check host cpu usage in daemon.log

Expected Behavior
------------------
The overall "Usage" reading from schedstats should be in close agreement or higher than the "Platform" reading from cgroup cpuacct depending whether k8s-addon is included in or excluded from "Platform" respectively.

Actual Behavior
----------------
The Platform cpu reading is considerably higher than the overall Usage reading. The issue appears to stem from logic flaw in cpu usage accounting when --cpu-manager-policy is set to none.

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Standard

Branch/Pull Time/Commit
-----------------------
Oct. 24th or newer

Last Pass
---------
N/A

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Evaluation

[Frank Miller (sensfan22)]

When --cpu-manager-policy is set to none, it appears that some k8s platform processes are floating across all cores and not exclusively running on just the platform cores.

As such marking this medium priority & stx.3.0 gating.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/697718

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697718
Committed: https://git.openstack.org/cgit/starlingx/utilities/commit/?id=1903bbab7f7dc48259b4770e76a80a6fc7e77805
Submitter: Zuul
Branch: master

commit 1903bbab7f7dc48259b4770e76a80a6fc7e77805
Author: Jim Gauld <email address hidden>
Date: Fri Dec 6 15:25:43 2019 -0500

    Correct reaffining of platform and K8S on AIO

    This ensures correct cpuset isolation when doing dynamic reaffining on
    AIO during host initialization, based on kubernetes cpu-manager-policy.

    This corrects a bug where some K8S registry and containerd-shim
    processes were floating across all cores instead of platform cores due
    to improper matching with 'docker' string pattern.

    This manually affines docker '/pause' tasks to cpu 0.

    K8S tasks are dynamically reaffined only for openstack computes.

    Change-Id: I00abbd10d1a3f4dfdc46cd961f55d75339932339
    Closes-Bug: 1850242
    Signed-off-by: Jim Gauld <email address hidden>

[Ghada Khalil (gkhalil)]

Next step is to cherrypick to the r/stx.3.0 branch. Please complete this by Tuesday Dec 10.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.3.0
Review: https://review.opendev.org/698313

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698313
Committed: https://git.openstack.org/cgit/starlingx/utilities/commit/?id=cbad6b792157d066dd601f0f9ce62dc177d4c848
Submitter: Zuul
Branch: r/stx.3.0

commit cbad6b792157d066dd601f0f9ce62dc177d4c848
Author: Jim Gauld <email address hidden>
Date: Fri Dec 6 15:25:43 2019 -0500

    Correct reaffining of platform and K8S on AIO

    This ensures correct cpuset isolation when doing dynamic reaffining on
    AIO during host initialization, based on kubernetes cpu-manager-policy.

    This corrects a bug where some K8S registry and containerd-shim
    processes were floating across all cores instead of platform cores due
    to improper matching with 'docker' string pattern.

    This manually affines docker '/pause' tasks to cpu 0.

    K8S tasks are dynamically reaffined only for openstack computes.

    Change-Id: I00abbd10d1a3f4dfdc46cd961f55d75339932339
    Closes-Bug: 1850242
    Signed-off-by: Jim Gauld <email address hidden>
    (cherry picked from commit 1903bbab7f7dc48259b4770e76a80a6fc7e77805)

[Frank Miller (sensfan22)]

Re-opening this LP for one additional corner case:
When using --cpu-manager-policy=none, the kube-system pods can run on application isolated cores but they should only run on platform cores.
While this is not strictly high priority, it would be desired to source the commit for this both in master and in the stx.3.0 maintenance release branch so that all commits for this LP are in the 3.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/698777

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698777
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=71033a4d8a577fa6925e8407bf481c9a53d10087
Submitter: Zuul
Branch: master

commit 71033a4d8a577fa6925e8407bf481c9a53d10087
Author: Jim Gauld <email address hidden>
Date: Thu Dec 12 12:22:59 2019 -0500

    kubelet cpu-manager-policy=none options for application-isolated

    This changes kubelet cpu-manager options so that --kube-reserved is
    not specified in conjunction with --cpu-manager-policy=none.

    In the 'none' case, we only reserve cpus with --system-reserved so
    that kube-system pods only run on platform cores and not
    application-isolated cores.

    Change-Id: I2cf77d68b1c615ae6a0052d2b1e5b96a5c80d3d7
    Closes-Bug: 1850242
    Signed-off-by: Jim Gauld <email address hidden>

[Ghada Khalil (gkhalil)]

@Jim Gauld, please cherrypick the additional commit to the r/stx.3.0 branch for inclusion in the first maintenance release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.3.0
Review: https://review.opendev.org/702280

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702280
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=b123b0b7ce4130263b70f0c4a7cf59251fbdf000
Submitter: Zuul
Branch: r/stx.3.0

commit b123b0b7ce4130263b70f0c4a7cf59251fbdf000
Author: Jim Gauld <email address hidden>
Date: Thu Dec 12 12:22:59 2019 -0500

    kubelet cpu-manager-policy=none options for application-isolated

    This changes kubelet cpu-manager options so that --kube-reserved is
    not specified in conjunction with --cpu-manager-policy=none.

    In the 'none' case, we only reserve cpus with --system-reserved so
    that kube-system pods only run on platform cores and not
    application-isolated cores.

    Change-Id: I2cf77d68b1c615ae6a0052d2b1e5b96a5c80d3d7
    Closes-Bug: 1850242
    Signed-off-by: Jim Gauld <email address hidden>
    (cherry picked from commit 71033a4d8a577fa6925e8407bf481c9a53d10087)