Collectd platform cpu breakdown has missing time

[Jim Gauld]

Brief Description
-----------------
There is a large discrepancy between the overall Platform cpu occupancy and platform time indicated as the sum of Base + k8s-system. This unaccounted time exaggerated on some systems when there is lots of health probes, and is greatly exaggerated on Debian (vs CentOS). It is unclear where all the time on the Platform cores is going, and this complicates traction against platform cputime alarms.

A better breakdown of the Platform cputime is require so we understand where the time goes so time adds to 100 percent and that makes sense. This must account for containerization overheads, systemd overheads, and kernel threads.

For system engineering tracking and debugging purposes, it is also desirable to have some level of cputime breakdown at the system.slice per-services level.

Eg, Propose new breakdown something like this:

2022-08-09T16:35:21.243 controller-0 collectd[3621482]: info platform cpu usage plugin Base usage: 29.1%; cpus: 2, (sm: 8.9, pmon: 6.1, containerd: 5.3, kubelet: 4.3, system-ceph: 1.2, etcd: 1.0, sw-patch-controller-daemon: 0.7, sm-watchdog: 0.7, collectd: 0.3, fm-api: 0.3, hbsAgent: 0.1)
2022-08-09T16:35:21.244 controller-0 collectd[3621482]: info platform cpu usage plugin Usage: 38.0% (avg per cpu); cpus: 2, Platform: 38.0% (Base: 29.1, k8s-system: 5.6), k8s-addon: 49.2, containers: 10.6, overhead: 3.3

Severity
--------
Major: in some cases we are no alarming platform cpu when lots of time is actually consumed. Platform performance is hard to diagnose and debug when we don't understand where time goes.

Steps to Reproduce
------------------
Fresh install Debian ISO. Monitor /var/log/collectd.log platform cpu usage logs. There is a large unaccounted discrepancy between overall cpu Usage compared to Platform (Base + k8s-system) breakdown -- these should actually match. This becomes exaggerated on Debian and systems with lots of health probes.

Expected Behavior
------------------
Expect the collectd cpu plugin to have matching overall Usage, (eg., see the following when it matches 32.2%).
platform cpu usage plugin Usage: 32.2% (avg per cpu); cpus: 2, Platform: 32.2% (Base: 23.7, k8s-system: 5.4)

Actual Behavior
----------------
Huge unexplained deviation between overall Usage and Platform Usage for the same cpus, especially on Debian, especially with large number of K8S health probes.

Reproducibility
---------------
100%

System Configuration
--------------------
AIO-SX, AIO-DX

Branch/Pull Time/Commit
-----------------------
BUILD_DATE="2022-06-25 02:49:38 -0400"

Last Pass
---------
Day one issue. Discrepancy is exaggerated on Debian since Jun8 2022.

Timestamp/Logs
--------------
Example when the discrepancy is small:
/var/log/collectd.log :
2022-06-22T16:58:22.940 controller-0 collectd[88488]: info platform cpu usage plugin Usage: 59.8% (avg per cpu); cpus: 4, Platform: 56.4% (Base: 56.4, k8s-system: 0.0), k8s-addon: 0.0

Test Activity
-------------
System engineering.

Workaround
----------
none available

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/monitoring/+/852653

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/monitoring/+/852653
Committed: https://opendev.org/starlingx/monitoring/commit/43f53c6b8371dec26a00db0029b31a513a9e268c
Submitter: "Zuul (22348)"
Branch: master

commit 43f53c6b8371dec26a00db0029b31a513a9e268c
Author: Jim Gauld <email address hidden>
Date: Tue Aug 9 12:36:42 2022 -0400

    Enhance collectd platform cpu plugin breakdown of where the time goes

    The collectd platform cpu plugin is modified to show a better breakdown
    of where the services cputime goes. This corrects a large unaccounted
    component of platform cputime. The top-down overall cpu time now closely
    matches the bottom-up aggregated calculation.

    New outputs are displayed:
    - new output line shows occupancy of hi-runner cgroup services
      within system.slice and user.slice.

    - new output field 'containers' shows the sum of cgroups related to
      containers platform services (containerd.service, docker.service,
      kubelet.service, etcd.service). This illustrates the overhead of
      hard-to-track short-lived processes under containerd-shim and
      runc when health/liveness probes are executed.

    - new output field 'overhead' derived measurement shows the
      remaining cputime not specifically tracked by first-level cgroups.
      This represents systemd init pid 1 and the kthreads running on
      Platform cpus. This overhead is now included in the Platform
      cputime.

    Tests:
    PASS: (Debian) AIO-SX verify new output 'containers', 'overhead',
          and system.slice hi-runners is displayed and makes sense.
    PASS: (CentOS) AIO-SX verify new output 'containers', 'overhead',
          and system.slice hi-runners is displayed and makes sense.
    PASS: (Debian) AIO-SX launch cpu hog 'yes' verify we see it show
          under 'Base' as user-x cgroup.
    PASS: (Debian) AIO-SX launch stressng pod in kube-system namespace
          verify it shows up as k8s-system.
    PASS: (Debian) Standard system verify new output on controller
          and workers.

    Closes-Bug: 1984129

    Signed-off-by: Jim Gauld <email address hidden>
    Change-Id: Ib52234592d8f5402d0e363d2417969e04ff2fe29