collectd platform stats do not track pods in newer namespaces

[Gerry Kopec]

Brief Description
-----------------
Collect platform cpu and memory usage logs need to be updated to include newer namespaces.
E.g.
2021-04-06T02:51:24.389 controller-0 collectd[134022]: info platform memory usage: Usage: 47.3%; Reserved: 8000.0 MiB, Platform: 3787.0 MiB (Base: 3173.3, k8s-system: 613.7), k8s-addon: 0.0

k8s-system will only include memory usage from pods in the kube-system namespace. This needs to be updated to include namespaces added to starlingx more recently: armada, cert-manager, portieris, vault & notification

Severity
--------
Minor

Steps to Reproduce
------------------
Create a cpu or memory stress pod in armada, cert-manager, portieris, vault or notification namespace.

Expected Behavior
------------------
Cpu and memory usage of stress pod should count against platform in collectd logs.

Actual Behavior
----------------
Cpu and memory usage of stress pod do not count against platform in collectd logs.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
All

Branch/Pull Time/Commit
-----------------------
n/a

Last Pass
---------
n/a

Timestamp/Logs
--------------
2021-04-06T02:51:24.372 controller-0 collectd[134022]: info platform cpu usage plugin Usage: 33.9% (avg per cpu); cpus: 2, Platform: 31.3% (Base: 26.5, k8s-system: 4.8), k8s-addon: 0.0
2021-04-06T02:51:24.389 controller-0 collectd[134022]: info platform memory usage: Usage: 47.3%; Reserved: 8000.0 MiB, Platform: 3787.0 MiB (Base: 3173.3, k8s-system: 613.7), k8s-addon: 0.0

Test Activity
-------------
System Engineering

Workaround
----------
none

[Gerry Kopec (gerry-kopec)]

Fix review: https://review.opendev.org/c/starlingx/monitoring/+/785085

[Gerry Kopec (gerry-kopec)]

Fix merged: https://opendev.org/starlingx/monitoring/commit/19460ecbd2c50b2d3fd8436d12066df5925f0fb4

[Ghada Khalil (gkhalil)]

screening: marking as stx.5.0 / medium as this appears to be fixed already

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/c/starlingx/monitoring/+/792244

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/monitoring/+/792244
Committed: https://opendev.org/starlingx/monitoring/commit/fdc0d099fb0d65cbf8f037fe0cc9ac8125410284
Submitter: "Zuul (22348)"
Branch: f/centos8

commit 2ef5451f442482636db3c0c3641e8412821bd8c5
Author: Takamasa Takenaka <email address hidden>
Date: Thu Apr 22 12:28:37 2021 -0300

    Format 2 lines ntpq data into 1 lines

    The problem was logic expected one line data for
    ntpq result. But it was 2 lines for each ntp server
    entry. When peer server is selected, script checked
    refid if refid is reliable or not but it could not
    find because refid is in the following line.
    This fix formats 2 lines data into 1 line.

    The minor alarm "minor alarm "NTP cannot reach
    external time source; syncing with peer controller
    only" is removed because NTP does not prioritize
    external time source over peer.

    Closes-Bug: 1889101

    Signed-off-by: Takamasa Takenaka <email address hidden>
    Change-Id: Icc8316bb1a7041bf0351165c671ebf35b97fa3bc

commit d37490b81408ca53b1b8fd61992c6c9337dbcaed
Author: Eric MacDonald <email address hidden>
Date: Tue Apr 20 10:03:07 2021 -0400

    Add alarm audit to starlingx collectd fm notifier plugin

    This update adds common plugin support for alarm state auditing.
    The audit is able to detect and correct the following alarm
    state errors:

       Error Case Correction Action
       ----------------------- -----------------
     - stale alarm ; delete alarm
     - missing alarm ; assert alarm
     - alarm severity mismatch ; refresh alarm

    The common audit is enabled for the fm_notifier plugin that supports
    alarm managment for the following resources.

     - CPU with alarm id 100.101
     - Memory with alarm id 100.103
     - Filesystem with alarm id 100.104

    Other plugins may use this common audit in the future but only the
    above resources have the audit enabled for them by this update.

    Test Plan:

    PASS: Verify stale alarm detection/correction handling
    PASS: Verify missing alarm detection/correction handling
    PASS: Verify alarm severity mismatch detection/correction handling
    PASS: Verify hosts only audits its own specified alarms
    PASS: Verify success path of monitoring a single and mix
          of base and instance alarms of varying severity while
          such alarm conditions come and go
    PASS: Verify alarm audit of mix of base and instance alarms
          over a collectd process restart
    PASS: Verify audit handling of alarm that migrates from
          major to critical to major to clear
    PASS: Verify audit handling transition between alarm and
          no alarm conditions
    PASS: Verify soak of random cpu, memory and filesystem
          overage alarm assertions and clears that also involve
          manual alarm deletions, assertions and severity changes
          that exercise new audit features

    Regression:

    PASS: Verify alarm and audit handling over Swact with mounted
          filesystem that has active alarm
  ...