StarlingX

  1. Bug #2009877
  2. Comment #0

Comment 0 for bug 2009877

Revision history for this message
Cesar Bombonate (cpompeud) wrote : collectd top 10 k8s system process list incorrectly has WRA processes

Brief Description
-----------------
Observe the collectd top 10 memory processes logs for "Kubernetes System" incorrectly contain WRA processes which should be only captured in "Kubernetes Addon" logs.
collectd separates pods by namespace into Kubernetes System for platform pods and Kubernetes Addon for WRA & openstack pods.
While this breakdown is correctly done in the overall platform memory usage log, the process top 10 list for Kubernetes System shows WRA processes (e.g. java, autodetect, etc.)

Will hamper debugging of memory usage issues.

Severity
--------
Minor

Steps to Reproduce
------------------
Install 22.12 build, apply WRA and observe collectd "top 10 memory rss process" logs

Expected Behavior
------------------
Processes in Kubernetes System and Kubernetes Addon should not overlap

Actual Behavior
----------------
Kubernetes System logs contain WRA processes

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Release: 22.12
Kernel: Real-time (low latency)
Hyperthreading: Disabled
Platform cores: 2
Application cores: 34
Labels
kube-cpu-mgr-policy=static
Huge pages: One 1 GB huge page was configured on each processo

Branch/Pull Time/Commit
-----------------------
https://review.opendev.org/c/starlingx/monitoring/+/840548

Last Pass
---------
The top 10 memory logs were introduced in 22.12

Timestamp/Logs
--------------
2023-01-09T22:25:32.172 controller-0 collectd[153770]: info The top 10 memory rss processes for the Kubernetes System are :[('java', '36.72 GiB'), ('java', '26.87 GiB'), ('java', '4.25 GiB'), ('java', '2.71 GiB'), ('autodetect', '860.24 MiB'), ('java', '826.97 MiB'), ('kube-apiserver', '801
.15 MiB'), ('autodetect', '606.67 MiB'), ('java', '363.57 MiB'), ('metricbeat', '249.55 MiB')]
2023-01-09T22:25:32.172 controller-0 collectd[153770]: info The top 10 memory rss processes Kubernetes Addon are :[('java', '36.70 GiB'), ('java', '26.87 GiB'), ('java', '4.25 GiB'), ('java', '2.71 GiB'), ('autodetect', '860.24 MiB'), ('java', '826.97 MiB'), ('autodetect', '606.67 MiB'), ('java', '363.57 MiB'), ('metricbeat', '251.21 MiB'), ('filebeat', '186.35 MiB')]

Test Activity
-------------
Performance Testing

Workaround
----------
NA