debian: enable kubernetes cgroupRoot k8s-infra

Bug #1978733 reported by Jim Gauld
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Jim Gauld

Bug Description

Brief Description
-----------------
Re-enable the cgroupRoot k8s-infra for Debian. This was disabled on Debian during the initial integration and is required to have kubernetes collectd cpu and memory plugins. Note this is tied to FM alarming of platform.

Severity
--------
Major: Missing K8S cpu and memory monitoring and alarming.

Steps to Reproduce
------------------
Debian fresh install.
Inspect /var/log/collectd.log, there are no cpu and memory plugin results.

Expected Behavior
------------------
cat /var/lib/kubelet/config.yaml | grep cgroup
cgroupDriver: cgroupfs
cgroupRoot: /k8s-infra

See breakdown like this in /var/log/collectd.log.
collectd[900868]: info platform memory usage: Usage: 52.4%; Reserved: 8000.0 MiB, Platform: 4193.8 MiB (Base: 3088.7, k8s-system: 1105.1), k8s-addon: 0.0
collectd[900868]: info 4K memory usage: Anon: 21.4%, Anon: 4071.0 MiB, cgroup-rss: 4208.3 MiB, Avail: 14936.1 MiB, Total: 19007.1 MiB
collectd[900868]: info 4K numa memory usage: node0, Anon: 20.92%, Anon: 4071.0 MiB, Avail: 15389.6 MiB, Total: 19460.6 MiB
collectd[900868]: info platform cpu usage plugin Usage: 16.2% (avg per cpu); cpus: 4, Platform: 15.8% (Base: 11.8, k8s-system: 4.0), k8s-addon: 0.0

Write down what was expected after taking the steps written above

Actual Behavior
----------------
See that the configured cgroupRoot is / instead of k8s-infra.
cat /var/lib/kubelet/config.yaml | grep cgroup
cgroupDriver: cgroupfs
cgroupRoot: /

Reproducibility
---------------
100%

System Configuration
--------------------
Debian all configs.

Branch/Pull Time/Commit
-----------------------
BUILD_ID="2022-06-11_22-00-50"

Last Pass
---------
CentOS.

Timestamp/Logs
--------------
na

Test Activity
-------------
Feature Testing, Developer Testing

Workaround
----------
Edit each kubernetes node /var/lib/kubelet/config.yaml
change cgroupRoot from "/" to "/k8s-infra"
lock/unlock host

Jim Gauld (jgauld)
Changed in starlingx:
assignee: nobody → Jim Gauld (jgauld)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-playbooks (master)
Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-playbooks (master)

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/845828
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/ec2caf54e52cc3df27c5449f6946dcf847ccd385
Submitter: "Zuul (22348)"
Branch: master

commit ec2caf54e52cc3df27c5449f6946dcf847ccd385
Author: Jim Gauld <email address hidden>
Date: Tue Jun 14 17:06:37 2022 -0400

    debian: reenable kubernetes cgroupRoot: k8s-infra

    This reenables the kubernetes cgroupRoot: /k8s-infra setting which got
    disabled during initial integration on Debian. This is required for
    custom cgroup configurability, and platform cpu and memory alarming
    using collectd plugins.

    Closes-Bug: 1978733

    TESTPLAN: Debian
    PASS: AIO-SX fresh install, verify cgroupRoot: /k8s-infra
          in /var/lib/kubelet/config.yaml
    PASS: Manually enable collectd plugins, verify K8S details in
          /var/log/collectd.log for cpu and memory.

    Signed-off-by: Jim Gauld <email address hidden>
    Change-Id: If1abd296f4b107b78e61348b7a5f55b9632615b9

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Medium
tags: added: stx.7.0 stx.containers stx.debian
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.