log flooding with "cgroup v2 not implemented for Stats: not implemented"

[Nobuto Murata]

How to reproduce:
$ juju deploy kubernetes-core
and it will deploy jammy + K8s 1.26 as of writing.

Then check syslog of a worker node with "cgroup v2 not implemented for Stats: not implemented".

$ juju ssh kubernetes-worker/0 \
    'grep -c "Jan 11 .*cgroup v2 not implemented for Stats: not implemented" /var/log/syslog'
-> 230512

So containerd generates 230k lines per day with the error.

According to the following link, it seems we need some face lift on containerd config:
https://github.com/containerd/containerd/discussions/6446

Generated config by the charm:

====
[plugins]
  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false
  [plugins."io.containerd.grpc.v1.cri"]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "rocks.canonical.com:443/cdk/pause:3.6"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    [plugins."io.containerd.grpc.v1.cri".containerd]
      no_pivot = false
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v1"
====

List of processes:

====
$ pgrep -af containerd
33032 /usr/bin/containerd
48394 /usr/bin/containerd-shim-runc-v1 -namespace k8s.io -id 40fb1e50dd8ef1fb80bcd29d9d1f006640d8569cac3b81ef0ab0394f772756f3 -address /run/containerd/containerd.sock
48723 /usr/bin/containerd-shim-runc-v1 -namespace k8s.io -id c5865f753ff361bd922e1b51f5afd0e98b44585c1cfe2678e6f7edc4b14a908d -address /run/containerd/containerd.sock
53116 /usr/bin/containerd-shim-runc-v1 -namespace k8s.io -id 905952486d79f2dd2ad3975881b1da56819bbb9236b25c1fa51529bf26676089 -address /run/containerd/containerd.sock
53196 /usr/bin/containerd-shim-runc-v2 -namespace default -id calico-node -address /run/containerd/containerd.sock
53595 /usr/bin/containerd-shim-runc-v1 -namespace k8s.io -id fa8b80c71b5dcb1b68571736d8b7a4c28c861016d63478a5f7261bf3fe3b8bb2 -address /run/containerd/containerd.sock

(...)

57434 /usr/bin/containerd-shim-runc-v1 -namespace k8s.io -id af2e6c870ddc8075cb71e91d31832b0cf1941b8ac695524e01b75d12ec15f46c -address /run/containerd/containerd.sock
57479 /usr/bin/containerd-shim-runc-v1 -namespace k8s.io -id 561d81174392b112e7936709e0937ca55177a575d760e40f0077f6cdfc687e79 -address /run/containerd/containerd.sock
62259 /snap/kubelet/2773/kubelet --kubeconfig=/root/cdk/kubeconfig --v=0 --node-ip=10.5.5.125 --container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --cloud-provider=external --config=/root/cdk/kubelet/config.yaml --pod-infra-container-image=rocks.canonical.com:443/cdk/pause:3.6
====

[Nobuto Murata (nobuto)]

Subscribing ~field-high.

[Angelos Kolaitis (aggkolaitis)]

Some insight from the microk8s side:

```
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v1"
```

We had this on MicroK8s as well, starting with containerd 1.6. `io.containerd.runc.v1` uses the v1 cgroup implementation, which causes a noisy Stats() call failing repeatedly (once every ~10-20 seconds, one line for each pod that is running).

In MicroK8s, we make sure to always "io.containerd.runc.v2" when available, which resolved it for us. See https://github.com/canonical/microk8s/blob/314d845827ede0159015e6a68f64d1b775c37bca/microk8s-resources/wrappers/run-containerd-with-args#L53-L58 for reference.

Hope this helps.

[Adam Dyess (addyess)]

https://github.com/charmed-kubernetes/charm-containerd/pull/89