[RFE] Have a consistent way to configure MTU across Charmed Kubernetes

[Pedro Guimarães]

As of today, there is no consistent way to configure MTU sizes on Charmed Kubernetes.
A user wanting to do so has to use config-flags or tweak with services deployed by CK manually to get it correctly.

We should provide one single config option that would enforce mtu sizing across the entire cluster.

One example, we can select MTU sizes with Calico, if we set options such as:
FELIX_IPINIPMTU, from: https://docs.projectcalico.org/reference/felix/configuration

That will allow us to define Calico's MTU size interface when using IP-over-IP.

Still, once you define it and you access a container, you can see that MTU size did not change within that container.
I am not entirely aware how can we enforce that across all containers.
There is an option for that on kubelet though:
--network-plugin-mtu int32
The MTU to be passed to the network plugin, to override the default. Set to 0 to use the default 1460 MTU. This docker-specific flag only works when container-runtime is set to docker.
Docs: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/

However, this seems to be docker-specific. Is there a way to achieve a pod-wide MTU setting with containerd as well?
On my tests with Calico + Containerd, containers seemed to be ignoring the MTU set on calico interface using FELIX_IPINIPMTU.

[Pedro Guimarães (pguimaraes)]

Fix for layer-calico available on: https://github.com/charmed-kubernetes/layer-calico/pull/55