Comment 7 for bug 1898726

Revision history for this message
Drew Freiberger (afreiberger) wrote :

@cynerva

Working with the Field team, we found that this is not reproducable with k8s-worker constraints of:
cores=4 mem=4G root-disk=16G root-disk-source=volume

But we are able to reproduce simply by increasing these constraints to:
cpu-cores=16 mem=131072 root-disk=153600 root-disk-source=volume

This doesn't change numa architecture (both vms end up with a single numa node), but is somehow affecting whether mlocking is attempted or not. It's either the larger number of cores or the larger memory footprint triggering the mlock attempt.

We found a workaround by setting the following in /etc/default/docker and restarting docker.service.

DOCKER_OPTS="--default-ulimit=memlock=-1:-1"

The docker charm's "docker-opts" does not populate this setting, so we had to manually configure this on the kubernetes-worker units. We've also filed related bug: https://bugs.launchpad.net/charm-docker/+bug/1939038