[k8s] Error with bootstrap in k8s cluster with pod security

[Bartłomiej Poniecki-Klotz]

I'm bootstrapping the vanila k8s provided by client.
The bootstraping process hangs on the Pod creation for controller-0. Pod is in the CrashLoopBackOff.
The Pod security mode is set to enforce on the cluster

Bootstrap log:
$ juju --debug bootstrap datalake-kubeflow-dev.k8s.local datalake-kubeflow
https://pastebin.canonical.com/p/ppwb3WKP8H/

Bootstrap results:
controller-0 pod describe - https://pastebin.canonical.com/p/2D4mfS4RdZ/
controller-0 pod logs (both containers) - https://pastebin.canonical.com/p/Wq5q2dSHrP/

K8s version: 1.24.6.
Juju version: 2.9.35-ubuntu-amd64

[Ian Booth (wallyworld)]

The security policies being enforced are not allowing the juju agent to operate. What policy are you enforcing? "baseline"? "restricted"?

The jujud agent expects to operate as root in order to do it's job. So at this stage, only a policy of "privileged" would be possible I suspect.

You can see that the jujud agent is not being allowed to do its job:

/bin/sh: 1: cannot create /root/mongo.sh: Permission denied
mkdir: cannot create directory '/var/lib/juju/tools': Permission denied

etc

You could set up the cluster to warn/audit on policy violations instead of erroring and run juju and then set up your admission rules to allow the access that juju needs to operate.

[Ian Booth (wallyworld)]

I'll mark as Wishlist so we don't lose the bug but there are not currently plans support anything other than "privileged" for jujud

[Jon Seager (jnsgruk)]

I think we should be looking deeper into this. I can't see a particular reason why `jujud` needs to run as root in the context of a Kubernetes deployment.

In this particular case, just adding a `juju` user to the container as part of the Dockerfile, pre-creating the `/var/lib/juju` directory and ensuring it's owned by the right user would probably allow jujud to run as a non-privileged user, no?

I think things are different on machines, but I can't see a particularly compelling reason to require root on Kubernetes?

[John A Meinel (jameinel)]

I think this is a general issue that we need to start addressing for the next minor release of Juju. It isn't one that I'd do a 'quick bugfix' for, but is worth making some plans for external acceptance of the juju ecosystem and stronger confinement support.

[Jon Seager (jnsgruk)]

I think this should be addressed in Juju 3.4, which should enable "rootless" deployments on Kubernetes.