kubernetes-core fails to deploy successfully on localhost provider

[Jon Seager]

Hi,

I've been testing the `kubernetes-core` bundle on LXD in a couple of different environments, including:

- Ubuntu 20.10 on a Ryzen desktop
- Ubuntu 20.04 inside a Multipass VM
- Ubunutu 20.04 on a DigitalOcean droplet

I'm seeing a failure in the deployment of the `easyrsa` charm. It seems that `easyrsa` is deployed to a LXD container inside machine 0, but the journal shows a load of errors about AppArmor confinement and therefore an inability to install snaps.

I'm installing like so:

```
sudo snap install lxd --classic
sudo lxd init --auto
sudo snap install juju --classic
sudo lxc network set lxdbr0 ipv6.address none
juju bootstrap localhost
juju deploy kubernetes-core
```

Juju Status shows: https://pastebin.canonical.com/p/c6J7jbXTzZ/
Juju debug log: https://pastebin.canonical.com/p/ppM3cqbvk7/
Troubleshooting showing broken LXD container on machine 0: https://pastebin.canonical.com/p/SpJKwjrWHJ/

Note that I can get the bundle to deploy if I patch the bundle with an overlay that deploys `easyrsa` to machine 0 directly, rather than a container within. The overlay looks like this:

```
---
applications:
  easyrsa:
    annotations:
      gui-x: "90"
      gui-y: "420"
    charm: cs:~containers/easyrsa-345
    num_units: 1
    resources:
      easyrsa: 5
    to:
      - "0"
  kubernetes-worker:
    options:
      kubelet-extra-config: '{protectKernelDefaults: false}'
```

Without the `options/kubelet-extra-config` section, the nodes never become ready for scheduling in my experience. This overlay has worked in all three of the aforementioned environments.

[George Kraft (cynerva)]

Thanks for the report. Seems like the nested LXD machine causes problems on the localhost provider. The overlay you've posted looks like a good workaround to me.

> Without the `options/kubelet-extra-config` section, the nodes never become ready for scheduling in my experience.

This is a known issue that is being tracked here: https://bugs.launchpad.net/charm-kubernetes-worker/+bug/1903566

[Nobuto Murata (nobuto)]

For the LXD part, it's also tricky to have LXD on top of OpenStack with Fan network (https://bugs.launchpad.net/juju/+bug/1936842).

In terms of the necessity of LXD, What's a rationale behind it? It seems to be that having lxd:0 for easyrsa is in that way since 2016. But having easyrsa in machine 0 co-located with kubernetes-master, etcd, containerd, and flannel is working for me as well.

If we can remove the LXD part from the default bundles in charm store, that would definitely lower the barrier of trying out charmed k8s and also saves time to deploy it since we don't have to download the LXD image (~400MB).