Deploying in an LXD container results in an inability to install kube-* snaps

[Chris Johnston]

Deploying kubernetes-master in a privileged container results in the kube-* snaps not being able to be installed:

sudo snap install kube-apiserver --channel=1.19/stable
error: cannot perform the following tasks:
- Run configure hook of "kube-apiserver" snap if present (run hook "configure": cannot perform operation: mount --bind /snap/core18/current/etc/apparmor /tmp/snap.rootfs_OYxyqR/etc/apparmor: Permission denied)

This appears to be due to a change that was recently released [1] which, in a privileged container, causes the container, which has a newer effective snap-confine profile than the host. Since the container does not have confinement, it reverts to using the hosts snap-confine profile from an older version of snapd (2.47.1) which does not contain the permissions to allow mounting /etc/apparmor inside the snap.

Upgrading the version of snapd on the host to 2.48+18.04 results in again being able to install the snaps in the container.

[1] https://github.com/snapcore/snapd/pull/9516

For @Canonical, entire discussion can be found at: https://chat.canonical.com/canonical/pl/brc4q5skzfbjinmyu4fd6nhcpy

[Chris Sanders (chris.sanders)]

I'm marking this as invalid for the charm, this is a change in snapd that the charm has no control over.

Hopefully there's something snapd can do about graceful fallback in this case to avoid the regression in functionality.

[Ian Johnson (anonymouse67)]

I'm setting this as WontFix because snapd can't support snaps being run inside privileged lxd containers, the host and container policies will clash with each other which is in fact exactly what happened here with this particular snap.

[Chris Sanders (chris.sanders)]

After discussing with the snap and lxd teams, the issue is related to the profile which marks the apparmor profile as unconfined. The path forward to avoid this is to not override the lxc apparmor profile. That's what's causing the issues and not privileged setting.

[George Kraft (cynerva)]

Added kubernetes-worker because it likely has the same problem - both master and worker use the same LXD profile.

[Kevin W Monroe (kwmonroe)]

PRs for review:

https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/168
https://github.com/charmed-kubernetes/charm-kubernetes-worker/pull/91

[George Kraft (cynerva)]

Follow-up PR: https://github.com/charmed-kubernetes/charm-kubernetes-worker/pull/93

[Kevin W Monroe (kwmonroe)]

We had to revert this due to a regression for calico/vxlan when charms were deployed to lxd:

https://bugs.launchpad.net/charm-calico/+bug/1942099

PRs:
- https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/177
- https://github.com/charmed-kubernetes/charm-kubernetes-worker/pull/97

It's back on the list to fix for 1.22-ck1. The workaround for now is to ensure snapd versions are consistent between host and lxd VMs.

[Kevin W Monroe (kwmonroe)]

I've removed the milestone. We've verified the workaround here (ensure snapd versions across units), and that is less risky than a confined lxd profile considering calico+vxlan's importance in charmed k8s.