So hopefully, that helps prove that LXD is behaving identically on both systems...
I've done the same checks of AppArmor profile hashes for the snap-store-proxy snap in both environments and they're similarly identical. Though snap-store-proxy is obviously quite happy in the 20.04 on 20.04 environment.
I then moved the 20.04 system to the same kernel as the 21.10 one (using linux-generic-hwe-20.04-edge) and I'm getting a similarly broken behavior for that snap.
LXD's own profile doesn't restrict the capabilities that containers can use, especially now "chown". So to me this suggests an apparmor kernel issue.
My suspicion is that it's either an apparmor change which somehow causes more strict confinement of profiles loaded in containers than loaded on the host (we've seen that in the past with apparmor in a container needing a rule for the binary being confined whereas it's unneeded on the host), or it's a kernel change (possibly the idmap mounts feature?) which is confusing apparmor somehow.
I'm thinking of that last one because of the fact that "chown" is showed repeatedly as blocked by apparmor and given we have a VFS layer that remaps uid/gid on the fly now, maybe that's causing some confusion?
In any case, adding a task for the kernel as this will trickle down to affecting all LTS users too once the kernel is promoted from -edge to normal HWE.
Created two VMs, one using impish, one using focal. Install the same LXD in both of them, same config for LXD and identical Ubuntu 20.04 container.
Then confirmed that LXD is configuring AppArmor in an identical way:
root@impish:~# sha256sum /var/snap/ lxd/common/ lxd/security/ apparmor/ profiles/ lxd-u1 0d1071b0dc36dcb d5e4be989fe3a77 14dea30b7d2f155 c296 /var/snap/ lxd/common/ lxd/security/ apparmor/ profiles/ lxd-u1
348610816fa5293
root@focal:~# sha256sum /var/snap/ lxd/common/ lxd/security/ apparmor/ profiles/ lxd-u1 0d1071b0dc36dcb d5e4be989fe3a77 14dea30b7d2f155 c296 /var/snap/ lxd/common/ lxd/security/ apparmor/ profiles/ lxd-u1
348610816fa5293
So AppArmor setup from LXD is bit for bit identical. Now checking that AppArmor namespacing and stacking is setup the same way:
root@impish:~# lxc info u1 | grep PID attr/current </var/snap/ lxd/common/ lxd>//& :lxd-u1_ <var-snap- lxd-common- lxd>:unconfined (enforce)
PID: 2852
root@impish:~# cat /proc/2852/
lxd-u1_
root@focal:~# lxc info u1 | grep PID attr/current </var/snap/ lxd/common/ lxd>//& :lxd-u1_ <var-snap- lxd-common- lxd>:unconfined (enforce)
PID: 14047
root@focal:~# cat /proc/14047/
lxd-u1_
So hopefully, that helps prove that LXD is behaving identically on both systems...
I've done the same checks of AppArmor profile hashes for the snap-store-proxy snap in both environments and they're similarly identical. Though snap-store-proxy is obviously quite happy in the 20.04 on 20.04 environment.
I then moved the 20.04 system to the same kernel as the 21.10 one (using linux-generic- hwe-20. 04-edge) and I'm getting a similarly broken behavior for that snap.
LXD's own profile doesn't restrict the capabilities that containers can use, especially now "chown". So to me this suggests an apparmor kernel issue.
My suspicion is that it's either an apparmor change which somehow causes more strict confinement of profiles loaded in containers than loaded on the host (we've seen that in the past with apparmor in a container needing a rule for the binary being confined whereas it's unneeded on the host), or it's a kernel change (possibly the idmap mounts feature?) which is confusing apparmor somehow.
I'm thinking of that last one because of the fact that "chown" is showed repeatedly as blocked by apparmor and given we have a VFS layer that remaps uid/gid on the fly now, maybe that's causing some confusion?
In any case, adding a task for the kernel as this will trickle down to affecting all LTS users too once the kernel is promoted from -edge to normal HWE.