juju-local LXC containers hang due to AppArmor denial of rpc_pipefs mount with local charms

[Charles Butler]

Ran into an issue with the local provider today that seems to be 'newish' behavior.

steps to reproduce:
juju bootstrap
juju deploy cs:trusty/ubuntu
juju deploy --repository=../ local:trusty/wordpress
juju deploy cs:trusty/mysql

Important to note - the race here, if mysql and ubuntu register the containers before the wordpress unit attempts to come online, things deploy as expected without wordpress coming online.

The specifics of the wordpress unit preventing from starting were obtained with dmesg:

[13804.451667] type=1400 audit(1400090535.864:127): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/rpc_pipefs/" pid=13800 comm="mount" fstype="rpc_pipefs" srcname="rpc_pipefs" flags="rw"

Adding :
  mount fstype=rpc_pipefs,

to /etc/apparmor.d/abstractions/lxc/container-base appears to have alleviated the behavior.

Additional details:
+++-===================-==============-==============-============================================
ii juju-local 1.19.2-0ubuntu all dependency package for the Juju local provid
ii juju 1.19.2-0ubuntu all next generation service orchestration system

this is on 14.04

[Tim Penhey (thumper)]

Charles, what is the wordpress charm doing that is triggering this? Do you know?

[Tyler Hicks (tyhicks)]

Would it be possible to attach your local wordpress charm?

[Tyler Hicks (tyhicks)]

I've marked this bug as affecting lxc, since the fix/workaround that Charles and I came up with involves modifying <abstractions/lxc/container-base>.

[Serge Hallyn (serge-hallyn)]

Marking this bug as valid against lxc and invalid against juju-core suggests that you think it is valid to have containers allow this mount by default. Is that the case?

[Curtis Hovey (sinzui)]

I do think it is fine for the mount be allowed.
By Invalid, I mean there is no change we can make to the juju-core code to solve this issue. If there is work for the juju-core developers, then I will change the status for juju-core to triages and get it scheduled to be fixed in time for the trusty fix.

[Serge Hallyn (serge-hallyn)]

Sorry, I was asking Tyler for a position as a security team member. If he
doesn't know offhand then I'll go look at the implementation, but I'm not
familiar with it myself.

[Tyler Hicks (tyhicks)]

Hi Serge - I'm still wanting a little more information. I tried to reproduce the bug myself and can't hit the AppArmor denial. I assume that it must be specific to Charles' local trusty/wordpress charm.

Charles and/or Curtis, can you explain what change occurred in juju-core that has caused the need to mount rpc_pipefs filesystems inside the container?

Serge, as far as allowing rpc_pipefs inside the container, I don't know how safe that would be off the top of my head. I looked at the other filesystems that are allowed by the container-base abstraction and was surprised to see debugfs was allowed. I can't imagine that allowing rpc_pipefs could be more dangerous that debugfs, but that also doesn't mean that we should allow rpc_pipefs. I need to spend some time today understanding more about rpc_pipefs.

[Serge Hallyn (serge-hallyn)]

Good point about debugfs.

I wonder if we should drop that. I find it hard to believe there are
container workloads which need that.

[Stéphane Graber (stgraber)]

wasn't debugfs allowed only because mountall required it?

I thought we allowed it and then had apparmor restrict where it can be mounted and then block any actual access to it (as we've been doing with any fs that's required by mountall).

[Serge Hallyn (serge-hallyn)]

Thank you, yes. We only allow it to be mounted under
/sys/fs/debugfs, and do not allow writes under that. phew.

[Stéphane Graber (stgraber)]

Been incomplete for years, closing.