No easy way to enable nested container support in juju local environment

[Daniel Manrique]

System information
==================
I tried this on a Trusty host with:

juju, juju-core, juju-local 1.18.4+dfsg-0ubuntu0.14.04.1
lxc 1.0.5-0ubuntu0.1

I'm attaching an easy charm to reproduce this with, but in essence it just
installs lxc inside the container so it can be used, so it's easy to reimplement
to test any theories.

Steps to reproduce
==================
- Put the attached minimal reproduction charm in test-nested-lxc.
- cd test-nested-lxc
- Switch juju to local environment (configuring a local environment is outside the scope of this repro case :)
- juju bootstrap
- juju deploy --repository charms local:trusty/nested-lxc-crasher
- wait until the charm is deployed and started
- juju ssh 1 (or whichever instance the charm was deployed to)
- sudo lxc-create -n crasher -t ubuntu -- -r trusty

Expected result:
- LXC instance created and can be started with e.g. lxc-start or lxc-attach

Actual result:

I: Extracting zlib1g...
W: Failure trying to run: chroot /var/cache/lxc/trusty/partial-amd64 mount -t proc proc /proc
W: See /var/cache/lxc/trusty/partial-amd64/debootstrap/debootstrap.log for details
lxc_container: container creation template for crasher failed
lxc_container: Error creating container crasher

Checking dmesg on the host system shows this, indicating a problem with apparmor access:

[ 1319.248614] type=1400 audit(1412960622.121:29): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/cgmanager/fs/none,name=systemd/" pid=5771 comm="cgmanager" fstype="cgroup" srcname="none,name=systemd" flags="rw"
[ 2197.665717] type=1400 audit(1412961500.537:30): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/var/cache/lxc/trusty/partial-amd64/proc/" pid=11680 comm="mount" fstype="proc" srcname="proc" flags="rw"
[ 2197.665767] type=1400 audit(1412961500.537:31): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/var/cache/lxc/trusty/partial-amd64/proc/" pid=11680 comm="mount" fstype="proc" srcname="proc" flags="ro"

Discussion
==========

The problem here is that the lxc container created first by Juju is not configured for nesting, which would need this in the config file (/var/lib/lxc/blah-local-machine-1/config):

lxc.aa_profile = lxc-container-default-with-nesting
lxc.mount.auto = cgroup:mixed

Workaround
==========

To test the above theory and provide a workaround, I edited the container config file by hand, added those config options, and restarted the container (sudo lxc-stop -n blah-local-machine-1; sudo lxc-start -d -n blah-local-machine-1). I then retried the reproduction case and now I was able to lxc-create and then lxc-start successfully. Here's the output from the juju host showing that nested containers are working:

$ sudo lxc-ls --nesting --fancy
NAME STATE IPV4 IPV6 AUTOSTART
----------------------------------------------------------------------
juju-trusty-template STOPPED - - NO
blah-local-machine-1 RUNNING 10.0.3.232, 10.0.4.1 - YES
 \_ crasher RUNNING 10.0.4.217 - NO

Our request
===========

Since I didn't find a place to tweak Juju-managed containers (I was hoping for a config file or setting of some sort), it would be good to have a way to do this, perhaps with a config setting in the environments.yaml section for the local environment. The versatile way would be a way to append settings to the config file, though this would need more familiarity with LXC. Another option would be a boolean nested-container-support setting that takes care of adding the config for the user. This should be well-documented so people can easily find out and enable this if needed.

Another possibility would be for Juju to create its containers with nesting support by default. I'm not familiar with the security and performance implications of this, and I guess if it's not enabled by default by LXC itself it's because it's better not to use it unless you need it, but this should be researched and considered as a possibility.

Thanks!