unable to launch lxc application containers when dropping cap_sysadmin

[Sebastian Wendland]

========================================
SRU Justification
1. Impact: cannot lxc-execute a container without cap_sys_admin
2. Development fix: don't fail if lxc-init cannot mount /proc
3. Stable fix: same as development fix.
4. Test case:
   sudo lxc-create -t ubuntu-cloud -n c1
   sudo lxc-start -n c1
       (log in)
             sudo apt-get -y install --no-install-recommends lxc
             sudo poweroff
   sudo lxc-execcute -n c1 -s lxc.cap.drop=sys_admin /bin/bash
5. Regression potential: none
========================================

Using the 0.8.0~rc1 lxc release, it was possible to start an application container with the lxc.cap.drop=sys_admin option (# lxc-execute -n foo -s lxc.cap.drop=sys_admin -- /bin/bash). Since the new 1.0.0~alpha1 release, this is not possible anymore; the application immediately crashes upon being called by lxc-init, thus killing the container. When any other capability (or combination of capabilities) is dropped, the container still starts up however, only dropping cap_sys_admin results in an error.

I've attached the debug output of # lxc-execute -o foo -l DEBUG -n foo -s lxc.cap.drop=sys_admin -- /bin/bash for reference.

Release: 12.04.3 with HWE, Kernel 3.8.0-32-generic #47~precise1-Ubuntu SMP Wed Oct 2 16:19:35 UTC 2013 x86_64
LXC version: 1.0.0~alpha1-0ubuntu13~ubuntu12.04.1

[Serge Hallyn (serge-hallyn)]

I can't reproduce this on precise with the ubuntu-lxc daily ppa, on 3.2 kernel.

Could you try installing ppa:ubuntu-lxc/daily and see if that fixes it for you?

It's possible this is a bug in the backport version only, that the newer kernel is doing something unexpected, or that something is wrong in the container itself. I don't see anything in the debug output, unfortunately, to help pin it down.

[Serge Hallyn (serge-hallyn)]

(Marking incomplete pending information; please re-mark as new when answering the above questions, even if only to say you can't test the ppa)

[Serge Hallyn (serge-hallyn)]

Just tried to reproduce it on a precise system with the precise-backports lxc, and still was not able to reproduce it.

Please try creating a new container, and doing 'apt-get -y install --no-install-recommends lxc' in it, then see if you can lxc-execute -s lxc.cap.drop=sys_admin -n newcontainer -- /bin/sh with that container.

[Serge Hallyn (serge-hallyn)]

(Finally, also tried with the 3.8 kernel, still was not able to reproduce)

[Sebastian Wendland (wendland-8)]

I've been trying to narrow it down by running lxc-execute on a few other systems, including a 12.04.3 with the 3.2.0-55-generic Kernel, no success. As you said, the debug output unfortunately does not provide any details.

What I know so far:
* lxc 0.7.5-3 (standard precise version) and 0.8.0~rc1 (backport) work fine on 3.2 and 3.8 kernels
* as soon as I upgrade to 1.0.0~alpha1 and any kernel, it does not work anymore
* upgrading to the latest daily (1.0.0~alpha3+master~20131122-0500-0ubuntu1~ppa1~precise1) does not resolve the issue on 3.2 and 3.8

I also tried to use a custom container (see attached lxc configuration) using # lxc-execute -n foo -f lxc.conf -- /bin/bash, but no luck. Note that I run lxc-execute directly from the command line on the host, not inside a OS container.

FYI: I use lxc to isolate Apache2 instances on my webservers, using complex container setups with my own templates. As I said, this worked perfectly with 0.8.0~rc1 and earlier, and with 1.0.0~alpha1 the containers work as well - except it is impossible to drop cap_sys_admin.

[Serge Hallyn (serge-hallyn)]

Quoting Sebastian Wendland (<email address hidden>):
> I've been trying to narrow it down by running lxc-execute on a few other
> systems, including a 12.04.3 with the 3.2.0-55-generic Kernel, no
> success. As you said, the debug output unfortunately does not provide
> any details.

Could you try installing strace in the container and running bash under
strace -f -o/debug.out ?

[Sebastian Wendland (wendland-8)]

Here is the strace for # strace -f -o/debug.out lxc-execute -o foo -l DEBUG -s lxc.cap.drop=sys_admin -n foo -- /bin/bash using lxc 1.0.0.alpha3 on 3.8.0-33 kernel

[Sebastian Wendland (wendland-8)]

Plus the lxc-execute debug output for the same command

[Serge Hallyn (serge-hallyn)]

I actually meant to run strace under lxc-execute, but this was also
helpful. Could you please add a proc entry to your lxc.mount.entry
lines in the container configuration, and see whether that helps?

[Sebastian Wendland (wendland-8)]

Running strace inside lxc-execute (i.e. lxc-execute -n foo -s lxc.cap.drop=sys_admin -- strace -f -o/root/debug.out /bin/bash) does not work as strace will immediately crash just like bash, thus producing no output.

Here is the lxc log for # lxc-execute -n foo -f lxc.conf -o foo -l DEBUG -- /bin/bash with the lxc config below (again on 3.8 with the daily lxc build).

------------
lxc.utsname = foo

lxc.cap.drop = sys_admin

lxc.tty = 1
lxc.console=/lxc/foo/console

lxc.rootfs = /lxc/foo/rootfs
lxc.mount.entry = /usr usr none ro,bind 0 0
lxc.mount.entry = /lib lib none ro,bind 0 0
lxc.mount.entry = /lib64 lib64 none ro,bind 0 0
lxc.mount.entry = /bin bin none ro,bind 0 0
lxc.mount.entry = /sbin sbin none ro,bind 0 0
lxc.mount.entry = /lxc/dev/null dev/null none bind 0 0
lxc.mount.entry = /lxc/dev/zero dev/zero none bind 0 0
lxc.mount.entry = /lxc/dev/random dev/random none bind 0 0
lxc.mount.entry = /lxc/dev/urandom dev/urandom none bind 0 0
lxc.mount.entry = tmpfs tmp tmpfs rw,size=100M,noexec,nodev,mode=1777 0 0
lxc.mount.entry = proc proc proc nodev,noexec,nosuid 0 0

[Serge Hallyn (serge-hallyn)]

Oh, now I see. lxc-init calls setup_fs() which fails if it cannot mount
proc. (I had looked for direct calls to mount proc but missed the
setup_fs call) Without cap_sys_admin you cannot mount proc. What's
unclear to me now is why this would have worked for you with older
lxc. This is not something that has recently changed, so it should have
always failed.

I think updating lxc-init to only warn if you could not mount /proc
would be good. Will send a patch upstream for that and see if anyone
can think of a good counter argument.

 status: confirmed
 importance: medium

[Serge Hallyn (serge-hallyn)]

Fix pushed to git.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 1.0.0~alpha3-0ubuntu3

---------------
lxc (1.0.0~alpha3-0ubuntu3) trusty; urgency=low

  * d/p/0002-don-t-fail-lxc-init-if-we-couldn-t-mount-proc.patch: fix
    failure to run lxc-init when lxc.cap.drop=sys_admin. (LP: #1253669)
 -- Serge Hallyn <email address hidden> Fri, 22 Nov 2013 15:57:59 -0600

[Sebastian Wendland (wendland-8)]

I just tried the last daily build and the issue is solved. Thanks for the quick help.

FYI: On the earlier versions, lxc complained about not being able to mount proc, but it always continued to run the application anyway so I ignored it. After all, most applications run just fine without proc, plus it makes breaking out of container a bit more challenging.

[Rolf Leggewie (r0lf)]

quantal has seen the end of its life and is no longer receiving any updates. Marking the quantal task for this ticket as "Won't Fix".

[Rolf Leggewie (r0lf)]

raring has seen the end of its life and is no longer receiving any updates. Marking the raring task for this ticket as "Won't Fix".

[Rolf Leggewie (r0lf)]

saucy has seen the end of its life and is no longer receiving any updates. Marking the saucy task for this ticket as "Won't Fix".