lxc-start fails with 1.1.5-0ubuntu1

[Colin Watson]

After upgrading to lxc 1.1.5-0ubuntu1, lxc-start fails like this:

lxc-start: start.c: preserve_ns: 149 Permission denied - failed to open '/proc/7170/ns/mnt'
lxc-start: start.c: lxc_spawn: 993 failed to store namespace references
lxc-start: start.c: __lxc_start: 1192 failed to spawn 'trusty-lpdev'
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.

This is with a trusty system container. precise system containers behave similarly. I don't have others to try. Downgrading liblxc1, lxc, lxc-templates, and python3-lxc to version 1.1.4-0ubuntu3 causes lxc-start to work again.

[Serge Hallyn (serge-hallyn)]

Could you post the container configuration file?

[Colin Watson (cjwatson)]

But I should also note that the same thing happened with a freshly-created container, using "sudo lxc-create -n trusty-test -t ubuntu -- -r trusty". So I don't think my custom apparmor hack there can be at fault.

[Stéphane Graber (stgraber)]

Hi Colin,

Could you:
 - Tell us what architecture you're running on (noticed the container config shows i386)
 - Confirm that both lxc and liblxc1 are at the same version
 - Tell us the exact kernel version you're running
 - Look for any apparmor denials in dmesg

Thanks!

[Colin Watson (cjwatson)]

My host is xenial amd64; the guest is i386 because a million lines of Python is rather more memory-efficient that way. :-)

Yes, lxc and liblxc1 are at the same version.

Kernel is linux-image-4.2.0-17-generic 4.2.0-17.21, though this also happened with 4.2.0-16.19 (I rebooted to see if that cleared it up).

There are no apparmor denials in dmesg.

[Serge Hallyn (serge-hallyn)]

What does

ls -l /proc/sys/kernel/yama/
cat /proc/sys/kernel/yama/ptrace_scope

show?

[Serge Hallyn (serge-hallyn)]

Nm, afaict yama should never prevent this.

[Stéphane Graber (stgraber)]

This bug was caused by a silent apparmor denial due to a local modification of /etc/apparmor.d/usr.bin.lxc-start (dropped attach_disconnected flag).

[Francis (francisd)]

Mmm I'm having the same problem with lxc 1.1.5 on ubuntu 12.04 (using the ppa). I had to downgrade to 1.1.4 to make my containers bootable again.

I don't think I manually modified the file /etc/apparmor.d/usr.bin.lxc-start. It contains:

#include <tunables/global>

/usr/bin/lxc-start flags=(complain) {
  #include <abstractions/lxc/start-container>
}

I'm using LXC on two ubuntu 12.04 box and only one is failing with lxc-1.1.5.

[Stéphane Graber (stgraber)]

That most definitely is not our default profile, so you or some script you ran modified it.

You may want to install "debsums" and run "debsums -sa lxc" to get a list of configuration files which have been modified on your system.

[Francis (francisd)]

debsums: changed file /etc/default/lxc (from lxc package)
debsums: changed file /etc/apparmor.d/usr.bin.lxc-start (from lxc package)

So I restored the /etc/apparmor.d/usr.bin.lxc-start file to:

#include <tunables/global>

/usr/bin/lxc-start flags=(attach_disconnected) {
  #include <abstractions/lxc/start-container>
}

The only thing I remember I did is running aa-complain /usr/bin/lxc-start a while ago (I don't remember why) and this is what caused the file to be modified. I reverted aa-complain with aa-enforce /usr/bin/lxc-start, I restored the apparmor profil and redid the aa-complain and the file changed again to:

#include <tunables/global>

/usr/bin/lxc-start flags=(complain) {
  #include <abstractions/lxc/start-container>
}

[Stéphane Graber (stgraber)]

Adding an apparmor task as the apparmor tools really shouldn't mangle our profiles in a way that can't be reverted...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Serge Hallyn (serge-hallyn)]

Ah, interesting. So this may be a bug in aa-complain, as it did not
retain (iiuc) the attach_disconnected flags.

[Christian Boltz (cboltz)]

Which apparmor version do you use?

I can't reproduce the problem using the latest version from the 2.9 branch and bzr trunk. (I tested with abstractions/base instead of abstractions/lxc/start-container in the profile, but that shouldn't matter.)

[Jean-Baptiste Lallement (jibel)]

It is also affecting devel-proposed phone images cf bug 1516971

[Jean-Baptiste Lallement (jibel)]

I confirm that the phone boots successfully after downgrading liblxc1, lxc and python3-lxc to version 1.1.4-0ubuntu3

[Francis (francisd)]

apparmor 2.7.102-0ubuntu3.10 running Ubuntu 12.04 LTS.

[Stéphane Graber (stgraber)]

Oh, the phone problem is different I think, there the problem is that your kernel is pretty old and doesn't have the right /proc/self/ns files available. Looks like that's an actual bug with the new implementation. I'll report it upstream.

Jean-Baptiste, can you un-duplicate the phone bug and open a lxc task for it? It's a different issue from what was reported in this bug report.

[Stéphane Graber (stgraber)]

Note that Jean-Baptiste's problem will only be seen on kernels higher than 3.2 and lower than 3.8. So it's not affecting any official distro kernel but it does affect those outdated android kernels.

[Christian Boltz (cboltz)]

AppArmor 2.7 is _very_ old - especially given the fact that the tools were rewritten in python for 2.9.

I just checked the perl code (which was used in 2.8.x and older) - it _sets_ the flags (instead of adding or removing them), so it's not surprising that attach_disconnected gets lost. (This is one of the fixes that went into the 2.9 during the rewrite to python.)

If someone is interested to fix this - the code is in Immunix/AppArmor.pm, sub complain() and sub enforce(), which both call setprofileflags().