After update, lxc does not start

[Matias Bordese]

After getting the latest updates, a previously working lucid container is failing to start:

sudo lxc-start -n u1-server
lxc-start: failed to mount rootfs
lxc-start: failed to setup rootfs for 'u1-server'
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'u1-server'
lxc-start: Device or resource busy - failed to remove cgroup '/mnt/cgroup//lxc/u1-server'

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: lxc 0.7.5-3ubuntu32
ProcVersionSignature: Ubuntu 3.2.0-18.28-generic 3.2.9
Uname: Linux 3.2.0-18-generic i686
ApportVersion: 1.94-0ubuntu1
Architecture: i386
Date: Mon Mar 5 21:07:22 2012
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Alpha i386 (20111129.1)
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: lxc
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[John O'Brien (jdobrien)]

This is what I'm seeing with dmesg:

[16241.285998] type=1400 audit(1331004691.503:33): apparmor="DENIED" operation="mount" parent=9376 profile="/usr/bin/lxc-start" name="/usr/lib/lxc/root/" pid=9387 comm="lxc-start" src_name="/var/lib/lxc/u1-server/rootfs/" flags="rw, rbind

[urusha (urusha)]

Seems it's a mistake in /etc/apparmor.d/usr.bin.lxc-start
Don't know how to fix it, but if you want to make lxc work quickly (without apparmor):
ln -s /etc/apparmor.d/usr.bin.lxc-start /etc/apparmor.d/disable/usr.bin.lxc-start
service apparmor restart
lxc-start ......

[Serge Hallyn (serge-hallyn)]

@stgraber,

I hope you don't mind I've assigned this to you, as you were looking at it yesterday, and I'm out until next monday.

I marked it high priority, because it will hit a lot of people. But on the other hand there *is* a workaround, so I guess the priority should be dropped... But I'll leave that to you.

If you definately do not have time for this, please feel free to assign it to me and push a package with the apparmor policy temporarily disabled.

Thanks!

[Stéphane Graber (stgraber)]

I'm happy to update our apparmor profile as soon as apparmor stops crashing :)

For now, the only way I found to get containers working again is to turn off the apparmor profile.

[Serge Hallyn (serge-hallyn)]

Quoting Stéphane Graber (<email address hidden>):
> I'm happy to update our apparmor profile as soon as apparmor stops
> crashing :)
>
> For now, the only way I found to get containers working again is to turn
> off the apparmor profile.

Yikes.

It sounds like temporarily disabling the apparmor profile (in the
package) is the way to go.

thanks,
-serge

[Stéphane Graber (stgraber)]

I pushed a minimal change to LXC disabling the apparmor profile for now.
Instead of removing the profile or using aa-disable I simply changed the path to /usr/bin/lxc-start to /usr/bin/lxc-start.disabled in the profile, whenever apparmor is fixed we'll just need to add the mount statements, bump the apparmor dependency and revert that one line change.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 0.7.5-3ubuntu33

---------------
lxc (0.7.5-3ubuntu33) precise; urgency=low

  * Update apparmor profile to temporarily disable it.
    This will be reverted once apparmor has been fixed. (LP: #947617)
 -- Stephane Graber <email address hidden> Tue, 06 Mar 2012 12:25:21 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package apparmor - 2.7.100-0ubuntu1

---------------
apparmor (2.7.100-0ubuntu1) precise; urgency=low

  * New upstream bug fix release which fixes (in addition to other bugs):
    - LP: #940362
    - LP: #947617
    - LP: #949891
  * Drop the following patches, included upstream:
    - 0004-lp918879.patch
    - 0007-lp941506.patch
    - 0008-lp941503.patch
    - 0009-lp943161.patch
  * Drop the following patch, no longer required:
    - 0005-disable-minimization.patch
  * Rename 0006-lp941808.patch 0004-lp941808.patch
  * debian/patches/0001-add-chromium-browser.patch: update for additional
    denials with newer chromium-browser. (LP: #937723)
  * debian/put-all-profiles-in-complain-mode.sh: deal with existing flags
 -- Jamie Strandboge <email address hidden> Fri, 09 Mar 2012 06:56:48 -0600

[Jan Evert van Grootheest (j-e-van-grootheest)]

For me this seems not fixed.
ii apparmor 2.7.102-0ubuntu3 User-space parser utility for AppArmor
ii lxc 0.7.5-3ubuntu56 Linux containers userspace tools

root@kira:~# lxc-start -n jake
lxc-start: failed to mount rootfs
lxc-start: failed to setup rootfs for 'jake'
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'jake'

And in dmesg
[92690.144338] type=1400 audit(1339010096.655:40): apparmor="DENIED" operation="mount" info="failed type match" error=-13 parent=3406 profile="/usr/bin/lxc-start" name="/var/tmp/lxc/jake/" pid=3429 comm="lxc-start" srcname="/var/lib/lxc/jake/rootfs/" flags="rw, rbind"

I have not yet tried the workaround from comment 4, but will try that shortly.

[Jan Evert van Grootheest (j-e-van-grootheest)]

After the workaround, my container is running.

[Serge Hallyn (serge-hallyn)]

@janevert,

it looks like you have some custom mounting going on. Making a custom profile would be the best way around it, otherwise disabling apparmor as you've done obviously works too.

The ubuntu server guide (for 12.04) lxc section shows how to create and use a custom profile. It also might be worth doing an askubuntu question to guide more people to the answers.

[Jan Evert van Grootheest (j-e-van-grootheest)]

Serge,
This container I created with oneiric with the lxc-sshd template. It worked there reasonably well (only needed to add a default route, which is missing).
I've compared (visually) what oneiric created for mounting and what precise would have created. It seems there is only 1 difference. Oneiric did not include a mount for /proc, which precise does add.

To me this looks like a regression after upgrading from oneiric.

[Serge Hallyn (serge-hallyn)]

@janevert,

yes that should be fixed. I've opened bug 1010598 to track that. Thanks.

[Alan Boudreault (aboudreault)]

Not sure if my issue is related to this. I migrate a container to a new machine. I have been able to start it... but inside it, I can't use pbuilder, which is used to create multiple environments to build debian packages. Getting this message:
aboudreault@packages:~$ pbuilder-dist precise amd64 login
I: Building the build Environment
I: extracting base tarball [/mnt/pbuilder-dist/precise-amd64-base.tgz]
I: creating local configuration
I: copying local configuration
I: Installing apt-lines
I: mounting /proc filesystem
mount: block device /proc is write-protected, mounting read-only
mount: cannot mount block device /proc read-only

Tried the workaround with no luck.

[Serge Hallyn (serge-hallyn)]

@Alan,

your container is not allowed to mount /proc because of the apparmor profile. The easiest way around this is to disable apparmor for that container, by edigint /var/lib/lxc/(containername)/config and uncommenting the line:

#lxc.aa_profile = unconfined

[Alan Boudreault (aboudreault)]

@Serge, thanks a lot it worked!

[Zarrar (zarrar-yousaf)]

Hi,

I am still having problems with the containers. Although i am able to launch and instantiate a container when i disable the apparmor, however the new instantiated container is missing the /proc filesystem as a result I am unable to do any meaningful operations (e.g., ping opeation).

I am running debian versoin 3.2.0-41, whereas i have absolutely no problems when i instantiate and use containers in debian versoin 3.0.0-31

Any help would be appreciated.

Zarrar

[Manisha Luthra (manisha.luthra)]

Hi,
I face the same issue
lxc-start: conf.c: setup_rootfs: 1279 Permission denied - Failed to make / rslave
lxc-start: conf.c: do_rootfs_setup: 3801 failed to setup rootfs for 'left'
lxc-start: conf.c: lxc_setup: 3883 Error setting up rootfs mount after spawn
lxc-start: start.c: do_start: 731 failed to setup the container
lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-start: start.c: __lxc_start: 1213 failed to spawn 'left'
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.

even disabling the apparmor profile doesnt help. Also, I need apparmor anyways because I want to communicate between multiple containers.

Issue is will the following kernel:
Linux ubuntu 4.2.0-16-generic

[Manisha Luthra (manisha.luthra)]

apparmor version: AppArmor 2.10
lxc version: Version 1.1.5