[lucid] 0.6.5 cannot umount /var properly if it is on a separate partition - container does not start

[Roman Yepishev]

=========================== SRU request description =================================

IMPACT: users with separate /var partition cannot start containers

HOW ADDRESSED: the fix was applied upstream some time ago, and is
rolled into the maverick package. The proposed package attached to this
bug cherrypicks the specific patch.

PATCH: a debdiff is attached below (comment #23)

TEST CASE: Install a system with a separate /var partition, and try to
start a container.

REGRESSION POTENTIAL: The patch is only to lxc code, so no non-lxc
code should be affected. As for lxc itself, this simply makes it do
'umount -l' instead of 'umount', so lxc should not regress either.

============================= Original description =============================

Binary package hint: lxc

I have /var on a separate partition.

As described in lxc-users, this prevents containers from startup:

$ lxc-start --name lemon -l DEBUG -o $(tty)
...
lxc-start [timestamp] ERROR lxc_conf - Device or resource busy - could not unmount old rootfs

The post http://<email address hidden>/msg00032.html contains the patch for this problem.
This patch is already included in git.

Attaching this patch here since the error message does not really have any clue about what to do.

Installed: 0.6.5-1
Linux fridge 2.6.32-21-server #32-Ubuntu SMP Fri Apr 16 09:17:34 UTC 2010 x86_64 GNU/Linux

[Roman Yepishev (rye)]

I have packaged the fix for 0.6.5-1 in ppa:~rye/ppa: 0.6.5-1ubuntu1~ppa1~lucid

[Roman Yepishev (rye)]

Correction - the URL for ppa is: ppa:rye/ppa
See https://launchpad.net/~rye/+archive/ppa for more info.

[Петр Слепцов (demiurg)]

The package from your repository did not help :(

System: Ubuntu 10.04 LTS

$ sudo lxc-start -n lxc_web -f config -l DEBUG -o $(tty)
      lxc-start 1272588706.041 INFO lxc_conf - tty's configured
      lxc-start 1272588706.041 DEBUG lxc_start - sigchild handler set
      lxc-start 1272588706.041 INFO lxc_start - 'lxc_web' is initialized
      lxc-start 1272588706.078 DEBUG lxc_cgroup - using cgroup mounted at '/cgroup'
      lxc-start 1272588706.078 DEBUG lxc_cgroup - '/cgroup/15060' renamed to '/cgroup/lxc_web'
      lxc-start 1272588706.079 DEBUG lxc_cgroup - using cgroup mounted at '/cgroup'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.deny' set to 'a'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 1:3 rwm'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 1:5 rwm'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 5:1 rwm'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 5:0 rwm'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 4:0 rwm'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 4:1 rwm'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 1:9 rwm'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 1:8 rwm'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 136:* rwm'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 5:2 rwm'
      lxc-start 1272588706.079 DEBUG lxc_conf - cgroup 'devices.allow' set to 'c 254:0 rwm'
      lxc-start 1272588706.079 INFO lxc_conf - cgroup has been setup
      lxc-start 1272588706.079 INFO lxc_conf - mount points have been setup
      lxc-start 1272588706.079 INFO lxc_conf - console '/dev/pts/0' mounted to '/virt/lxc/guests/web/rootfs/dev/console'
      lxc-start 1272588706.079 INFO lxc_conf - 4 tty(s) has been setup
      lxc-start 1272588706.080 DEBUG lxc_conf - temporary mountpoint for old rootfs is './lxc-oldrootfs-HAwj5l'
      lxc-start 1272588706.080 DEBUG lxc_conf - pivot_root syscall to '/lxc-oldrootfs-HAwj5l' successful
      lxc-start 1272588706.590 DEBUG lxc_conf - umounted '/lxc-oldrootfs-HAwj5l/dev/pts'
      lxc-start 1272588706.798 DEBUG lxc_conf - umounted '/lxc-oldrootfs-HAwj5l/dev/shm'
      lxc-start 1272588707.214 DEBUG lxc_conf - umounted '/lxc-oldrootfs-HAwj5l/sys/fs/fuse/connections'
      lxc-start 1272588707.422 DEBUG lxc_conf - umounted '/lxc-oldrootfs-HAwj5l/sys/kernel/debug'
      lxc-start 1272588707.630 DEBUG lxc_conf - umounted '/lxc-oldrootfs-HAwj5l/sys/kernel/security'
      lxc-start 1272588707.838 DEBUG lxc_conf - umounted '/lxc-oldrootfs-HAwj5l/proc'
      lxc-start 1272588708.046 DEBUG lxc_conf - umounted '/lxc-oldrootfs-HAwj5l/var/run'
      lxc-start 1272588708.254 DEBUG lxc_conf - umounted '/lxc-oldrootfs-HAwj5l/var/lock'
      lxc-start 1272588708.462 DEBUG lxc_conf - umounted '/lxc-oldrootfs-HAwj5l/lib/init/rw'
      lxc-start 1272588...

[Serge Hallyn (serge-hallyn)]

Can you confirm whether this is still a problem on lucid?

[Victor Vargas (kamus)]

This issue apparently is solved in lxc 0.7.1 included in development version of Ubuntu (Maverick Meerkat) , so please can you check if this problem is still affecting you in that version? Thanks

[Andrew Kolchoogin (sallevan)]

Yes, the problem is solved in 0.7 branch of lxc.

There's a patch available and OpenSUSE RPM exists that doesn't contain the problem being discussed: http://<email address hidden>/msg00266.html

[Hugin (g-schwartzmann)]

This bug affect my system too :
Ubuntu 10.04 LTS

# lxc-version
lxc version: 0.6.5

# lxc-start -n web
lxc-start: Device or resource busy - could not unmount old rootfs
lxc-start: failed to pivot_root to '/opt/lxc/web/rootfs'

My containers is on another device too.
Cordially

[Serge Hallyn (serge-hallyn)]

Thanks, Hugin, for the reminder. I've uploaded a version of 0.6.5
to my private repo at https://launchpad.net/~serge-hallyn/+archive/lxc-lucid-lazyumount.
It'll likely take a few hours to build, but then you can

   sudo add-apt-repository ppa:serge-hallyn/lxc-lucid-lazyumount
   sudo apt-get update
   sudo apt-get upgrade

Please let me know if this package fixes your problem. It has the
lazy umount patch which IIUC should do the trick.

[Serge Hallyn (serge-hallyn)]

(if this package is confirmed to fix the bug, I'll try to get an SRU
to get it into lucid-proposed)

[Hugin (g-schwartzmann)]

I'm really interested in testing the package.
Actually the package failed to build.

[Serge Hallyn (serge-hallyn)]

Quoting Hugin (<email address hidden>):
> I'm really interested in testing the package.
> Actually the package failed to build.

Grr, thanks - I'll fix that up later this morning, build
locallly, and upload to this report.

[Serge Hallyn (serge-hallyn)]

Locally compiled .deb of the proposed fix.

[Serge Hallyn (serge-hallyn)]

Sorry, Hugin, I had failed to include quilt in the new build-depends. I've
re-uploaded with that fixed, and the result should be in my ppa in a few
hours. In the meantime, the locally built version (which succeeded since
quilt was already installed here) is uploaded in comment 13. Please let
me know if that package fixes the bug for you.

[Hugin (g-schwartzmann)]

> Sorry, Hugin, I had failed to include
> quilt in the new build-depends.  I've
> re-uploaded with that fixed, and the result should be in my
> ppa in a few
> hours.  In the meantime, the locally built version
> (which succeeded since
> quilt was already installed here) is uploaded in comment
> 13.  Please let
> me know if that package fixes the bug for you.

I can't test for this architecture (64), sorry

[Serge Hallyn (serge-hallyn)]

Hugin: here is the i386 version.

[Hugin (g-schwartzmann)]

Thanks for the package.
The package work correctly but don't fixe the bug :-/
I'm sorry !

[Serge Hallyn (serge-hallyn)]

Hugin - that's surprising! Thanks for testing. Back to the original
email thread...

[Serge Hallyn (serge-hallyn)]

Hugin,

that was my complete lack of debian package handling skills at fault -
the patch was not being applied! Please try this latest one that I
just uploaded (lxc_0.6.5-1ubuntu3_i386.deb).

[Hugin (g-schwartzmann)]

\o/ this package fixe the bug.
Thanks for your work.

[R G (netritious)]

Is the patch applied to the amd64 .deb attached in post #13? Or another case of post #20? :)

[Serge Hallyn (serge-hallyn)]

Quoting Rich Gray (<email address hidden>):
> Is the patch applied to the amd64 .deb attached in post #13? Or another
> case of post #20? :)

The amd64 .deb in post #13 was wrong - the patch wasn't being applied.
The one in post 20 was confirmed to fix the bug. The debdiff is in
comment #23 for trivial recreation of amd64 .deb.

Odd, this seems to have fallen off the radar. It should be proposed
for SRU. I'll start the necessary steps in the morning, or will be happy
if you'd like to do it sooner.

[R G (netritious)]

Thanks for the swift reply Serge. Sadly I've never applied a patch, at least not without instruction. How exactly is the patch applied? Should I apply it to the .deb from the repos or the attached amd64 .deb?

[R G (netritious)]

Patch applied using Ubuntu wiki and forum how to's. Installs ok. Testing for the bug now. Any feedback for a noob packager would be appreciated!

[R G (netritious)]

Do not use the 'amd64 deb of proposed fix' (#27). I was to hasty and should have known patching/packaging could not be that easy. I left the attachment in the event someone can tell me what I'm failing to do properly. I used 'dpkg -i lxc_0.6.5-1_amd64.deb' to install which might have not been the best idea.

[Serge Hallyn (serge-hallyn)]

Thanks Rich - I went ahead and proposed the fix for merge into lucid-proposed.

[Nigel Babu (nigelbabu)]

Please follow the full list of SRU procedures documented at https://wiki.ubuntu.com/StableReleaseUpdates#Procedure, specfically, please provide a debdiff and subscribe sponsors first.

[Serge Hallyn (serge-hallyn)]

@Nigel - thanks, but near as I can tell all that was done on August 30. Likely I goofed up and
missed some step. Should I have placed *this* bug in FixReleased state? (I thought that was
about the development release version, which is N/A here)

Thanks for any hints.

[Mathias Gug (mathiaz)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. However, I am closing it because the bug has been fixed in the latest development version of Ubuntu - Maverick Meerkat.

[Serge Hallyn (serge-hallyn)]

(Marking invalid as lxc on lucid is officially unsupported, and network namespaces are not supported in the kernel :( )