Semaphores cannot be created in lxc container

Reported by Gary Poster on 2012-04-05
24
This bug affects 9 people
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Unassigned
lxc (Ubuntu)
High
Stéphane Graber
Precise
High
Stéphane Graber
Quantal
High
Stéphane Graber
sysvinit (Debian)
Fix Released
Unknown
sysvinit (Ubuntu)
High
Stéphane Graber
Precise
High
Stéphane Graber
Quantal
High
Stéphane Graber

Bug Description

[rational]
Current initscripts doesn't deal properly with chroots, bind-mounting /run/shm to /dev/shm instead of making the later a symlink to the former.
LXC has been carrying a workaround for that, sadly the workaround stopped working late in the 12.04 cycle, meaning that an extra chunk of code had to be added to properly fix any broken container before applying the upgrade as any initscripts update would otherwise fail.

[test case]
Two things to test:
1) lxc-create -t ubuntu -n p1
2) Check that /dev/shm is a symlink to /run/shm in the container

1) Get an existing container
2) Update to -proposed
3) Check that /dev/shm is now a symlink to /run/shm

[regression potential]
There's a known risk for applications using shm during the upgrade as the entries will be moved to another file system. As long as all the processes accessing the file keep running, it should be fine, but it won't if another one is spawned as it will try reading it from the new location. A warning is displayed recommending a reboot.

The rest of the code was already tested with all known scenario and should be sufficiently guarded to only apply to the affected systems.

>>> from multiprocessing import synchronize
>>> synchronize.Lock()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 117, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 49, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented

Thanks to hallyn, there's a workaround and a fix: adding this line to the container's fstab will fix the problem:
none dev/shm tmpfs defaults 0 0

He is updating the ubuntu template with this change.

Original bug report description was Launchpad related (celery tests fail on parallel testing instances)

Gary Poster (gary) wrote :

This is easily and repeatably duplicatable on in my lxc lucid container. The error for the last test (with the multiprocessing SemLock OSError seems the most likely to be a shared root cause to me. Interestingly, the OSError was slightly different for me, with a newer lxc: "OSError: [Errno 38] Function not implemented". I checked the syslog and some random other log files, without really knowing where to look, and saw nothing of interest.

As you'd expect from the traceback, you can repeat this error with a single Python call to the standard library.

>>> from multiprocessing import synchronize
>>> synchronize.Lock()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 117, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/usr/lib/python2.6/multiprocessing/synchronize.py", line 49, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented

38 is ENOSYS

I talked with hallyn and this is an lxc issue. Adding this line to the container's fstab will fix the problem:
none dev/shm tmpfs defaults 0 0

He is updating the ubuntu template with this change.

Changed in launchpad:
status: Triaged → Invalid
summary: - celery tests fail on parallel testing instances
+ Semaphores cannot be created in lxc container
description: updated
James Page (james-page) on 2012-04-12
Changed in lxc (Ubuntu):
importance: Undecided → High
assignee: nobody → Serge Hallyn (serge-hallyn)
milestone: none → ubuntu-12.04
status: New → Confirmed
Serge Hallyn (serge-hallyn) wrote :

There are a few parts to this. At core, we need /dev/shm to be a symbolic link to /dev/shm, which *is* mounted.

/etc/init/mounted-dev.conf is supposed to create that symbolic link. However, it does not run because we never mount /dev.

If it did run, it would do the wrong thing. That is because the initscripts postinst leaves a /dev/shm directory. mounted-dev.conf simply does

[ -e /dev/shm ] || ln -s /run/shm /dev/shm

That would cause /dev/shm/shm to be the symbolic link to /run/shm.

The simplest solution is probably to rmdir $rootfs/dev/shm at the end of the container creation templates.

Serge Hallyn (serge-hallyn) wrote :

Note, the fix will only affect new containers.

The WORKAROUND for existing containers is to manually rmdir /var/lib/lxc/{container}/rootfs/dev/shm

Serge Hallyn (serge-hallyn) wrote :

For P we are working around this in lxc. No change is needed in P in initscripts.

For Q and beyond, initscripts should be fixed. If running from debootstrap, /dev/shm should be created as a symlink. That is not happening. Note that a system which mounts /dev as devtmpfs (or tmpfs) will do the right thing on reboot. But a system which has /dev on rootfs is stuck with /dev/shm as whatever initscripts' postinst left it as.

Changed in sysvinit (Ubuntu):
status: New → Confirmed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 0.7.5-3ubuntu51

---------------
lxc (0.7.5-3ubuntu51) precise; urgency=low

  * 0070-templates-rmdir-dev-shm: in precise containers, rmdir $rootfs/dev/shm
    and and create it as a symbolic link to /run/shm. (LP: #974584)
 -- Serge Hallyn <email address hidden> Thu, 12 Apr 2012 09:54:22 -0500

Changed in lxc (Ubuntu):
status: Confirmed → Fix Released
Serge Hallyn (serge-hallyn) wrote :

Sorry, the workaround I listed above was for precise, and actually was incomplete (you also need to ln -s /run/shm $rootfs/dev/shm).

For older releases, the /run transition was not done, so /dev/shm is supposed to be mounted by mountall through /lib/init/fstab. So the workaround is to add a dev/shm entry to /var/lib/lxc/<container>/fstab, or to add it to /var/lib/lxc/<container>/rootfs/lib/init/fstab.lxc.

Ursula Junque (ursinha) on 2012-04-17
Changed in sysvinit (Ubuntu):
importance: Undecided → High
Serge Hallyn (serge-hallyn) wrote :

It would appear part of the problem is simply that ischroot, provided by debianutils, does not work. initscripts' postinst counts on ischroot working.

Serge Hallyn (serge-hallyn) wrote :

This has taken me longer than it should have because I made some bad initial assumptions.

The appearance of a /dev/shm directory was a bit of a false flag. That was happening because the chroot case higher in initscripts.postinst was failing to create the /dev/shm symbolic link in the first place. It's source directory (/run/shm) needed to first be created, and then the arguments to compat_link() were reversed. I assumed they were correct because they were the same for the /var/lock symlink which is correctly created - but in fact the /var/lock symlink exists before that block of code.

Here is a debdiff which appears to fix the /dev/shm symlink creation in a container for me.

Serge Hallyn (serge-hallyn) wrote :

Per irc discussion with slangasek, further patching is needed to make sure that the /run/lock and /run->/var/run update works right in an upgrade with /run transition.

Steve Langasek (vorlon) wrote :

I can confirm Serge's analysis here. I'm surprised that this hasn't bitten us more severely before now, given that I see all my compat symlinks are wrong in my (iteratively upgraded) precise chroots here; apparently things really don't care so much about the compat these days.

We should definitely fix this for .1.

Changed in lxc (Ubuntu Precise):
status: New → Fix Released
Changed in sysvinit (Ubuntu Quantal):
status: Confirmed → Triaged
Changed in sysvinit (Ubuntu Precise):
status: New → Triaged
importance: Undecided → High
milestone: none → ubuntu-12.04.1
tags: added: patch
Serge Hallyn (serge-hallyn) wrote :

(thinking out loud)

my patch works for a fresh debootstrap. It should have zero effect for a machine install, since it is only in chroots. Similarly, containers are treated as hosts so will not be affected when upgraded under lxc.

Cases to still test, then, include an update from oneiric or lucid to precise inside a chroot or schroot, as well as an oneiric container image upgraded under chroot (not inside lxc). Not sure what else.

Serge Hallyn (serge-hallyn) wrote :

My patch is however not right - it somehow leaves /run/shm mounted.

Serge Hallyn (serge-hallyn) wrote :

I wonder if it would be overkill to detect a /dev/shm mount and non-existent /run/shm and, if both are the case, move the mount. Otherwise, throw our hands up and assume user did something weird by hand.

Serge Hallyn (serge-hallyn) wrote :

Ah, a simple 'chroot dpkg -i *.deb' of course fails because ischroot fails with /proc not mounted.

Do we care about that? The fix is simple, but if we don't care about it then I won't add it.

Serge Hallyn (serge-hallyn) wrote :

New debdiff which also makes sure proc is mounted before ischroot is called, so that 'chroot precise-rootfs dpkg -i *.deb' succeeds and does not leave dangling /run/shm mounts.

Serge Hallyn (serge-hallyn) wrote :

The attachment in #15 (sysvinit-shm2.debdiff) is the best fix in my opinion. I don't have upload rights, and this ought to be reviewed anyway. If noone speaks up here reviewing the patch in a few days, I'll ping on irc.

Serge Hallyn (serge-hallyn) wrote :

Doing
   debootstrap sid sid
   chroot sid dpkg -i /var/cache/apt/archives/initscr*.deb

(on a sid host) also leaves a mount on sid/run/shm.

Changed in sysvinit (Debian):
status: Unknown → New
Serge Hallyn (serge-hallyn) wrote :

From the debian bug, here is my new suggestion for a replacement for the ischroot case in initscripts.postinst:

=============================
if ischroot; then
        # Symlink /var/run from /run
        # Note var/run is relative
        if compat_link /var/run /run; then
                # Symlink /var/lock from /run/lock
                # Note that it's really /var/run/lock
                compat_link /var/lock /run/lock

                # Symlink /dev/shm from /run/shm
                # Note that it's really /var/run/shm
  [ ! -d /run/shm ] && mkdir /run/shm
  if [ ! mountpoint -q /dev ]; then
   [ ! -d /dev/shm ] && compat_link /run/shm /dev/shm
  else
   compat_link /dev/shm /run/shm
  fi
        fi
# Host system, not a chroot.
else
=============================

So if the chroot has /dev/ bind-mounted from the host, assume they
always will, and will want /run/shm as a symlink to /dev/shm.

Otherwise if /dev/shm exists, leave it alone, because we're definately
not in debootstrap.

Otherwise, assume /dev and /run won't get cleaned up at 'shutdown'
(chroot exit) so set up *exactly* what we want to see as the end
result: /dev/shm as a symbolic link to /run/shm.

Serge Hallyn (serge-hallyn) wrote :

Here is version tested and working in debootstrap.

diff -u sysvinit-2.88dsf/debian/changelog sysvinit-2.88dsf/debian/changelog
--- sysvinit-2.88dsf/debian/changelog
+++ sysvinit-2.88dsf/debian/changelog
@@ -1,3 +1,12 @@
+sysvinit (2.88dsf-13.10ubuntu12) quantal; urgency=low
+
+ * initscripts.postinst: if /dev is not a separate partition and we're in a
+ chroot, then create /run/shm and make /dev/shm a symbolic link to it, as
+ we would expect to find in a upgraded and rebooted running system.
+ (LP: #974584) (Closes: #674178)
+
+ -- Serge Hallyn <email address hidden> Wed, 30 May 2012 12:17:37 -0500
+
 sysvinit (2.88dsf-13.10ubuntu11) precise; urgency=low

   * Only try to move links in /etc/rc{0,6}.d that match "S0*". LP: #941867.
diff -u sysvinit-2.88dsf/debian/initscripts.postinst sysvinit-2.88dsf/debian/initscripts.postinst
--- sysvinit-2.88dsf/debian/initscripts.postinst
+++ sysvinit-2.88dsf/debian/initscripts.postinst
@@ -255,7 +255,12 @@

   # Symlink /dev/shm from /run/shm
   # Note that it's really /var/run/shm
- compat_link /dev/shm /run/shm
+ if ! mountpoint -q /dev ; then
+ [ ! -d /run/shm ] && mkdir -p /run/shm
+ [ ! -d /dev/shm ] && compat_link /run/shm /dev/shm
+ else
+ compat_link /dev/shm /run/shm
+ fi
  fi
 # Host system, not a chroot.
 else

James Page (james-page) on 2012-06-19
Changed in lxc (Ubuntu Quantal):
milestone: ubuntu-12.04 → quantal-alpha-2
tags: added: rls-q-incoming
Steve Langasek (vorlon) wrote :

I think there are the following possible valid end states that we want to get to at the end of the postinst:

 - we're not in a chroot. /dev/shm is bind mounted to /run/shm, with the reboot script handling the fix-up to make /dev/shm a symlink to /run/shm before next boot.
 - we're in a chroot and /dev is not bind mounted. sysvshm should be mounted *somewhere* within the chroot, but we don't know where this will be done and we're not going to do it ourselves. So dictate that this should be /run/shm, and make /dev/shm a symlink there.
 - we're in a chroot and /dev is bind mounted, and /dev/shm is a symlink to /run/shm (because the parent environment is also a recent Debian/Ubuntu release). Make the /run/shm directory.
 - we're in a chroot, /dev is bind mounted, and /dev/shm is also bind mounted. This is not a good situation to be in, because it means that the future correctness of the chroot is dependent on the setup remaining the same on the host system - which it won't if the host system is an older Debian/Ubuntu release that is later upgraded. But we do the best we can for now, and make /run/shm a symlink to /dev/shm.
 - we're in a chroot, /dev/ is bind mounted, and /dev/shm is a non-bind-mounted directory. We can't fix this, so we should just create the /run/shm directory and leave /dev/shm alone.

Does that make sense?

Steve Langasek (vorlon) wrote :

If it does make sense, the implementation that follows is:

if mountpoint -q /dev && mountpoint -q /dev/shm; then
  compat_link /dev/shm /run/shm
else
  [ -d /run/shm ] || mkdir -p /run/shm
  mountpoint -q /dev/ || compat_link /run/shm /dev/shm
fi

BTW, I realize there's another possible scenario, which is that /dev/shm is a bind mount when /dev itself is not. We should probably handle that the same as if they both are (make /run/shm a symlink to /dev/shm), in which case the first check just needs to be "if mountpoint -q /dev/shm".

tags: removed: rls-q-incoming
Serge Hallyn (serge-hallyn) wrote :

Comment #20 looks very much correct.

The shm_overview(7) man page does say that it always uses a tmpfs filesystem, so does that mean that there is no way for /dev/shm to be a unmounted-over directory holding valuable shm state?

In any case, both your code in comment #21 and the update you propose there would suffice to fix the bug in debootstrap creating images not usable in a chroot/container.

Changed in sysvinit (Debian):
status: New → Fix Released
Serge Hallyn (serge-hallyn) wrote :

As this bug is fixed in debian, should the patch be cherrypicked into ubuntu, or should i open a bug requesting al merge of 2.88dsf-28?

Changed in sysvinit (Ubuntu Precise):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Changed in sysvinit (Ubuntu Quantal):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Changed in sysvinit (Ubuntu Quantal):
assignee: Canonical Foundations Team (canonical-foundations) → Stéphane Graber (stgraber)
Changed in sysvinit (Ubuntu Precise):
assignee: Canonical Foundations Team (canonical-foundations) → Stéphane Graber (stgraber)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sysvinit - 2.88dsf-13.10ubuntu12

---------------
sysvinit (2.88dsf-13.10ubuntu12) quantal; urgency=low

  * Cherry-pick fix from git to better handle chroot in postinst and change
    the check to allow for the case where /dev isn't a mountpoint but /dev/shm
    is one. (LP: #974584)
  * Add logic to detect broken environments (mostly LXC) where both /dev/shm
    and /run/shm exist as directories (and aren't bind-mounts).
    In such case, move the content of /dev/shm to /run/shm and make /dev/shm
    a symlink to /run/shm.
 -- Stephane Graber <email address hidden> Fri, 20 Jul 2012 16:55:04 -0400

Changed in sysvinit (Ubuntu Quantal):
status: Triaged → Fix Released
description: updated
Changed in sysvinit (Ubuntu Precise):
status: Triaged → In Progress
Changed in lxc (Ubuntu Precise):
importance: Undecided → High
status: Fix Released → Triaged
assignee: nobody → Stéphane Graber (stgraber)
Changed in lxc (Ubuntu Quantal):
assignee: Serge Hallyn (serge-hallyn) → Stéphane Graber (stgraber)
status: Fix Released → Triaged
milestone: quantal-alpha-2 → none
Changed in lxc (Ubuntu Precise):
milestone: none → ubuntu-12.04.1
Stéphane Graber (stgraber) wrote :

SRU team: Please wait for matching lxc upload before accepting into -proposed or moving to -updates or creating container will fail because of the broken check in lxc-ubuntu and lxc-ubuntu-cloud.

description: updated

Hello Gary, or anyone else affected,

Accepted sysvinit into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/sysvinit/2.88dsf-13.10ubuntu11.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in sysvinit (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in lxc (Ubuntu Precise):
status: Triaged → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Gary, or anyone else affected,

Accepted lxc into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/lxc/0.7.5-3ubuntu61 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 0.8.0~rc1-4ubuntu21

---------------
lxc (0.8.0~rc1-4ubuntu21) quantal; urgency=low

  [ Stéphane Graber ]
  * Fix lxc-ubuntu and lxc-ubuntu-cloud to fix the /dev/shm workaround to only
    trigger when /dev/shm is not a symlink. (LP: #974584)

  [ Serge Hallyn ]
  * lxc.lxc-net.upstart: replace the check for USE_LXC_BRIDGE (which could be
    changed from true to false after starting lxc-net) with one for the
    existence /var/run/lxc. (LP: #1019290)
  * 0095-lxc-clone-change-uuid-on-xfs.patch: give each cloned xfs-backed
    lvm partition a unique uuid so they can be mounted simultaneously.
    (LP: #1013549)
  * 0096-lxc-wait-add-timeout.patch: patch submitted upstream to add a timeout
    option to lxc-wait. (LP: #1020179)
 -- Serge Hallyn <email address hidden> Thu, 26 Jul 2012 17:40:36 +0000

Changed in lxc (Ubuntu Quantal):
status: Triaged → Fix Released
Stéphane Graber (stgraber) wrote :

Tested before/after for both lxc and sysvinit on precise with precise and quantal containers, everything seems to work as expected.

tags: added: verification-done
removed: verification-needed
Stéphane Graber (stgraber) wrote :

As discussed in #ubuntu-release, a new lxc package was accepted resetting the testing period to 0 today.

As both sysvinit and lxc absolutely NEED to land at the same time for this bug to get fixed without causing any regression, I'd appreciate if whoever copies sysvinit copies lxc as well, regardless of how many days are left for lxc.

lxc_0.7.5-3ubuntu61 was properly tested days ago and I have confirmed that lxc_0.7.5-3ubuntu62 is good too.

So when sysvinit reaches 7 days in -proposed, the matching change in lxc will too and so both should be considered good to go.

Here's the 61 to 62 delta: http://paste.ubuntu.com/1123344/

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 0.7.5-3ubuntu62

---------------
lxc (0.7.5-3ubuntu62) precise-proposed; urgency=low

  * lxc.postrm: support "purge" command (LP: #1029716)
 -- Serge Hallyn <email address hidden> Fri, 27 Jul 2012 03:31:39 +0000

Changed in lxc (Ubuntu Precise):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package sysvinit - 2.88dsf-13.10ubuntu11.1

---------------
sysvinit (2.88dsf-13.10ubuntu11.1) precise-proposed; urgency=low

  * Cherry-pick fix from git to better handle chroot in postinst and change
    the check to allow for the case where /dev isn't a mountpoint but /dev/shm
    is one. (LP: #974584)
  * Add logic to detect broken environments (mostly LXC) where both /dev/shm
    and /run/shm exist as directories (and aren't bind-mounts).
    In such case, move the content of /dev/shm to /run/shm and make /dev/shm
    a symlink to /run/shm.
 -- Stephane Graber <email address hidden> Fri, 20 Jul 2012 16:55:04 -0400

Changed in sysvinit (Ubuntu Precise):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.