systemd 226 (moving pid 1 into /init.scope cgroup) breaks lxc-attach

Bug #1497420 reported by Chris West on 2015-09-18
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cgmanager (Ubuntu)
High
Unassigned
Vivid
Undecided
Unassigned
Wily
Undecided
Unassigned
docker (Debian)
Fix Released
Unknown
lxc (Ubuntu)
High
Unassigned
Vivid
Undecided
Unassigned
Wily
Undecided
Unassigned
lxcfs (Ubuntu)
High
Unassigned
Vivid
Undecided
Unassigned
Wily
Undecided
Unassigned

Bug Description

Once systemd 226 is installed in an unprivileged Debian Sid container, lxc-attach no-longer functions:

% lxc-attach -n siddy
lxc-attach: cgmanager.c: lxc_cgmanager_enter: 698 call to cgmanager_move_pid_abs_sync failed: invalid request
lxc-attach: cgmanager.c: cgm_attach: 1460 Failed to enter group /user.slice/user-1000.slice/session-1.scope/lxc/siddy/init.scope
lxc-attach: attach.c: lxc_attach: 909 error communicating with child process

% journalctl -u cgmanager | tail -n 1
Sep 18 20:20:44 astoria cgmanager[1169]: cgmanager:per_ctrl_move_pid_main: pid 21918 (uid 1000 gid 1000) may not write to /run/cgmanager/fs/none,name=systemd//user.slice/user-1000.slice/session-1.scope/lxc/siddy/init.scope/tasks

Full walkthrough:

% lxc-create -n siddy -t download -- -d debian -r sid -a amd64
Using image from local cache
Unpacking the rootfs
...

% lxc-start -n siddy
% lxc-attach -n siddy

root@siddy:/# apt-get update
Get:1 http://http.debian.net sid InRelease [253 kB]
...
Get:7 http://http.debian.net sid/non-free amd64 Packages [91.3 kB]
Fetched 13.2 MB in 3s (3789 kB/s)
Reading package lists... Done

root@siddy:/# apt-get install systemd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libsystemd0
Suggested packages:
  systemd-ui systemd-container
Recommended packages:
  libpam-systemd dbus
The following packages will be upgraded:
  libsystemd0 systemd
2 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.
Need to get 3668 kB of archives.
After this operation, 1100 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://http.debian.net/debian/ sid/main libsystemd0 amd64 226-2 [242 kB]
Get:2 http://http.debian.net/debian/ sid/main systemd amd64 226-2 [3426 kB]
...
Setting up systemd (226-2) ...
Installing new version of config file /etc/X11/xinit/xinitrc.d/50-systemd-user.sh ...
Installing new version of config file /etc/dbus-1/system.d/org.freedesktop.login1.conf ...
addgroup: The group `systemd-journal' already exists as a system group. Exiting.
Failed to set capabilities on file `/usr/bin/systemd-detect-virt' (Invalid argument)
The value of the capability argument is not permitted for a file. Or the file is not a regular (non-symlink) file
Failed to execute operation: File exists
Removing obsolete conffile /etc/dbus-1/system.d/org.freedesktop.machine1.conf ...

root@siddy:/# exit

% lxc-attach -n siddy
lxc-attach: cgmanager.c: lxc_cgmanager_enter: 698 call to cgmanager_move_pid_abs_sync failed: invalid request
lxc-attach: cgmanager.c: cgm_attach: 1460 Failed to enter group /user.slice/user-1000.slice/session-1.scope/lxc/siddy/init.scope
lxc-attach: attach.c: lxc_attach: 909 error communicating with child process

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: lxc 1.1.3-0ubuntu1 [modified: usr/lib/x86_64-linux-gnu/lxc/lxc-net]
ProcVersionSignature: Ubuntu 4.2.0-10.11-generic 4.2.0
Uname: Linux 4.2.0-10-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.18.1-0ubuntu1
Architecture: amd64
Date: Fri Sep 18 20:19:58 2015
SourcePackage: lxc
UpgradeStatus: Upgraded to wily on 2015-08-28 (20 days ago)
defaults.conf:
 lxc.network.type = veth
 lxc.network.link = lxcbr0
 lxc.network.flags = up
 lxc.network.hwaddr = 00:16:3e:xx:xx:xx

Chris West (faux) wrote :
Changed in systemd (Debian):
status: Unknown → New
Martin Pitt (pitti) on 2015-10-13
summary: - systemd 226-2 (debian sid) breaks lxc-attach
+ systemd 226 (moving pid 1 into /init.scope cgroup) breaks lxc-attach
affects: systemd (Ubuntu) → lxcfs (Ubuntu)

Hi,

I'm hoping to fix this with a cgmanager patch to tasks to escape
'init.scope'. I don't want the list of escapable cgroups to keep
growing, so we'll need to come up with something better long-term.

Changed in cgmanager (Ubuntu):
status: New → In Progress
Changed in lxc (Ubuntu):
status: New → Invalid
Changed in lxcfs (Ubuntu):
status: New → In Progress
Changed in cgmanager (Ubuntu):
importance: Undecided → High
Changed in lxcfs (Ubuntu):
importance: Undecided → High
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxcfs - 0.10-0ubuntu2

---------------
lxcfs (0.10-0ubuntu2) wily; urgency=medium

  * Allow systemd to escape its custom cgroup. (LP: #1497420)

 -- Serge Hallyn <email address hidden> Thu, 15 Oct 2015 08:56:13 -0500

Changed in lxcfs (Ubuntu):
status: In Progress → Fix Released
Serge Hallyn (serge-hallyn) wrote :

lxc-attach still needs a fix to do the same thing - ignore a trailing /init.scope in the init pid's cgroup.

Changed in lxc (Ubuntu):
status: Invalid → Triaged
importance: Undecided → High
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cgmanager - 0.39-2ubuntu2

---------------
cgmanager (0.39-2ubuntu2) wily; urgency=medium

  * Allow systemd to escape its custom cgroup. (LP: #1497420)

 -- Serge Hallyn <email address hidden> Thu, 15 Oct 2015 09:00:25 -0500

Changed in cgmanager (Ubuntu):
status: In Progress → Fix Released
Martin Pitt (pitti) on 2015-10-16
affects: systemd (Debian) → docker (Debian)
Changed in docker (Debian):
status: New → Confirmed
Timo Jyrinki (timo-jyrinki) wrote :

This affects running xenial Ubuntu now from 14.04 LTS.

Package versions:
cgmanager 0.39-2ubuntu2~ubuntu14.04.1~ppa1
liblxc1 1.1.4-0ubuntu1.1~ubuntu14.04.1
lxc 1.1.4-0ubuntu1.1~ubuntu14.04.1
lxcfs 0.11-0ubuntu3~ubuntu14.04.1
python3-lxc 1.1.4-0ubuntu1.1~ubuntu14.04.1

Changed in docker (Debian):
status: Confirmed → Fix Released
Martin Pitt (pitti) wrote :

FTR, I tried the above lxc patch locally, and it does not resolve the reboot bug in a xenial container. I. e. "sudo reboot" in such a container still fails during very early boot due to a cgroup setup error.

Martin Pitt (pitti) wrote :

Specifically, the output is

Rebooting.
systemd 227 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 16.04!

Set hostname to <adt-xenial.new>.
Failed to install release agent, ignoring: No such file or directory
Failed to create /lxc/adt-xenial/init.scope control group: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

During that time, I get the following errors:

Nov 09 08:42:31 donald cgmanager[864]: cgmanager: Invalid path /run/cgmanager/fs/none,name=systemd///lxc/adt-xenial/init.scope (No such file or directory)
Nov 09 08:42:31 donald cgmanager[864]: cgmanager:list_children_main: Could not determine the requested cgroup (name=systemd:lxc/adt-xenial/init.scope)
Nov 09 08:42:31 donald cgmanager[864]: cgmanager: Invalid path /run/cgmanager/fs/none,name=systemd///lxc/adt-xenial/init.scope (No such file or directory)
Nov 09 08:42:31 donald cgmanager[864]: cgmanager:list_keys_main: Could not determine the requested cgroup (name=systemd:lxc/adt-xenial/init.scope)
Nov 09 08:42:31 donald lxcfs[882]: ** (process:882): WARNING **: cgmanager method call org.linuxcontainers.cgmanager0_0.ListChildren failed: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: invalid request. Use G_DBUS_DEBUG=message for more info.
Nov 09 08:42:31 donald lxcfs[882]: ** (process:882): WARNING **: cgmanager method call org.linuxcontainers.cgmanager0_0.ListKeys failed: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: invalid request. Use G_DBUS_DEBUG=message for more info.

Serge Hallyn (serge-hallyn) wrote :

Hi Martin,

thanks for that info. During a reboot, lxc deletes the container's
cgroup, then recreates it. Is systemd expecting the cgroup it
previously created to stick around?

Changed in lxc (Ubuntu):
status: Triaged → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 1.1.5-0ubuntu1

---------------
lxc (1.1.5-0ubuntu1) xenial; urgency=medium

  * New upstream bugfix release (1.1.5)
    (LP: #1514558, LP: #1497420, LP: #1466458, LP: #1510619)
  * Drop proxy detection from the autopkgtest exercise script.

 -- Stéphane Graber <email address hidden> Mon, 09 Nov 2015 14:22:16 -0500

Changed in lxc (Ubuntu):
status: In Progress → Fix Released
Martin Pitt (pitti) wrote :

That did not fix reboot yet. I filed bug 1514690 as it's a different root cause apparenlty.

@Serge: there's nothing that should "stick around", it's a reboot after all and everything including pid 1 gets restarted. It just seems that the second boot can't create the init.scope cgroup as the "outside" host still has some state from the first boot?

Hello Chris, or anyone else affected,

Accepted lxc into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in lxc (Ubuntu Vivid):
status: New → Fix Committed
tags: added: verification-needed
Changed in lxc (Ubuntu Wily):
status: New → Fix Committed
Chris J Arges (arges) wrote :

Hello Chris, or anyone else affected,

Accepted lxc into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Chris West (faux) wrote :

Yep, fixed.

With wily-proposed enabled,

# apt-get install lxc

installs:
2015-11-15 11:29:28 status installed liblxc1:amd64 1.1.5-0ubuntu0.15.10.2
2015-11-15 11:29:29 status installed python3-lxc:amd64 1.1.5-0ubuntu0.15.10.2
2015-11-15 11:29:30 status installed lxc:amd64 1.1.5-0ubuntu0.15.10.2

... and, immediately, lxc works as I expected.

I can confirm reboot is still broken, as per bug 1514690 , but this doesn't matter for my usecase, so woo, thanks.

Martin Pitt (pitti) on 2015-11-16
tags: added: verification-done
removed: verification-needed
Chris J Arges (arges) wrote :

Hello Chris, or anyone else affected,

Accepted lxc into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-done
tags: added: verification-needed
Chris J Arges (arges) wrote :

Hello Chris, or anyone else affected,

Accepted lxc into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 1.1.5-0ubuntu0.15.04.3

---------------
lxc (1.1.5-0ubuntu0.15.04.3) vivid-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix preserve_ns to work on < 3.8 kernels. (LP: #1516971)
    - Fix process title rewrite to not mangle the environment. (LP: #1517107)

lxc (1.1.5-0ubuntu0.15.04.2) vivid-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix ubuntu-cloud template to detect compression algorithm instead
      of hardcoding xz. Also update list of supported releases and use trusty
      as the fallback release. (LP: #1515463)
  * Update lxc-tests description to make it clear that this package is
    meant to be used by developers and by automated testing.

lxc (1.1.5-0ubuntu0.15.04.1) vivid-proposed; urgency=medium

  * New upstream bugfix release (MRE) (1.1.5)
    (LP: #1497420, LP: #1436723, LP: #1441068, LP: #1504496,
     LP: #1466458, LP: #1510619)
  * Drop proxy detection from the autopkgtest exercise script.

 -- Stéphane Graber <email address hidden> Wed, 18 Nov 2015 13:41:23 -0500

Changed in lxc (Ubuntu Vivid):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 1.1.5-0ubuntu0.15.10.3

---------------
lxc (1.1.5-0ubuntu0.15.10.3) wily-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix preserve_ns to work on < 3.8 kernels. (LP: #1516971)
    - Fix process title rewrite to not mangle the environment. (LP: #1517107)

lxc (1.1.5-0ubuntu0.15.10.2) wily-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix ubuntu-cloud template to detect compression algorithm instead
      of hardcoding xz. Also update list of supported releases and use trusty
      as the fallback release. (LP: #1515463)
  * Update lxc-tests description to make it clear that this package is
    meant to be used by developers and by automated testing.

lxc (1.1.5-0ubuntu0.15.10.1) wily-proposed; urgency=medium

  * New upstream bugfix release (MRE) (1.1.5)
    (LP: #1497420, LP: #1441068, LP: #1466458, LP: #1510619)
  * Drop proxy detection from the autopkgtest exercise script.

 -- Stéphane Graber <email address hidden> Wed, 18 Nov 2015 13:40:28 -0500

Changed in lxc (Ubuntu Wily):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for lxc has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.