systemd 226 (moving pid 1 into /init.scope cgroup) breaks lxc-attach

[Chris West]

Once systemd 226 is installed in an unprivileged Debian Sid container, lxc-attach no-longer functions:

% lxc-attach -n siddy
lxc-attach: cgmanager.c: lxc_cgmanager_enter: 698 call to cgmanager_move_pid_abs_sync failed: invalid request
lxc-attach: cgmanager.c: cgm_attach: 1460 Failed to enter group /user.slice/user-1000.slice/session-1.scope/lxc/siddy/init.scope
lxc-attach: attach.c: lxc_attach: 909 error communicating with child process

% journalctl -u cgmanager | tail -n 1
Sep 18 20:20:44 astoria cgmanager[1169]: cgmanager:per_ctrl_move_pid_main: pid 21918 (uid 1000 gid 1000) may not write to /run/cgmanager/fs/none,name=systemd//user.slice/user-1000.slice/session-1.scope/lxc/siddy/init.scope/tasks

Full walkthrough:

% lxc-create -n siddy -t download -- -d debian -r sid -a amd64
Using image from local cache
Unpacking the rootfs
...

% lxc-start -n siddy
% lxc-attach -n siddy

root@siddy:/# apt-get update
Get:1 http://http.debian.net sid InRelease [253 kB]
...
Get:7 http://http.debian.net sid/non-free amd64 Packages [91.3 kB]
Fetched 13.2 MB in 3s (3789 kB/s)
Reading package lists... Done

root@siddy:/# apt-get install systemd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libsystemd0
Suggested packages:
  systemd-ui systemd-container
Recommended packages:
  libpam-systemd dbus
The following packages will be upgraded:
  libsystemd0 systemd
2 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.
Need to get 3668 kB of archives.
After this operation, 1100 kB disk space will be freed.
Do you want to continue? [Y/n]
Get:1 http://http.debian.net/debian/ sid/main libsystemd0 amd64 226-2 [242 kB]
Get:2 http://http.debian.net/debian/ sid/main systemd amd64 226-2 [3426 kB]
...
Setting up systemd (226-2) ...
Installing new version of config file /etc/X11/xinit/xinitrc.d/50-systemd-user.sh ...
Installing new version of config file /etc/dbus-1/system.d/org.freedesktop.login1.conf ...
addgroup: The group `systemd-journal' already exists as a system group. Exiting.
Failed to set capabilities on file `/usr/bin/systemd-detect-virt' (Invalid argument)
The value of the capability argument is not permitted for a file. Or the file is not a regular (non-symlink) file
Failed to execute operation: File exists
Removing obsolete conffile /etc/dbus-1/system.d/org.freedesktop.machine1.conf ...

root@siddy:/# exit

% lxc-attach -n siddy
lxc-attach: cgmanager.c: lxc_cgmanager_enter: 698 call to cgmanager_move_pid_abs_sync failed: invalid request
lxc-attach: cgmanager.c: cgm_attach: 1460 Failed to enter group /user.slice/user-1000.slice/session-1.scope/lxc/siddy/init.scope
lxc-attach: attach.c: lxc_attach: 909 error communicating with child process

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: lxc 1.1.3-0ubuntu1 [modified: usr/lib/x86_64-linux-gnu/lxc/lxc-net]
ProcVersionSignature: Ubuntu 4.2.0-10.11-generic 4.2.0
Uname: Linux 4.2.0-10-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.18.1-0ubuntu1
Architecture: amd64
Date: Fri Sep 18 20:19:58 2015
SourcePackage: lxc
UpgradeStatus: Upgraded to wily on 2015-08-28 (20 days ago)
defaults.conf:
 lxc.network.type = veth
 lxc.network.link = lxcbr0
 lxc.network.flags = up
 lxc.network.hwaddr = 00:16:3e:xx:xx:xx

[Serge Hallyn (serge-hallyn)]

Hi,

I'm hoping to fix this with a cgmanager patch to tasks to escape
'init.scope'. I don't want the list of escapable cgroups to keep
growing, so we'll need to come up with something better long-term.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxcfs - 0.10-0ubuntu2

---------------
lxcfs (0.10-0ubuntu2) wily; urgency=medium

  * Allow systemd to escape its custom cgroup. (LP: #1497420)

 -- Serge Hallyn <email address hidden> Thu, 15 Oct 2015 08:56:13 -0500

[Serge Hallyn (serge-hallyn)]

lxc-attach still needs a fix to do the same thing - ignore a trailing /init.scope in the init pid's cgroup.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cgmanager - 0.39-2ubuntu2

---------------
cgmanager (0.39-2ubuntu2) wily; urgency=medium

  * Allow systemd to escape its custom cgroup. (LP: #1497420)

 -- Serge Hallyn <email address hidden> Thu, 15 Oct 2015 09:00:25 -0500

[Timo Jyrinki (timo-jyrinki)]

This affects running xenial Ubuntu now from 14.04 LTS.

Package versions:
cgmanager 0.39-2ubuntu2~ubuntu14.04.1~ppa1
liblxc1 1.1.4-0ubuntu1.1~ubuntu14.04.1
lxc 1.1.4-0ubuntu1.1~ubuntu14.04.1
lxcfs 0.11-0ubuntu3~ubuntu14.04.1
python3-lxc 1.1.4-0ubuntu1.1~ubuntu14.04.1

[Serge Hallyn (serge-hallyn)]

Yup, we need

https://github.com/lxc/lxc/commit/f348e47c93568b4f0c371cf5df1c98d4e816a86c

in the packages.

[Martin Pitt (pitti)]

FTR, I tried the above lxc patch locally, and it does not resolve the reboot bug in a xenial container. I. e. "sudo reboot" in such a container still fails during very early boot due to a cgroup setup error.

[Martin Pitt (pitti)]

Specifically, the output is

Rebooting.
systemd 227 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu 16.04!

Set hostname to <adt-xenial.new>.
Failed to install release agent, ignoring: No such file or directory
Failed to create /lxc/adt-xenial/init.scope control group: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.
Freezing execution.

During that time, I get the following errors:

Nov 09 08:42:31 donald cgmanager[864]: cgmanager: Invalid path /run/cgmanager/fs/none,name=systemd///lxc/adt-xenial/init.scope (No such file or directory)
Nov 09 08:42:31 donald cgmanager[864]: cgmanager:list_children_main: Could not determine the requested cgroup (name=systemd:lxc/adt-xenial/init.scope)
Nov 09 08:42:31 donald cgmanager[864]: cgmanager: Invalid path /run/cgmanager/fs/none,name=systemd///lxc/adt-xenial/init.scope (No such file or directory)
Nov 09 08:42:31 donald cgmanager[864]: cgmanager:list_keys_main: Could not determine the requested cgroup (name=systemd:lxc/adt-xenial/init.scope)
Nov 09 08:42:31 donald lxcfs[882]: ** (process:882): WARNING **: cgmanager method call org.linuxcontainers.cgmanager0_0.ListChildren failed: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: invalid request. Use G_DBUS_DEBUG=message for more info.
Nov 09 08:42:31 donald lxcfs[882]: ** (process:882): WARNING **: cgmanager method call org.linuxcontainers.cgmanager0_0.ListKeys failed: GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: invalid request. Use G_DBUS_DEBUG=message for more info.

[Serge Hallyn (serge-hallyn)]

Hi Martin,

thanks for that info. During a reboot, lxc deletes the container's
cgroup, then recreates it. Is systemd expecting the cgroup it
previously created to stick around?

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 1.1.5-0ubuntu1

---------------
lxc (1.1.5-0ubuntu1) xenial; urgency=medium

  * New upstream bugfix release (1.1.5)
    (LP: #1514558, LP: #1497420, LP: #1466458, LP: #1510619)
  * Drop proxy detection from the autopkgtest exercise script.

 -- Stéphane Graber <email address hidden> Mon, 09 Nov 2015 14:22:16 -0500

[Martin Pitt (pitti)]

That did not fix reboot yet. I filed bug 1514690 as it's a different root cause apparenlty.

@Serge: there's nothing that should "stick around", it's a reboot after all and everything including pid 1 gets restarted. It just seems that the second boot can't create the init.scope cgroup as the "outside" host still has some state from the first boot?

[Chris J Arges (arges)]

Hello Chris, or anyone else affected,

Accepted lxc into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Chris, or anyone else affected,

Accepted lxc into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris West (faux)]

Yep, fixed.

With wily-proposed enabled,

# apt-get install lxc

installs:
2015-11-15 11:29:28 status installed liblxc1:amd64 1.1.5-0ubuntu0.15.10.2
2015-11-15 11:29:29 status installed python3-lxc:amd64 1.1.5-0ubuntu0.15.10.2
2015-11-15 11:29:30 status installed lxc:amd64 1.1.5-0ubuntu0.15.10.2

... and, immediately, lxc works as I expected.

I can confirm reboot is still broken, as per bug 1514690 , but this doesn't matter for my usecase, so woo, thanks.

[Chris J Arges (arges)]

Hello Chris, or anyone else affected,

Accepted lxc into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Chris, or anyone else affected,

Accepted lxc into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1.1.5-0ubuntu0.15.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 1.1.5-0ubuntu0.15.04.3

---------------
lxc (1.1.5-0ubuntu0.15.04.3) vivid-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix preserve_ns to work on < 3.8 kernels. (LP: #1516971)
    - Fix process title rewrite to not mangle the environment. (LP: #1517107)

lxc (1.1.5-0ubuntu0.15.04.2) vivid-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix ubuntu-cloud template to detect compression algorithm instead
      of hardcoding xz. Also update list of supported releases and use trusty
      as the fallback release. (LP: #1515463)
  * Update lxc-tests description to make it clear that this package is
    meant to be used by developers and by automated testing.

lxc (1.1.5-0ubuntu0.15.04.1) vivid-proposed; urgency=medium

  * New upstream bugfix release (MRE) (1.1.5)
    (LP: #1497420, LP: #1436723, LP: #1441068, LP: #1504496,
     LP: #1466458, LP: #1510619)
  * Drop proxy detection from the autopkgtest exercise script.

 -- Stéphane Graber <email address hidden> Wed, 18 Nov 2015 13:41:23 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 1.1.5-0ubuntu0.15.10.3

---------------
lxc (1.1.5-0ubuntu0.15.10.3) wily-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix preserve_ns to work on < 3.8 kernels. (LP: #1516971)
    - Fix process title rewrite to not mangle the environment. (LP: #1517107)

lxc (1.1.5-0ubuntu0.15.10.2) wily-proposed; urgency=medium

  * Cherry-pick from upstream:
    - Fix ubuntu-cloud template to detect compression algorithm instead
      of hardcoding xz. Also update list of supported releases and use trusty
      as the fallback release. (LP: #1515463)
  * Update lxc-tests description to make it clear that this package is
    meant to be used by developers and by automated testing.

lxc (1.1.5-0ubuntu0.15.10.1) wily-proposed; urgency=medium

  * New upstream bugfix release (MRE) (1.1.5)
    (LP: #1497420, LP: #1441068, LP: #1466458, LP: #1510619)
  * Drop proxy detection from the autopkgtest exercise script.

 -- Stéphane Graber <email address hidden> Wed, 18 Nov 2015 13:40:28 -0500

[Stéphane Graber (stgraber)]

The verification of the Stable Release Update for lxc has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.