containers no longer start after upgrade to 2.0.0~rc11-0ubuntu1

[Colin Watson]

Running xenial, after upgrading to lxc 2.0.0~rc11-0ubuntu1, I can't start containers any more:

  $ sudo lxc-start -F -n precise-lptest-base
  ln: failed to create symbolic link '/usr/lib/x86_64-linux-gnu/lxc/sys/fs/cgroup/cpu/cpu,cpuacct': Read-only file system
  lxc-start: conf.c: run_buffer: 347 Script exited with status 1
  lxc-start: conf.c: lxc_setup: 3750 failed to run mount hooks for container 'precise-lptest-base'.
  lxc-start: start.c: do_start: 793 failed to setup the container
  lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
  lxc-start: start.c: __lxc_start: 1286 failed to spawn 'precise-lptest-base'
  lxc-start: lxc_start.c: main: 344 The container failed to start.
  lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.

Downgrading all the lxc binary packages I have installed to 2.0.0~rc10-0ubuntu2 makes things work again.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Curtis Hovey (sinzui)]

This was also seen by the in Juju CI on the wily hosts that pull lxd/lxc packages from the lxd ppa. We downgrades the wily machines to packages from wily-updates to get lxc working again.

[Stéphane Graber (stgraber)]

Curtis: can you paste a log to confirm it's the same issue as Colin?

It looks like some of the cgroup:mixed logic recently introduced in cgfsng (to match that of the old cgfs backend) is a bit wrong here and attempts to create a cpu/cpuset,cpuacct symlink instead of symlinking cpu to cpuset,cpuacct.

Serge: can you take a look?

[Stéphane Graber (stgraber)]

Serge is out today actually, I'll try to take a look.

[Curtis Hovey (sinzui)]

This is the error
jenkins@wily-slave:~$ sudo lxc-start -n curtis -F
ln: failed to create symbolic link ‘/usr/lib/x86_64-linux-gnu/lxc/sys/fs/cgroup/cpu/cpu,cpuacct’: Read-only file system
lxc-start: conf.c: run_buffer: 347 Script exited with status 1
lxc-start: conf.c: lxc_setup: 3750 failed to run mount hooks for container 'curtis'.
lxc-start: start.c: do_start: 793 failed to setup the container
lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-start: start.c: __lxc_start: 1286 failed to spawn 'curtis'
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.

Attached is the log of what happened.

The wily-slave host uses the lxd ppa. it has a lower priority.
These packages were installed and get updates outside of regular wily-updates:
    sudo apt-get install --no-install-recommends lxc=2.0* lxc-templates=2.0* lxc1=2.0* python3-lxc=2.0* liblxc1=2.0*

jenkins@wily-slave:~$ apt-cache policy lxc
lxc:
  Installed: 2.0.0~rc11-0ubuntu1~ubuntu15.10.1~ppa1
  Candidate: 2.0.0~rc11-0ubuntu1~ubuntu15.10.1~ppa1
  Package pin: 2.0.0~rc11-0ubuntu1~ubuntu15.10.1~ppa1
  Version table:
 *** 2.0.0~rc11-0ubuntu1~ubuntu15.10.1~ppa1 500
        200 http://ppa.launchpad.net/ubuntu-lxc/lxd-stable/ubuntu/ wily/main amd64 Packages
        100 /var/lib/dpkg/status
     1.1.5-0ubuntu0.15.10.3 500
        500 http://us-sw-1.joyent.archive.ubuntu.com/ubuntu/ wily-updates/main amd64 Packages
     1.1.4-0ubuntu1 500
        500 http://us-sw-1.joyent.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages

jenkins@wily-slave:~$ apt-cache policy lxcfs
lxcfs:
  Installed: 2.0.0~rc6-0ubuntu1~ubuntu15.10.1~ppa1
  Candidate: 2.0.0~rc6-0ubuntu1~ubuntu15.10.1~ppa1
  Package pin: 2.0.0~rc6-0ubuntu1~ubuntu15.10.1~ppa1
  Version table:
 *** 2.0.0~rc6-0ubuntu1~ubuntu15.10.1~ppa1 500
        200 http://ppa.launchpad.net/ubuntu-lxc/lxd-stable/ubuntu/ wily/main amd64 Packages
        100 /var/lib/dpkg/status
     0.10-0ubuntu2.3 500
        400 http://archive.ubuntu.com/ubuntu/ wily-proposed/main amd64 Packages
     0.10-0ubuntu2.1 500
        500 http://us-sw-1.joyent.archive.ubuntu.com/ubuntu/ wily-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ wily-security/main amd64 Packages
     0.10-0ubuntu2 500
        500 http://us-sw-1.joyent.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages

[Stéphane Graber (stgraber)]

Ok, cool, that confirms it's the same thing.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxcfs - 2.0.0~rc6-0ubuntu2

---------------
lxcfs (2.0.0~rc6-0ubuntu2) xenial; urgency=medium

  * Cherry-pick tentative upstream fix for the mount hook. (LP: #1559169)

 -- Stéphane Graber <email address hidden> Fri, 18 Mar 2016 23:29:21 -0400