agent-state-info: 'error executing "lxc-start": command get_cgroup failed to receive response'

[Antonio Rosales]

We are seeing a few deploys in our automated charm testing fail due to not being able to execute lxc-start.

The error we are seeing is:
  "1":
    agent-state-info: 'error executing "lxc-start": command get_cgroup failed to receive
      response'
    instance-id: pending
    series: precise
ref: http://reports.vapour.ws/charm-tests/charm-bundle-test-10698-results/charm/charm-testing-lxc/2

Tim V. has done some initial research on it:
https://github.com/lxc/lxc/issues/189
which lead to https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1296459
The fix there was to have 2.8.95~2430-0ubuntu3 installed. Our charm testing infrastructure is running 2.8.95~2430-0ubuntu5.

Any help on determining why lxc-start is failing here would be much appreciated.

-thanks,
Antonio

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug.

If there is a simple way to reproduce this ("spin up ami-xxxxxx on amazon and do 'juju deploy whatzit"), please show it here.

Which release are you on, and which ppas do you have installed?

What do the following show:
 dpkg -l | egrep -e "(lxc|apparmor|cgmanager)"
 ps -ef | egrep -e "(cgproxy|cgmanager)"

Can you find the lxcpath juju uses and the container name, and try
starting the container by hand? I.e. if it is /var/lib/juju/lxc and
the container name is foo, then do

 sudo lxc-start -P /var/lib/juju/lxc -n foo -F -l trace -o outout

and attach 'outout' to this bug.

[Tim Van Steenburgh (tvansteenburgh)]

I don't have an easy way to repro, all this occurs fairly regularly on our jenkins slave.

Ubuntu 14.04 LTS (GNU/Linux 3.13.0-29-generic x86_64)

juju version: 1.20.13-trusty-amd64

1 ppa installed: http://ppa.launchpad.net/juju/stable/ubuntu

$ dpkg -l | egrep -e "(lxc|apparmor|cgmanager)"
ii apparmor 2.8.95~2430-0ubuntu5 amd64 User-space parser utility for AppArmor
ii cgmanager 0.24-0ubuntu7 amd64 Central cgroup manager daemon
ii dh-apparmor 2.8.95~2430-0ubuntu5.1 all AppArmor debhelper routines
ii libapparmor-perl 2.8.95~2430-0ubuntu5 amd64 AppArmor library Perl bindings
ii libapparmor1:amd64 2.8.95~2430-0ubuntu5 amd64 changehat AppArmor library
ii libcgmanager0:amd64 0.24-0ubuntu6 amd64 Central cgroup manager daemon (client library)
ii liblxc1 1.0.3-0ubuntu3 amd64 Linux Containers userspace tools (library)
ii lxc 1.0.3-0ubuntu3 amd64 Linux Containers userspace tools
ii lxc-templates 1.0.3-0ubuntu3 amd64 Linux Containers userspace tools (templates)
ii python3-lxc 1.0.3-0ubuntu3 amd64 Linux Containers userspace tools (Python 3.x bindings)

$ ps -ef | egrep -e "(cgproxy|cgmanager)"
root 348 1 0 Nov03 ? 00:07:44 /sbin/cgmanager --sigstop -m name=systemd
jenkins 10661 9679 0 18:36 pts/14 00:00:00 egrep -e (cgproxy|cgmanager)

There are many containers on this host and I'm not sure which one(s) are being used, but regardless of which I pick I get the same result:

jenkins@charm-bundle-slave:/var/lib/juju/containers$ sudo lxc-start -P /var/lib/juju/containers -n juju-trusty-lxc-template -L -l trace -o outout
lxc-start: Permission denied - Failed to make / rslave
lxc-start: failed to setup rootfs for 'juju-trusty-lxc-template'
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'juju-trusty-lxc-template'

Advice?

[Curtis Hovey (sinzui)]

lxc*1.0.3 looks odd. That is one version I don't expect to see.

trusty has 1.0.6 which is what I think the host is and should be using.
a precise host has 1.0.4
    found in the cloud tools archive http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/l/lxc/
juju stable ppa provides 1.0.0-alpha1, a copy of what was formerly in clout-tools.

Does the host need to be upgraded or if precise, use the correct archive?

[Curtis Hovey (sinzui)]

The host machine is trusty and was missing updated. We have updated the lxc and other packages. We will watch the processes to see if the issue is resolved or still exists.

[Launchpad Janitor (janitor)]

[Expired for lxc (Ubuntu) because there has been no activity for 60 days.]