LXC containers fail to start on trusty hosts

[Haw Loeung]

Hi,

We've recently destroyed our old precise environment and am trying to bring up the same environment with trusty. We're seeing issues with service units that we're trying to bring up in LXC containers. The non-LXC service units are fine.

Snippet from 'juju status':

| "5":
| agent-state: started
| agent-version: 1.18.1.1
| dns-name: gambooge.scalingstack
| instance-id: /MAAS/api/1.0/nodes/node-429f2bc4-a8be-11e3-8cfc-0025b3e27a30/
| series: trusty
| containers:
| 5/lxc/0:
| agent-state-info: '(error: error executing "lxc-start": command get_cgroup
| failed to receive response)'
| instance-id: pending
| series: trusty
| 5/lxc/1:
| agent-state-info: '(error: error executing "lxc-start": command get_cgroup
| failed to receive response)'
| instance-id: pending
| series: trusty
| 5/lxc/2:
| agent-state-info: '(error: error executing "lxc-start": command get_cgroup
| failed to receive response)'
| instance-id: pending
| series: trusty
| 5/lxc/3:
| agent-state-info: '(error: error executing "lxc-start": command get_cgroup
| failed to receive response)'
| instance-id: pending
| series: trusty
| 5/lxc/4:
| agent-state-info: '(error: error executing "lxc-start": command get_cgroup
| failed to receive response)'
| instance-id: pending
| series: trusty
| 5/lxc/5:
| agent-state-info: '(error: error executing "lxc-start": command get_cgroup
| failed to receive response)'
| instance-id: pending
| series: trusty

Juju logs:

| ubuntu@gambooge:~$ cat /var/log/juju/machine-5.log
| 2014-05-06 06:37:39 INFO juju.cmd supercommand.go:297 running juju-1.18.1.1-trusty-amd64 [gc]
| 2014-05-06 06:37:39 INFO juju.cmd.jujud machine.go:127 machine agent machine-5 start (1.18.1.1-trusty-amd64 [gc])
| 2014-05-06 06:37:39 DEBUG juju.agent agent.go:384 read agent config, format "1.18"
| 2014-05-06 06:37:39 INFO juju runner.go:262 worker: start "api"
| 2014-05-06 06:37:39 INFO juju apiclient.go:114 state/api: dialing "wss://10.222.64.2:17070/"
| 2014-05-06 06:37:39 INFO juju runner.go:262 worker: start "termination"
| 2014-05-06 06:37:39 INFO juju apiclient.go:124 state/api: connection established
| 2014-05-06 06:37:39 DEBUG juju.agent agent.go:523 writing configuration file
| 2014-05-06 06:37:39 INFO juju runner.go:262 worker: start "upgrader"
| 2014-05-06 06:37:39 INFO juju runner.go:262 worker: start "upgrade-steps"
| 2014-05-06 06:37:39 INFO juju runner.go:262 worker: start "machiner"
| 2014-05-06 06:37:39 INFO juju.cmd.jujud machine.go:458 upgrade to 1.18.1.1-trusty-amd64 already completed.
| 2014-05-06 06:37:39 INFO juju.cmd.jujud machine.go:445 upgrade to 1.18.1.1-trusty-amd64 completed.
| 2014-05-06 06:37:39 INFO juju runner.go:262 worker: start "logger"
| 2014-05-06 06:37:39 DEBUG juju.worker.logger logger.go:35 initial log config: "<root>=DEBUG"
| 2014-05-06 06:37:39 INFO juju runner.go:262 worker: start "machineenvironmentworker"
| 2014-05-06 06:37:39 INFO juju runner.go:262 worker: start "rsyslog"
| 2014-05-06 06:37:39 DEBUG juju.worker.logger logger.go:60 logger setup
| 2014-05-06 06:37:39 DEBUG juju.worker.machineenvironment machineenvironmentworker.go:70 write system files: true
| 2014-05-06 06:37:39 DEBUG juju.worker.rsyslog worker.go:76 starting rsyslog worker mode 1 for "machine-5" ""
| 2014-05-06 06:37:39 INFO juju runner.go:262 worker: start "authenticationworker"
| 2014-05-06 06:37:39 INFO juju.worker.machiner machiner.go:85 setting addresses for machine-5 to ["local-machine:127.0.0.1" "local-cloud:10.222.64.4" "local-machine:::1" "fe80::223:7dff:fee1:76b6"]
| 2014-05-06 06:37:39 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>=WARNING;unit=DEBUG"
| 2014-05-06 06:38:11 ERROR juju.container.lxc lxc.go:217 container failed to start: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:11 ERROR juju.provisioner.lxc lxc-broker.go:87 failed to start container: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:11 ERROR juju.provisioner provisioner_task.go:437 cannot start instance for machine "5/lxc/0": error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:16 ERROR juju.container.lxc lxc.go:217 container failed to start: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:16 ERROR juju.provisioner.lxc lxc-broker.go:87 failed to start container: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:16 ERROR juju.provisioner provisioner_task.go:437 cannot start instance for machine "5/lxc/1": error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:21 ERROR juju.container.lxc lxc.go:217 container failed to start: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:21 ERROR juju.provisioner.lxc lxc-broker.go:87 failed to start container: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:21 ERROR juju.provisioner provisioner_task.go:437 cannot start instance for machine "5/lxc/2": error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:26 ERROR juju.container.lxc lxc.go:217 container failed to start: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:26 ERROR juju.provisioner.lxc lxc-broker.go:87 failed to start container: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:26 ERROR juju.provisioner provisioner_task.go:437 cannot start instance for machine "5/lxc/3": error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:31 ERROR juju.container.lxc lxc.go:217 container failed to start: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:31 ERROR juju.provisioner.lxc lxc-broker.go:87 failed to start container: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:31 ERROR juju.provisioner provisioner_task.go:437 cannot start instance for machine "5/lxc/4": error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:37 ERROR juju.container.lxc lxc.go:217 container failed to start: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:37 ERROR juju.provisioner.lxc lxc-broker.go:87 failed to start container: error executing "lxc-start": command get_cgroup failed to receive response
| 2014-05-06 06:38:37 ERROR juju.provisioner provisioner_task.go:437 cannot start instance for machine "5/lxc/5": error executing "lxc-start": command get_cgroup failed to receive response

Output from trying to start these containers up manually:

| root@gambooge:/var/log/apt# lxc-start --name=juju-machine-5-lxc-1
| lxc-start: No such file or directory - failed to create symlink for kmsg
| lxc-start: failed to setup kmsg for 'juju-machine-5-lxc-1'
| lxc-start: No such file or directory - Error creating /usr/lib/x86_64-linux-gnu/lxc/dev/fd
| lxc-start: failed to setup /dev symlinks for 'juju-machine-5-lxc-1'
| lxc-start: failed to setup the container
| lxc-start: invalid sequence number 1. expected 2

Snippet from running lxc-start through strace:

| [pid 25920] mount("/var/log/juju", "/usr/lib/x86_64-linux-gnu/lxc/var/log/juju", 0x7fffebb76e4b, MS_REMOUNT|MS_BIND, NULL) = 0
| [pid 25920] read(11, "", 4096) = 0
| [pid 25920] close(11) = 0
| [pid 25920] munmap(0x7fecab07b000, 4096) = 0
| [pid 25920] access("/usr/lib/x86_64-linux-gnu/lxc/dev/console", F_OK) = -1 ENOENT (No such file or directory)
| [pid 25920] unlink("/usr/lib/x86_64-linux-gnu/lxc/dev/kmsg") = -1 ENOENT (No such file or directory)
| [pid 25920] symlink("console", "/usr/lib/x86_64-linux-gnu/lxc/dev/kmsg") = -1 ENOENT (No such file or directory)
| [pid 25920] write(3, " lxc-start 1399359039.165 E"..., 113) = 113
| [pid 25920] write(2, "lxc-start: ", 11lxc-start: ) = 11
| [pid 25920] write(2, "No such file or directory - fail"..., 61No such file or directory - failed to create symlink for kmsg) = 61
| [pid 25920] write(2, "\n", 1
| ) = 1
| [pid 25920] write(3, " lxc-start 1399359039.165 E"..., 99) = 99
| [pid 25920] write(2, "lxc-start: ", 11lxc-start: ) = 11
| [pid 25920] write(2, "failed to setup kmsg for 'juju-m"..., 47failed to setup kmsg for 'juju-machine-5-lxc-0') = 47
| [pid 25920] write(2, "\n", 1
| ) = 1
| [pid 25920] symlink("/proc/self/fd", "/usr/lib/x86_64-linux-gnu/lxc/dev/fd") = -1 ENOENT (No such file or directory)
| [pid 25920] write(3, " lxc-start 1399359039.166 E"..., 131) = 131
| [pid 25920] write(2, "lxc-start: ", 11lxc-start: ) = 11
| [pid 25920] write(2, "No such file or directory - Erro"..., 79No such file or directory - Error creating /usr/lib/x86_64-linux-gnu/lxc/dev/fd) = 79
| [pid 25920] write(2, "\n", 1
| ) = 1
| [pid 25920] write(3, " lxc-start 1399359039.166 E"..., 108) = 108
| [pid 25920] write(2, "lxc-start: ", 11lxc-start: ) = 11
| [pid 25920] write(2, "failed to setup /dev symlinks fo"..., 56failed to setup /dev symlinks for 'juju-machine-5-lxc-0') = 56
| [pid 25920] write(2, "\n", 1
| ) = 1
| [pid 25920] write(3, " lxc-start 1399359039.167 E"..., 82) = 82
| [pid 25920] write(2, "lxc-start: ", 11lxc-start: ) = 11
| [pid 25920] write(2, "failed to setup the container", 29failed to setup the container) = 29

Any ideas what's going on here?

ii lxc 1.0.3-0ubuntu3 amd64

juju-core:
  Installed: 1.18.1-0ubuntu1~ctools0
  Candidate: 1.18.1-0ubuntu1~ctools0

apparmor:
  Installed: 2.8.95~2430-0ubuntu5
  Candidate: 2.8.95~2430-0ubuntu5

Thanks,

Haw

[Haw Loeung (hloeung)]

Turns out to be related to MAAS overriding our DNS configs so LXC containers weren't created correctly. I've marked this as invalid.