error: error executing "lxc-start" when juju client runs inside kvm

[Manoj Iyer]

Deploying a (mysql) service on a KVM instance using juju local lxc fails in lxc-start.

 agent-state-info: '(error: error executing "lxc-start": The container fa
iled to start.; To get more details, run the container in foreground mode.;
Additional information can be obtained by setting the --logfile and --log-priority options.)'

Here are the steps to reproduce:

* juju switch local
* juju bootstrap
* juju set-constraints mem=2G
* juju add-machine

* (OPTIONAL) ssh to machine and modify network setting, this is because we want to keep the ip address of the service the same as the ip that was assigned to the kvm instance. You can skip this part and you will see the same failure.

* juju ssh 2

sudo rm /etc/network/interfaces.d/eth0.cfg
sudo vi /etc/network/interfaces.d/lxcbr0.cfg
-------------- lxcbr0.cfg -------------------
auto eth0
iface eth0 inet manual

auto lxcbr0
iface lxcbr0 inet dhcp
    bridge_ports eth0
--------------------------------------------------

* juju deploy mysql --lxc:2

environment: local
machines:
  "0":
    agent-state: started
    agent-version: 1.18.1.1
    dns-name: localhost
    instance-id: localhost
    series: trusty
  "2":
    agent-state: started
    agent-version: 1.18.1.1
    dns-name: 10.0.3.247
    instance-id: ubuntu-local-machine-2
    series: trusty
    containers:
      2/lxc/0:
        agent-state-info: '(error: error executing "lxc-start": The container fa
iled
          to start.; To get more details, run the container in foreground mode.;
 Additional
          information can be obtained by setting the --logfile and --log-priorit
y
          options.)'
        instance-id: pending
        series: trusty
    hardware: arch=amd64
services:
  mysql:
    charm: cs:trusty/mysql-1
    exposed: false
    relations:
      cluster:
      - mysql
    units:
      mysql/0:
        agent-state: pending
        machine: 2/lxc/0

ubuntu@ubuntu-local-machine-2:~$ sudo lxc-ls --stopped
juju-machine-2-lxc-0
ubuntu@ubuntu-local-machine-2:~$

ubuntu@ubuntu-local-machine-2:~$ sudo lxc-start -n juju-machine-2-lxc-0
lxc-start: cgroupfs failed to detect cgroup metadata
lxc-start: failed initializing cgroup support
lxc-start: failed to spawn 'juju-machine-2-lxc-0'
lxc-start: The container failed to start.
lxc-start: Additional information can be obtained by setting the --logfile and --log-priority options.
ubuntu@ubuntu-local-machine-2:~$

[Manoj Iyer (manjo)]

ubuntu@ubuntu-local-machine-2:~$ cat /var/log/lxc/juju-machine-2-lxc-0.log
      lxc-start 1405096303.317 ERROR lxc_cgfs - cgroupfs failed to detect cgroup metadata
      lxc-start 1405096303.317 ERROR lxc_start - failed initializing cgroup support
      lxc-start 1405096303.339 ERROR lxc_start - failed to spawn 'juju-machine-2-lxc-0'
      lxc-start 1405096303.339 ERROR lxc_start_ui - The container failed to start.
      lxc-start 1405096303.339 ERROR lxc_start_ui - Additional information can be obtained by setting the --logfile and --log-priority options.
      lxc-start 1405096708.183 INFO lxc_start_ui - using rcfile /var/lib/lxc/juju-machine-2-lxc-0/config
      lxc-start 1405096708.184 WARN lxc_log - lxc_log_init called with log already initialized
      lxc-start 1405096708.184 INFO lxc_lsm - LSM security driver AppArmor
      lxc-start 1405096708.184 DEBUG lxc_conf - allocated pty '/dev/pts/2' (5/6)
      lxc-start 1405096708.184 DEBUG lxc_conf - allocated pty '/dev/pts/3' (7/8)
      lxc-start 1405096708.184 DEBUG lxc_conf - allocated pty '/dev/pts/4' (9/10)
      lxc-start 1405096708.184 DEBUG lxc_conf - allocated pty '/dev/pts/5' (11/12)
      lxc-start 1405096708.184 INFO lxc_conf - tty's configured
      lxc-start 1405096708.184 DEBUG lxc_start - sigchild handler set
      lxc-start 1405096708.184 DEBUG lxc_console - opening /dev/tty for console peer
      lxc-start 1405096708.184 DEBUG lxc_console - using '/dev/tty' as console
      lxc-start 1405096708.184 DEBUG lxc_console - 3426 got SIGWINCH fd 17
      lxc-start 1405096708.185 DEBUG lxc_console - set winsz dstfd:14 cols:80 rows:55
      lxc-start 1405096708.185 INFO lxc_start - 'juju-machine-2-lxc-0' is initialized
      lxc-start 1405096708.195 DEBUG lxc_start - Not dropping cap_sys_boot or watching utmp
      lxc-start 1405096708.197 DEBUG lxc_conf - instanciated veth 'vethDP2WB6/vethNEM4PX', index is '7'
      lxc-start 1405096708.197 INFO lxc_cgroup - cgroup driver cgroupfs initing for juju-machine-2-lxc-0
      lxc-start 1405096708.197 ERROR lxc_cgfs - cgroupfs failed to detect cgroup metadata
      lxc-start 1405096708.197 ERROR lxc_start - failed initializing cgroup support
      lxc-start 1405096708.219 ERROR lxc_start - failed to spawn 'juju-machine-2-lxc-0'
      lxc-start 1405096708.219 ERROR lxc_start_ui - The container failed to start.
      lxc-start 1405096708.219 ERROR lxc_start_ui - Additional information can be obtained by setting the --logfile and --log-priority options.
ubuntu@ubuntu-local-machine-2:~$

[Manoj Iyer (manjo)]

ubuntu@ubuntu-local-machine-2:~$ dpkg -l | grep cgroup
ii cgmanager 0.24-0ubuntu7 amd64 Central cgroup manager daemon
ii libcgmanager0:amd64 0.24-0ubuntu7 amd64 Central cgroup manager daemon (client library)
ubuntu@ubuntu-local-machine-2:~$

ubuntu@ubuntu-local-machine-2:~$ cat /proc/self/mounts
rootfs / rootfs rw 0 0
/dev/disk/by-uuid/cf350644-b197-4162-9d18-4448a5c90afc / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
/dev/disk/by-uuid/cf350644-b197-4162-9d18-4448a5c90afc /var/log/juju ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
devpts /dev/lxc/console devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
devpts /dev/lxc/tty1 devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
devpts /dev/lxc/tty2 devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
devpts /dev/lxc/tty3 devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
devpts /dev/lxc/tty4 devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0
devpts /dev/ptmx devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/firmware/efi/efivars efivarfs rw,relatime 0 0
none /run tmpfs rw,nosuid,noexec,relatime,size=1611264k,mode=755 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
ubuntu@ubuntu-local-machine-2:~$

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Manoj Iyer (manjo)]

ubuntu@ubuntu-local-machine-2:~$ find /sys/fs/cgroup -maxdepth 3
/sys/fs/cgroup
/sys/fs/cgroup/cgmanager
/sys/fs/cgroup/cgmanager/sock
ubuntu@ubuntu-local-machine-2:~$

[Manoj Iyer (manjo)]

I am not sure if this might be the cause of failure.. ignore this comment if this does not make sense.

Reading this patch to lxc-dev:
https://lists.linuxcontainers.org/pipermail/lxc-devel/2014-February/007780.html

I see:

+ if (geteuid() == 0)
+ d->cgroup_pattern = lxc_global_config_value("lxc.cgroup.pattern");
+ if (!d->cgroup_pattern)
+ d->cgroup_pattern = "%n";
+
+ d->meta = lxc_cgroup_load_meta();
+ if (!d->meta) {
+ ERROR("cgroupfs failed to detect cgroup metadata");
+ goto err2;
+ }

should it be d->cgroup_pattern = "/%n"; instead ?

[Serge Hallyn (serge-hallyn)]

Hi,

could you try logging into the kvm instance and manually creating and starting a container, to see if you get the same results?

[Serge Hallyn (serge-hallyn)]

Are you still seeing this?