After running and terminating around 6000 containers overnight, something happened on my box that is affecting every new LXC container I try to start. The DEBUG log file looks like:
lxc-start 1372615570.399 WARN lxc_start - inherited fd 9
lxc-start 1372615570.399 INFO lxc_apparmor - aa_enabled set to 1
It looks like a resource leak, but I'm not yet sure of what that would be.
If it matters, I SIGKILL my lxc-start processes instead of using lxc-stop. Could that have any negative implications?
Oh, and cgroups had almost 6000 entries for VMs that are long dead (I'm guessing it's due to my SIGKILL). I've run cgclear and my /sys/fs/cgroup/*/ dirs are now totally empty, but the new containers still hang.
After running and terminating around 6000 containers overnight, something happened on my box that is affecting every new LXC container I try to start. The DEBUG log file looks like:
lxc-start 1372615570.399 WARN lxc_start - inherited fd 9
lxc-start 1372615570.399 INFO lxc_apparmor - aa_enabled set to 1
lxc-start 1372615570.399 DEBUG lxc_conf - allocated pty '/dev/pts/302' (5/6)
lxc-start 1372615570.399 DEBUG lxc_conf - allocated pty '/dev/pts/303' (7/8)
lxc-start 1372615570.399 DEBUG lxc_conf - allocated pty '/dev/pts/304' (10/11)
lxc-start 1372615570.399 DEBUG lxc_conf - allocated pty '/dev/pts/305' (12/13)
lxc-start 1372615570.399 DEBUG lxc_conf - allocated pty '/dev/pts/306' (14/15)
lxc-start 1372615570.399 DEBUG lxc_conf - allocated pty '/dev/pts/307' (16/17)
lxc-start 1372615570.399 DEBUG lxc_conf - allocated pty '/dev/pts/308' (18/19)
lxc-start 1372615570.399 DEBUG lxc_conf - allocated pty '/dev/pts/309' (20/21)
lxc-start 1372615570.399 INFO lxc_conf - tty's configured
lxc-start 1372615570.399 DEBUG lxc_start - sigchild handler set
lxc-start 1372615570.399 INFO lxc_start - 'vm-59' is initialized
lxc-start 1372615570.404 DEBUG lxc_start - Not dropping cap_sys_boot or watching utmp
lxc-start 1372615570.404 INFO lxc_start - stored saved_nic #0 idx 12392 name vethP59
lxc-start 1372615570.404 INFO lxc_conf - opened /home/x/ vm/vm-59. hold as fd 25
It stops there. In 'ps faux', it looks like:
root 31621 0.0 0.0 25572 1272 ? D 14:06 0:00 \_ lxc-start -n vm-59 -f /tmp/tmp.fG6T6ERZpS -l DEBUG -o /home/x/ lxcdebug/ vm-59.txt -- /usr/sbin/dropbear -F -E -m
On a successful LXC run (prior to the server getting into this state), this hangs just before:
lxc-start 1372394092.208 DEBUG lxc_cgroup - checking '/' (rootfs) cgroup/ cpuset' (cgroup) cgroup/ cpuset' ,opts=' rw,relatime, cpuset, clone_children'
lxc-start 1372394092.208 DEBUG lxc_cgroup - checking '/sys' (sysfs)
lxc-start 1372394092.208 DEBUG lxc_cgroup - checking '/proc' (proc)
lxc-start 1372394092.208 DEBUG lxc_cgroup - checking '/dev' (devtmpfs)
lxc-start 1372394092.208 DEBUG lxc_cgroup - checking '/dev/pts' (devpts)
lxc-start 1372394092.208 DEBUG lxc_cgroup - checking '/run' (tmpfs)
lxc-start 1372394092.208 DEBUG lxc_cgroup - checking '/' (btrfs)
lxc-start 1372394092.208 DEBUG lxc_cgroup - checking '/sys/fs/cgroup' (tmpfs)
lxc-start 1372394092.208 DEBUG lxc_cgroup - checking '/sys/fs/
lxc-start 1372394092.208 INFO lxc_cgroup - [1] found cgroup mounted at '/sys/fs/
lxc-start 1372394092.208 DEBUG lxc_cgroup - get_init_cgroup: found init cgroup for subsys (null) at /
It looks like a resource leak, but I'm not yet sure of what that would be.
If it matters, I SIGKILL my lxc-start processes instead of using lxc-stop. Could that have any negative implications?
Oh, and cgroups had almost 6000 entries for VMs that are long dead (I'm guessing it's due to my SIGKILL). I've run cgclear and my /sys/fs/cgroup/*/ dirs are now totally empty, but the new containers still hang.