Terminated LXD instances processes stick around

[James Page]

I spun up a number of lxd instances using the lxd driver and then deleted them; the instances disappear from lxc list, but processes remain:

ceilome+ 4050 1 0 10:18 ? 00:00:10 /usr/bin/python /usr/bin/ceilometer-agent-compute --config-file=/etc/ceilometer/ceilometer.conf --log-file=/var/log/ceilometer/ceilometer-agent-compute.log
neutron 7897 1 2 10:19 ? 00:02:09 /usr/bin/python /usr/bin/neutron-openvswitch-agent --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --config-file=/etc/neutron/neutron.conf --log-file=/var/log/neutron/neutron-plugin-openvswitch-agent.log
root 8265 7897 0 10:19 ? 00:00:00 sudo neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport --format=json
root 8267 8265 0 10:19 ? 00:00:00 /usr/bin/python /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport --format=json
root 8269 8267 0 10:19 ? 00:00:00 /usr/bin/ovsdb-client monitor Interface name,ofport --format=json
root 8543 1 0 09:52 ? 00:00:00 /bin/bash /var/lib/juju/init/jujud-machine-16/exec-start.sh
root 8549 8543 0 09:52 ? 00:00:00 /var/lib/juju/tools/machine-16/jujud machine --data-dir /var/lib/juju --machine-id 16 --debug
root 8607 1 0 09:52 ? 00:00:00 /bin/bash /var/lib/juju/init/jujud-unit-nova-compute-2/exec-start.sh
root 8611 8607 0 09:52 ? 00:00:02 /var/lib/juju/tools/unit-nova-compute-2/jujud unit --data-dir /var/lib/juju --unit-name nova-compute/2 --debug
root 15377 1 0 10:52 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 0f57a93e-7449-448a-a64b-94a06de3581c
100000 15384 15377 0 10:52 ? 00:00:00 init
100000 15493 15384 0 10:52 ? 00:00:00 /sbin/syslogd -m 0
100000 15716 15384 0 10:52 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
uuidd 16562 1 0 10:58 ? 00:00:00 /usr/sbin/uuidd --socket-activation
root 16920 2 0 11:01 ? 00:00:00 [kworker/1:3]
root 16969 867 0 11:02 ? 00:00:00 sshd: ubuntu [priv]
ubuntu 17028 16969 0 11:02 ? 00:00:00 sshd: ubuntu@pts/1
ubuntu 17029 17028 0 11:02 pts/1 00:00:00 -bash
root 17050 17029 0 11:02 pts/1 00:00:00 sudo su
root 17051 17050 0 11:02 pts/1 00:00:00 su
root 17052 17051 0 11:02 pts/1 00:00:00 bash
root 17132 1 0 09:53 ? 00:00:10 /sbin/cgmanager -m name=systemd
root 17296 1 0 11:02 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 98d89f17-f861-4800-b4c1-8e07cb867418
100000 17312 17296 0 11:02 ? 00:00:00 init
100000 17412 17312 0 11:02 ? 00:00:00 /sbin/syslogd -m 0
100000 17478 17312 0 11:02 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
root 17678 1 0 09:53 ? 00:00:00 /usr/bin/lxcfs -f -s -o allow_other /var/lib/lxcfs/
lxc-dns+ 18169 1 0 09:53 ? 00:00:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-in
root 18689 1 0 11:09 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 40bd8478-a0d4-4ebf-bb88-7cdaa25a96c8
root 18697 2 0 11:09 ? 00:00:00 [kworker/0:3]
100000 18709 18689 0 11:09 ? 00:00:00 init
100000 18814 18709 0 11:09 ? 00:00:00 /sbin/syslogd -m 0
100000 18880 18709 0 11:09 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
nova 19280 1 2 11:12 ? 00:00:40 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova-compute.conf --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-compute.log
root 19405 1 0 09:54 ? 00:00:00 /bin/bash /var/lib/juju/init/jujud-unit-ceilometer-agent-0/exec-start.sh
root 19409 19405 0 09:54 ? 00:00:00 /var/lib/juju/tools/unit-ceilometer-agent-0/jujud unit --data-dir /var/lib/juju --unit-name ceilometer-agent/0 --debug
root 19439 1 0 09:54 ? 00:00:00 /bin/bash /var/lib/juju/init/jujud-unit-neutron-openvswitch-0/exec-start.sh
root 19443 19439 0 09:54 ? 00:00:01 /var/lib/juju/tools/unit-neutron-openvswitch-0/jujud unit --data-dir /var/lib/juju --unit-name neutron-openvswitch/0 --debug
root 19549 1 0 11:12 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 6786c602-b82f-4a23-bd1f-59a63539cae5
100000 19563 19549 0 11:12 ? 00:00:00 init
100000 19660 19563 0 11:12 ? 00:00:00 /sbin/syslogd -m 0
100000 19726 19563 0 11:12 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
root 21246 1 0 11:25 ? 00:00:01 /usr/bin/lxd --group lxd --debug --tcp [::]:8443
root 21252 2 0 11:25 ? 00:00:00 [kworker/1:0]
root 21990 1 0 11:26 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 96f97989-159e-4181-8b91-39da572b71db
root 21997 2 0 11:26 ? 00:00:00 [kworker/0:2]
100000 22020 21990 0 11:26 ? 00:00:00 init
100000 22104 22020 0 11:26 ? 00:00:00 /sbin/syslogd -m 0
100000 22181 22020 0 11:26 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
root 22447 1 0 09:55 ? 00:00:00 ovsdb-server: monitoring pid 22448 (healthy)
root 22448 22447 0 09:55 ? 00:00:00 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certifica
root 22459 1 0 09:55 ? 00:00:00 ovs-vswitchd: monitoring pid 22460 (healthy)
root 22460 22459 0 09:55 ? 00:00:07 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --det
root 22545 1 0 11:27 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 35ab12d3-05c3-4cd8-b417-c87236a3821e
100000 22553 22545 0 11:27 ? 00:00:00 init
100000 22657 22553 0 11:27 ? 00:00:00 /sbin/syslogd -m 0
100000 22728 22553 0 11:28 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
root 24128 17052 0 11:38 pts/1 00:00:00 ps -aef

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: nova-compute-lxd 0.5.1-0ubuntu1
ProcVersionSignature: Ubuntu 3.19.0-15.15-generic 3.19.3
Uname: Linux 3.19.0-15-generic x86_64
ApportVersion: 2.17.2-0ubuntu1
Architecture: amd64
Date: Wed Apr 22 11:39:15 2015
Ec2AMI: ami-0000031a
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.medium
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
PackageArchitecture: all
SourcePackage: nova-compute-lxd
UpgradeStatus: No upgrade log present (probably fresh install)

[James Page (james-page)]

I also get residual veth state:

13: vethMY0SC0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:c4:c9:af:e4:83 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fcc4:c9ff:feaf:e483/64 scope link
       valid_lft forever preferred_lft forever
18: vethRP1GL1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:6e:ea:31:b4:f0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc6e:eaff:fe31:b4f0/64 scope link
       valid_lft forever preferred_lft forever
23: veth0TH2LA: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:86:87:39:42:10 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc86:87ff:fe39:4210/64 scope link
       valid_lft forever preferred_lft forever
28: veth7C0G4C: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:f2:cf:fe:13:c4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fcf2:cfff:fefe:13c4/64 scope link
       valid_lft forever preferred_lft forever
33: vethO2EI0J: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:b5:ae:fa:a3:4a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fcb5:aeff:fefa:a34a/64 scope link
       valid_lft forever preferred_lft forever
38: vethTG4UPW: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:2e:7a:ff:07:42 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc2e:7aff:feff:742/64 scope link
       valid_lft forever preferred_lft forever

[James Page (james-page)]

Output from LXD:

Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15 handling DELETE /1.0/containers/551da5cd-f879-42d8-b554-2e0ea6efe8af
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "type": "async",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status": "OK",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status_code": 100,
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "operation": "/1.0/operations/5e74a764-21c0-4636-9365-42bb31e0db80",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "resources": null,
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "metadata": null
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: }
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15 Error cleaning up /var/lib/lxd/lxc/551da5cd-f879-42d8-b554-2e0ea6efe8af: readdirent: no such file or directory
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15 operation %!s(func() shared.OperationResult=0x5470f0) finished: { <nil>}

[James Page (james-page)]

Complete lxd delete logs:

Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15 handling GET /1.0/containers/8b9df5b8-4cab-4115-bfbf-a4c92b7922c6
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15 Configured device eth0
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "type": "sync",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status": "Success",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status_code": 200,
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "metadata": {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "name": "8b9df5b8-4cab-4115-bfbf-a4c92b7922c6",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "profiles": [
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "default"
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: ],
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "config": {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "raw.lxc": "lxc.console.logfile=/var/lib/lxd/lxc/8b9df5b8-4cab-4115-bfbf-a4c92b7922c6/console.log\n",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "volatile.eth0.hwaddr": "00:16:3e:19:ef:68"
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: },
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "userdata": "",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status": {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status": "RUNNING",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status_code": 3,
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "init": 27041,
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "ips": [
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "interface": "eth0",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "protocol": "IPV4",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "address": "10.0.3.52"
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: },
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "interface": "lo",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "protocol": "IPV4",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "address": "127.0.0.1"
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: },
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "interface": "lo",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "protocol": "IPV6",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "address": "::1"
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: }
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: ]
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: },
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "devices": {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "eth0": {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "hwaddr": "00:16:3e:19:ef:68",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "nictype": "bridged",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "parent": "lxcbr0",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "type": "nic"
Apr 22 11:47:15 ju...

[Michał Sawicz (saviq)]

I've just experimented locally with lxd itself and have deleted two instances. While no processes/nics are left behind, the containers are not purged:

$ lxc list --debug
2015/05/06 14:27:00 fingering the daemon
2015/05/06 14:27:00 raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"api_compat":1,"auth":"trusted","config":{"trust-password":false},"environment":{"backing_fs":"btrfs","driver":"lxc","kernel_version":"3.19.0-16-generic","lxc_version":"1.1.2"}}}
2015/05/06 14:27:00 pong received
2015/05/06 14:27:00 raw response: {"type":"sync","status":"Success","status_code":200,"metadata":[]}
+------+-------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | EPHEMERAL |
+------+-------+------+------+-----------+
+------+-------+------+------+-----------+

$ lxc launch images:ubuntu/vivid/i386 ubuntu-32-2
Creating container...error: Container exists

$ ls /var/lib/lxd/lxc/
lxc-monitord.log ubuntu-32 ubuntu-32-2

Not sure if it's the same for you, but for me it's likely a systemd/lxc/btrfs issue:
$ btrfs sub list .
[...]
ID 586 gen 303191 top level 560 path var/lib/lxd/lxc/ubuntu-32/rootfs/var/lib/machines
ID 587 gen 303187 top level 560 path var/lib/lxd/lxc/ubuntu-32-2/rootfs/var/lib/machines

Looks like the database isn't cleared either:

$ sqlite3 lxd.db .dump
INSERT INTO "containers_config" VALUES(1,1,'volatile.eth0.hwaddr','00:16:3e:c0:c9:ce');
INSERT INTO "containers_config" VALUES(2,2,'volatile.eth0.hwaddr','00:16:3e:ab:ac:24');
[...]
INSERT INTO "containers_profiles" VALUES(1,1,1,1);
INSERT INTO "containers_profiles" VALUES(2,2,1,1);

[Michał Sawicz (saviq)]

Related: https://lists.linuxcontainers.org/pipermail/lxc-devel/2015-February/011367.html

[Michał Sawicz (saviq)]

OK the database leftovers are https://github.com/lxc/lxd/issues/570, not sure if your issue could be btrfs-related?

[Stéphane Graber (stgraber)]

The LXD side has been fixed.