Terminated LXD instances processes stick around

Bug #1447127 reported by James Page
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
lxd (Ubuntu)
Fix Released
Undecided
Unassigned
nova-compute-lxd (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

I spun up a number of lxd instances using the lxd driver and then deleted them; the instances disappear from lxc list, but processes remain:

ceilome+ 4050 1 0 10:18 ? 00:00:10 /usr/bin/python /usr/bin/ceilometer-agent-compute --config-file=/etc/ceilometer/ceilometer.conf --log-file=/var/log/ceilometer/ceilometer-agent-compute.log
neutron 7897 1 2 10:19 ? 00:02:09 /usr/bin/python /usr/bin/neutron-openvswitch-agent --config-file=/etc/neutron/plugins/ml2/ml2_conf.ini --config-file=/etc/neutron/neutron.conf --log-file=/var/log/neutron/neutron-plugin-openvswitch-agent.log
root 8265 7897 0 10:19 ? 00:00:00 sudo neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport --format=json
root 8267 8265 0 10:19 ? 00:00:00 /usr/bin/python /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ovsdb-client monitor Interface name,ofport --format=json
root 8269 8267 0 10:19 ? 00:00:00 /usr/bin/ovsdb-client monitor Interface name,ofport --format=json
root 8543 1 0 09:52 ? 00:00:00 /bin/bash /var/lib/juju/init/jujud-machine-16/exec-start.sh
root 8549 8543 0 09:52 ? 00:00:00 /var/lib/juju/tools/machine-16/jujud machine --data-dir /var/lib/juju --machine-id 16 --debug
root 8607 1 0 09:52 ? 00:00:00 /bin/bash /var/lib/juju/init/jujud-unit-nova-compute-2/exec-start.sh
root 8611 8607 0 09:52 ? 00:00:02 /var/lib/juju/tools/unit-nova-compute-2/jujud unit --data-dir /var/lib/juju --unit-name nova-compute/2 --debug
root 15377 1 0 10:52 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 0f57a93e-7449-448a-a64b-94a06de3581c
100000 15384 15377 0 10:52 ? 00:00:00 init
100000 15493 15384 0 10:52 ? 00:00:00 /sbin/syslogd -m 0
100000 15716 15384 0 10:52 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
uuidd 16562 1 0 10:58 ? 00:00:00 /usr/sbin/uuidd --socket-activation
root 16920 2 0 11:01 ? 00:00:00 [kworker/1:3]
root 16969 867 0 11:02 ? 00:00:00 sshd: ubuntu [priv]
ubuntu 17028 16969 0 11:02 ? 00:00:00 sshd: ubuntu@pts/1
ubuntu 17029 17028 0 11:02 pts/1 00:00:00 -bash
root 17050 17029 0 11:02 pts/1 00:00:00 sudo su
root 17051 17050 0 11:02 pts/1 00:00:00 su
root 17052 17051 0 11:02 pts/1 00:00:00 bash
root 17132 1 0 09:53 ? 00:00:10 /sbin/cgmanager -m name=systemd
root 17296 1 0 11:02 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 98d89f17-f861-4800-b4c1-8e07cb867418
100000 17312 17296 0 11:02 ? 00:00:00 init
100000 17412 17312 0 11:02 ? 00:00:00 /sbin/syslogd -m 0
100000 17478 17312 0 11:02 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
root 17678 1 0 09:53 ? 00:00:00 /usr/bin/lxcfs -f -s -o allow_other /var/lib/lxcfs/
lxc-dns+ 18169 1 0 09:53 ? 00:00:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-in
root 18689 1 0 11:09 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 40bd8478-a0d4-4ebf-bb88-7cdaa25a96c8
root 18697 2 0 11:09 ? 00:00:00 [kworker/0:3]
100000 18709 18689 0 11:09 ? 00:00:00 init
100000 18814 18709 0 11:09 ? 00:00:00 /sbin/syslogd -m 0
100000 18880 18709 0 11:09 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
nova 19280 1 2 11:12 ? 00:00:40 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova-compute.conf --config-file=/etc/nova/nova.conf --log-file=/var/log/nova/nova-compute.log
root 19405 1 0 09:54 ? 00:00:00 /bin/bash /var/lib/juju/init/jujud-unit-ceilometer-agent-0/exec-start.sh
root 19409 19405 0 09:54 ? 00:00:00 /var/lib/juju/tools/unit-ceilometer-agent-0/jujud unit --data-dir /var/lib/juju --unit-name ceilometer-agent/0 --debug
root 19439 1 0 09:54 ? 00:00:00 /bin/bash /var/lib/juju/init/jujud-unit-neutron-openvswitch-0/exec-start.sh
root 19443 19439 0 09:54 ? 00:00:01 /var/lib/juju/tools/unit-neutron-openvswitch-0/jujud unit --data-dir /var/lib/juju --unit-name neutron-openvswitch/0 --debug
root 19549 1 0 11:12 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 6786c602-b82f-4a23-bd1f-59a63539cae5
100000 19563 19549 0 11:12 ? 00:00:00 init
100000 19660 19563 0 11:12 ? 00:00:00 /sbin/syslogd -m 0
100000 19726 19563 0 11:12 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
root 21246 1 0 11:25 ? 00:00:01 /usr/bin/lxd --group lxd --debug --tcp [::]:8443
root 21252 2 0 11:25 ? 00:00:00 [kworker/1:0]
root 21990 1 0 11:26 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 96f97989-159e-4181-8b91-39da572b71db
root 21997 2 0 11:26 ? 00:00:00 [kworker/0:2]
100000 22020 21990 0 11:26 ? 00:00:00 init
100000 22104 22020 0 11:26 ? 00:00:00 /sbin/syslogd -m 0
100000 22181 22020 0 11:26 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
root 22447 1 0 09:55 ? 00:00:00 ovsdb-server: monitoring pid 22448 (healthy)
root 22448 22447 0 09:55 ? 00:00:00 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certifica
root 22459 1 0 09:55 ? 00:00:00 ovs-vswitchd: monitoring pid 22460 (healthy)
root 22460 22459 0 09:55 ? 00:00:07 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --det
root 22545 1 0 11:27 ? 00:00:00 [lxc monitor] /var/lib/lxd/lxc 35ab12d3-05c3-4cd8-b417-c87236a3821e
100000 22553 22545 0 11:27 ? 00:00:00 init
100000 22657 22553 0 11:27 ? 00:00:00 /sbin/syslogd -m 0
100000 22728 22553 0 11:28 ? 00:00:00 udhcpc -p /var/run/udhcpc.eth0.pid -R -n -T 60 -i eth0 -s /sbin/cirros-dhcpc -O mtu -O staticroutes -x hostname cirros
root 24128 17052 0 11:38 pts/1 00:00:00 ps -aef

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: nova-compute-lxd 0.5.1-0ubuntu1
ProcVersionSignature: Ubuntu 3.19.0-15.15-generic 3.19.3
Uname: Linux 3.19.0-15-generic x86_64
ApportVersion: 2.17.2-0ubuntu1
Architecture: amd64
Date: Wed Apr 22 11:39:15 2015
Ec2AMI: ami-0000031a
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.medium
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
PackageArchitecture: all
SourcePackage: nova-compute-lxd
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

I also get residual veth state:

13: vethMY0SC0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:c4:c9:af:e4:83 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fcc4:c9ff:feaf:e483/64 scope link
       valid_lft forever preferred_lft forever
18: vethRP1GL1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:6e:ea:31:b4:f0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc6e:eaff:fe31:b4f0/64 scope link
       valid_lft forever preferred_lft forever
23: veth0TH2LA: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:86:87:39:42:10 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc86:87ff:fe39:4210/64 scope link
       valid_lft forever preferred_lft forever
28: veth7C0G4C: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:f2:cf:fe:13:c4 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fcf2:cfff:fefe:13c4/64 scope link
       valid_lft forever preferred_lft forever
33: vethO2EI0J: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:b5:ae:fa:a3:4a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fcb5:aeff:fefa:a34a/64 scope link
       valid_lft forever preferred_lft forever
38: vethTG4UPW: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master lxcbr0 state UP group default qlen 1000
    link/ether fe:2e:7a:ff:07:42 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc2e:7aff:feff:742/64 scope link
       valid_lft forever preferred_lft forever

Revision history for this message
James Page (james-page) wrote :

Output from LXD:

Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15 handling DELETE /1.0/containers/551da5cd-f879-42d8-b554-2e0ea6efe8af
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "type": "async",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status": "OK",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status_code": 100,
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "operation": "/1.0/operations/5e74a764-21c0-4636-9365-42bb31e0db80",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "resources": null,
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "metadata": null
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: }
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15 Error cleaning up /var/lib/lxd/lxc/551da5cd-f879-42d8-b554-2e0ea6efe8af: readdirent: no such file or directory
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15 operation %!s(func() shared.OperationResult=0x5470f0) finished: { <nil>}

Revision history for this message
James Page (james-page) wrote :
Download full text (5.2 KiB)

Complete lxd delete logs:

Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15 handling GET /1.0/containers/8b9df5b8-4cab-4115-bfbf-a4c92b7922c6
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15 Configured device eth0
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: 2015/04/22 11:47:15
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "type": "sync",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status": "Success",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status_code": 200,
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "metadata": {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "name": "8b9df5b8-4cab-4115-bfbf-a4c92b7922c6",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "profiles": [
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "default"
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: ],
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "config": {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "raw.lxc": "lxc.console.logfile=/var/lib/lxd/lxc/8b9df5b8-4cab-4115-bfbf-a4c92b7922c6/console.log\n",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "volatile.eth0.hwaddr": "00:16:3e:19:ef:68"
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: },
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "userdata": "",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status": {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status": "RUNNING",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "status_code": 3,
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "init": 27041,
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "ips": [
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "interface": "eth0",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "protocol": "IPV4",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "address": "10.0.3.52"
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: },
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "interface": "lo",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "protocol": "IPV4",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "address": "127.0.0.1"
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: },
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "interface": "lo",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "protocol": "IPV6",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "address": "::1"
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: }
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: ]
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: },
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "devices": {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "eth0": {
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "hwaddr": "00:16:3e:19:ef:68",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "nictype": "bridged",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "parent": "lxcbr0",
Apr 22 11:47:15 juju-devel3-machine-16 lxd[21246]: "type": "nic"
Apr 22 11:47:15 ju...

Read more...

Revision history for this message
Michał Sawicz (saviq) wrote :

I've just experimented locally with lxd itself and have deleted two instances. While no processes/nics are left behind, the containers are not purged:

$ lxc list --debug
2015/05/06 14:27:00 fingering the daemon
2015/05/06 14:27:00 raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"api_compat":1,"auth":"trusted","config":{"trust-password":false},"environment":{"backing_fs":"btrfs","driver":"lxc","kernel_version":"3.19.0-16-generic","lxc_version":"1.1.2"}}}
2015/05/06 14:27:00 pong received
2015/05/06 14:27:00 raw response: {"type":"sync","status":"Success","status_code":200,"metadata":[]}
+------+-------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | EPHEMERAL |
+------+-------+------+------+-----------+
+------+-------+------+------+-----------+

$ lxc launch images:ubuntu/vivid/i386 ubuntu-32-2
Creating container...error: Container exists

$ ls /var/lib/lxd/lxc/
lxc-monitord.log ubuntu-32 ubuntu-32-2

Not sure if it's the same for you, but for me it's likely a systemd/lxc/btrfs issue:
$ btrfs sub list .
[...]
ID 586 gen 303191 top level 560 path var/lib/lxd/lxc/ubuntu-32/rootfs/var/lib/machines
ID 587 gen 303187 top level 560 path var/lib/lxd/lxc/ubuntu-32-2/rootfs/var/lib/machines

Looks like the database isn't cleared either:

$ sqlite3 lxd.db .dump
INSERT INTO "containers_config" VALUES(1,1,'volatile.eth0.hwaddr','00:16:3e:c0:c9:ce');
INSERT INTO "containers_config" VALUES(2,2,'volatile.eth0.hwaddr','00:16:3e:ab:ac:24');
[...]
INSERT INTO "containers_profiles" VALUES(1,1,1,1);
INSERT INTO "containers_profiles" VALUES(2,2,1,1);

Revision history for this message
Michał Sawicz (saviq) wrote :
Revision history for this message
Michał Sawicz (saviq) wrote :

OK the database leftovers are https://github.com/lxc/lxd/issues/570, not sure if your issue could be btrfs-related?

Revision history for this message
Stéphane Graber (stgraber) wrote :

The LXD side has been fixed.

Changed in lxd (Ubuntu):
status: New → Fix Released
Chuck Short (zulcss)
Changed in nova-compute-lxd (Ubuntu):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.