lxd provider: juju destroy-controller hangs

[Casey Marshall]

When destroying my LXD-bootstrapped environment, the command hangs:

---

$ juju destroy-controller lxd
WARNING! This command will destroy the "lxd" controller.
This includes all machines, services, data and other resources.

Continue [y/N]? y
Destroying controller "lxd"

---

`lxc list` reveals that the controller (juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-0) container was deleted, but the other machines were not:

$ lxc list
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | EPHEMERAL | SNAPSHOTS |
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+
| aseprite-build | STOPPED | | | NO | 0 |
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+
| hkpdev | STOPPED | | | NO | 0 |
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+
| juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-1 | STOPPED | | | NO | 0 |
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+
| juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-2 | RUNNING | 10.0.3.218 (eth0) | | NO | 0 |
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+
| juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-3 | RUNNING | 10.0.3.167 (eth0) | | NO | 0 |
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+
| juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-4 | RUNNING | 10.0.3.37 (eth0) | | NO | 0 |
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+
| juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-5 | RUNNING | 10.0.3.139 (eth0) | | NO | 0 |
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+
| juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-6 | STOPPED | | | NO | 0 |
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+
| reactdev | STOPPED | | | NO | 0 |
+-----------------------------------------------------+---------+-------------------+------+-----------+-----------+

[Casey Marshall (cmars)]

Here's my recent /var/log/lxd.log:

t=2016-02-01T16:08:07-0600 lvl=info msg=handling ip=10.0.3.1:47410 method=GET url=/1.0/operations/d2885ddc-24fc-4813-b381-ade1eb22045c/wait
t=2016-02-01T16:08:08-0600 lvl=info msg=handling method=GET url="/internal/containers/512/onstop?target=unknown" ip=@
t=2016-02-01T16:08:09-0600 lvl=eror msg="Running apparmor" action=R output="apparmor_parser: Unable to remove \"lxd-juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-0_</var/lib/lxd>\". Profile doesn't exist\n" err="exit status 254"
t=2016-02-01T16:08:09-0600 lvl=info msg=handling method=DELETE url=/1.0/containers/juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-0 ip=10.0.3.1:47410
t=2016-02-01T16:08:09-0600 lvl=info msg=handling method=GET url=/1.0/operations/776478a9-e842-4fea-8ae3-f72a75c2a08d/wait ip=10.0.3.1:47410
t=2016-02-01T16:08:09-0600 lvl=info msg=handling url=/1.0 ip=10.0.3.1:47414 method=GET
t=2016-02-01T16:08:10-0600 lvl=info msg=handling method=GET url=/1.0/profiles ip=10.0.3.1:47414
t=2016-02-01T16:08:10-0600 lvl=info msg=handling method=GET url="/1.0/containers?recursion=1" ip=10.0.3.1:47414
t=2016-02-01T16:08:12-0600 lvl=info msg=handling ip=10.0.3.1:47410 method=GET url=/1.0/containers/juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-1
t=2016-02-01T16:08:12-0600 lvl=info msg=handling method=PUT url=/1.0/containers/juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-1/state ip=10.0.3.1:47410
t=2016-02-01T16:08:12-0600 lvl=info msg=handling method=GET url=/1.0/operations/913b9d3a-6d2b-469a-ac9a-64a3641e8166/wait ip=10.0.3.1:47410
t=2016-02-01T16:08:13-0600 lvl=info msg=handling method=GET url="/internal/containers/513/onstop?target=unknown" ip=@
t=2016-02-01T16:08:13-0600 lvl=eror msg="Running apparmor" action=R output="apparmor_parser: Unable to remove \"lxd-juju-e4b833a5-fe21-413c-8ae6-170a582f0575-machine-1_</var/lib/lxd>\". Profile doesn't exist\n" err="exit status 254"
t=2016-02-01T16:17:01-0600 lvl=info msg=handling url="/1.0/images?recursion=1" ip=@ method=GET

Looks to me like machine 0 is getting removed first.

[Casey Marshall (cmars)]

http://reviews.vapour.ws/r/3797/ may fix this

[John A Meinel (jameinel)]

Doesn't the multi-environment code already have logic around "kill the controllers last"? I'm wondering if this wasn't in place when you ran into it.

Also, I've noticed that if you ever do have an environment where you destroy the controller out of band, then run "juju kill-controller" it will hang for a while, until it gives up trying to talk to the API server and goes and cleans up anyway. However, I think it doesn't give good feedback while it is waiting, so appears to hang. (minutes, IIRC)

[Andrew Wilkins (axwalk)]

I routinely destroy-controller with lxd, have not hit any such issues. Casey, does this still affect you with -beta3?

[Casey Marshall (cmars)]

I haven't seen this with beta3 or newer. I think we can close / mark invalid.