Machine not getting removed

[Casey Marshall]

I have a machine that's stuck in a staging model on a 2.4.3 controller (prodstack-is). I've tried `juju remove-machine 21` and `juju remove-machine 21 --force` many times, but the machine is not getting removed. I also tried shutting down the instance and removing it with nova, no luck.

(mojo-stg-omnibus)stg-omnibus@wendigo:~$ juju status
Model Controller Cloud/Region Version SLA Timestamp
stg-omnibus prodstack-is prodstack-45/bootstack-ps45 2.4.3 unsupported 21:23:45Z
...
Machine State DNS Inst id Series AZ Message
...
21 stopped 10.50.144.24 4a3bb491-733e-46d3-a7cc-89a8e7acc5cb xenial prodstack-zone-1 ACTIVE
...

[Casey Marshall (cmars)]

Ok, more background on this...

There was storage previously attached to the unit that was placed onto this machine.

I removed the unit, but never removed the storage that was attached to it. The unit went into dying state and eventually removed, but the machine was stuck in a "stopped" state.

I noticed the storage left behind by this unit, removed it, and the machine promptly died and Juju cleaned it up.

So, I think this is still a bug, but there is a potential workaround worth documenting here, in case anyone gets into a similar predicament: if you have a machine that doesn't die, check to make sure the storage that was attached to it is also removed.

[Casey Marshall (cmars)]

May be related to https://bugs.launchpad.net/juju/+bug/1734287

[Ian Booth (wallyworld)]

Sometimes the cloud can encounter errors removing / detaching storage. The way forward is to manually fix the issue, but as you say, Juju needs to surface this better to allow to user to act. On that basis, and given what was done to resolve the problem, I'll mark this as a dup of bug 1734287