[2.5.2-edge] Unable to remove pending machines

[Dmitrii Shcherbakov]

Had some tag constraints applied which were not possible to satisfy in MAAS and decided to destroy model but failed.

snap info juju | grep installed:
installed: 2.5.2+2.5-d7d3bba (6655) 71MB classic

mongodump:
https://private-fileshare.canonical.com/~dima/juju-dumps/mongodump-1550269783.tar.gz

juju status
Model Controller Cloud/Region Version SLA Timestamp Notes
default samaas samaas 2.5.2 unsupported 17:45:30-05:00 attempt 1 to destroy model failed (will retry): model not empty, found 8 machines, 9 applications (model not empty)

App Version Status Scale Charm Store Rev OS Notes
apache2 waiting 0/1 apache2 jujucharms 26 ubuntu exposed
elasticsearch waiting 0/1 elasticsearch jujucharms 32 ubuntu
etcd waiting 0/3 etcd jujucharms 378 ubuntu
grafana waiting 0/1 grafana jujucharms 27 ubuntu exposed
graylog waiting 0/1 graylog jujucharms 23 ubuntu
kubeapi-load-balancer waiting 0/1 kubeapi-load-balancer jujucharms 525 ubuntu exposed
kubernetes-master waiting 0/1 kubernetes-master jujucharms 542 ubuntu
kubernetes-worker waiting 0/1 kubernetes-worker jujucharms 398 ubuntu exposed
prometheus waiting 0/1 prometheus2 jujucharms 8 ubuntu

Unit Workload Agent Machine Public address Ports Message
apache2/0 waiting allocating 4 waiting for machine
elasticsearch/0 waiting allocating 4 waiting for machine
etcd/0 waiting allocating 0 waiting for machine
etcd/1 waiting allocating 1 waiting for machine
etcd/2 waiting allocating 2 waiting for machine
grafana/0 waiting allocating 4 waiting for machine
graylog/0 waiting allocating 4 waiting for machine
kubeapi-load-balancer/0 waiting allocating 1 waiting for machine
kubernetes-master/0 waiting allocating 0 waiting for machine
kubernetes-worker/0 waiting allocating 2 waiting for machine
prometheus/0 waiting allocating 4 waiting for machine

Machine State DNS Inst id Series AZ Message
0 down pending bionic No available machine matches constraints: [('agent_name', ['789afd30-e332-486a-89b1-cf53079d2b90']), ('tags', ['bootstrap']), ('zone', ['default'])] (resolved to "tags=bootstrap zone=default")
1 down pending bionic No available machine matches constraints: [('agent_name', ['789afd30-e332-486a-89b1-cf53079d2b90']), ('interfaces', ['proxy:space=8;loadbalancer:space=8;certificates:space=8;cluster:space=8;db:space=8;apiserver:space=8;nrpe-external-master:space=8;website:space=8;0:space=8']), ('tags', ['bootstrap']), ('zone', ['default'])] (resolved to "interfaces=proxy:space=8;loadbalancer:space=8;certificates:space=8;cluster:space=8;db:space=8;apiserver:space=8;nrpe-external-master:space=8;website:space=8;0:space=8 tags=bootstrap zone=default")
2 pending pending bionic
3 pending pending bionic
4 pending pending bionic
5 pending pending bionic
6 pending pending bionic
7 pending pending bionic

d error: failed to process updated machines: failed to ensure machine dead id:0, details:0: machine 0 has unit "etcd/0" assigned
447e79a8-3080-4928-8174-b340dbdad034: machine-0 2019-02-15 23:44:17 ERROR juju.worker.dependency engine.go:636 "firewaller" manifold worker returned unexpected error: machine 0 not provisioned
447e79a8-3080-4928-8174-b340dbdad034: machine-0 2019-02-15 23:44:19 ERROR juju.worker.dependency engine.go:636 "charm-revision-updater" manifold worker returned unexpected error: setting the resource failed
447e79a8-3080-4928-8174-b340dbdad034: machine-0 2019-02-15 23:44:20 ERROR juju.worker.dependency engine.go:636 "compute-provisioner" manifold worker returned unexpected error: failed to process updated machines: failed to ensure machine dead id:0, details:0: machine 0 has unit "etcd/0" assigned
447e79a8-3080-4928-8174-b340dbdad034: machine-0 2019-02-15 23:44:20 WARNING juju.state cleanup.go:161 cleanup failed in model 789afd30-e332-486a-89b1-cf53079d2b90 for units("apache2"): cannot destroy unit "apache2/0": machine 4 not provisioned

https://paste.ubuntu.com/p/7yNkBN23kw/

[Anastasia (anastasia-macmood)]

I think this is a duplicate of bug # 1814271. I'll mark it as such.

While testing the fix for it, as per my comment on that bug, I did test a scenario where machines were in pending status and never started. --force has worked for me.

Please retry with Juju 2.6.