This looks specific to the case where you also specify tags as constraints but I managed to reproduce it even by just using an AZ constraint.



The first batch of tests makes it feel like tags are a problem:

# success
juju add-machine --constraints zones=AZ1
juju add-machine --constraints zones=AZ2
juju add-machine --constraints zones=AZ3

# immediate failure
juju add-machine --constraints 'tags=control-openstack zones=AZ1'
created machine 3

juju status
Model      Controller        Cloud/Region      Version  SLA          Timestamp
openstack  foundations-maas  foundations-maas  2.5.2    unsupported  01:52:35Z

Machine  State    DNS             Inst id  Series  AZ   Message
0        pending  192.0.2.159  xfyxt6   bionic  AZ1  Deploying: 'cloudinit' searching for network datasources
1        pending  192.0.2.161  qexg6r   bionic  AZ2  Deploying: /images/ubuntu/amd64/ga-18.04/bionic/daily/squashfs
2        pending  192.0.2.163  836e34   bionic  AZ3  Deploying: /images/ubuntu/amd64/ga-18.04/bionic/daily/boot-initrd
3        down                     pending  bionic       suitable availability zone for machine 3 not found

I tried many different tags and there are definitely free nodes out there in the right AZ and with the right tag. As soon as a tag constraint was added juju immediately failed with the message above (no retries, retry-provisioning doesn't work as well).

2019-03-14 02:01:41 ERROR juju.provisioner provisioner_task.go:1178 cannot start instance for machine "<num>": suitable availability zone for machine <num> not found

I then did some random testing as below and managed to make juju fail with adding nodes in AZ1 with `juju add-machine --constraints 'zones=AZ1'` (see the screenshot - there are quite a few of the machines ready in AZ1):

juju status
Model      Controller        Cloud/Region      Version  SLA          Timestamp
openstack  foundations-maas  foundations-maas  2.5.2    unsupported  02:03:48Z

Machine  State    DNS             Inst id  Series  AZ   Message
0        started  192.0.2.159  xfyxt6   bionic  AZ1  Deployed
1        started  192.0.2.161  qexg6r   bionic  AZ2  Deployed
2        started  192.0.2.163  836e34   bionic  AZ3  Deployed
3        down                     pending  bionic       suitable availability zone for machine 3 not found
4        down                     pending  bionic       suitable availability zone for machine 4 not found
5        down                     pending  bionic       suitable availability zone for machine 5 not found
6        down                     pending  bionic       suitable availability zone for machine 6 not found
7        down                     pending  bionic       suitable availability zone for machine 7 not found
8        down                     pending  bionic       suitable availability zone for machine 8 not found
9        down                     pending  bionic       suitable availability zone for machine 9 not found
10       pending  192.0.2.164  pgk84a   bionic  AZ2  Deploying: Power state queried: on
11       down                     pending  bionic       suitable availability zone for machine 11 not found
12       down                     pending  bionic       suitable availability zone for machine 12 not found
13       down                     pending  bionic       suitable availability zone for machine 13 not found
14       down                     pending  bionic       suitable availability zone for machine 14 not found
15       pending  192.0.2.165  ystyka   bionic  AZ2  Deploying
16       pending  192.0.2.166  tn7ymk   bionic  AZ3  Deploying
17       down                     pending  bionic       suitable availability zone for machine 17 not found
18       down                     pending  bionic       suitable availability zone for machine 18 not found


history | grep add-machine
 2140  juju add-machine --constraints zones=AZ1
 2143  juju add-machine --constraints zones=AZ1
 2144  juju add-machine --constraints zones=AZ2
 2145  juju add-machine --constraints zones=AZ3
 2148  juju add-machine --constraints tags=control-openstack,control-1 zones=AZ1
 2149  juju add-machine --constraints 'tags=control-openstack,control-1 zones=AZ1'
 2151  #juju add-machine --constraints 'tags=control-openstack zones=AZ1'
 2153  juju add-machine --constraints 'tags=control-openstack zones=AZ1'
 2155  juju add-machine --constraints 'zones=AZ1 tags=control-openstack'
 2157  juju add-machine --constraints 'zones=AZ1 tags=compute'
 2159  juju add-machine --constraints 'zones=AZ1'
 2161  juju add-machine --constraints 'zones=AZ1'
 2163  juju add-machine --constraints 'zones=AZ1'
 2165  juju add-machine --constraints 'zones=AZ2'
 2167  juju add-machine --constraints 'zones=AZ1'
 2169  juju add-machine --constraints 'zones=az1'
 2171  juju add-machine --constraints 'zones=az1'
 2175  juju add-machine --constraints 'zones=az2'
 2177  juju add-machine --constraints 'zones=AZ2'
 2179  juju add-machine --constraints 'zones=AZ3'
 2180  juju add-machine --constraints 'zones=AZ1'
 2182  juju add-machine --constraints 'zones=AZ1'
 2184  history | grep add-machine