deploying juju on top of ussuri-ovn openstack results in error "ERROR failed to bootstrap model: cannot start bootstrap instance: cannot allocate a public IP as needed: could not find an external network in availability zone [] not found", unless external-network is supplied

[Jason Hobbs]

2020-08-11-19:11:44 root DEBUG ['juju', 'bootstrap', '--config', 'bootstrap-timeout=1800', '--model-default', '/home/ubuntu/project/generated/model_defaults.yaml', 'openstack_cloud', 'foundation-openstack']
2020-08-11-19:11:44 root DEBUG [localhost]: juju bootstrap --config bootstrap-timeout=1800 --model-default /home/ubuntu/project/generated/model_defaults.yaml openstack_cloud foundation-openstack
Creating Juju controller "foundation-openstack" on openstack_cloud/RegionOne
Looking for packaged Juju agent version 2.8.1 for amd64
Launching controller instance(s) on openstack_cloud/RegionOne...
ERROR failed to bootstrap model: cannot start bootstrap instance: cannot allocate a public IP as needed: could not find an external network in availability zone [] not found
2020-08-11-19:12:31 root ERROR [localhost] Command failed: juju bootstrap --config bootstrap-timeout=1800 --model-default /home/ubuntu/project/generated/model_defaults.yaml openstack_cloud foundation-openstack
2020-08-11-19:12:31 root ERROR [localhost] STDOUT follows:
Creating Juju controller "foundation-openstack" on openstack_cloud/RegionOne
Looking for packaged Juju agent version 2.8.1 for amd64
Launching controller instance(s) on openstack_cloud/RegionOne...
ERROR failed to bootstrap model: cannot start bootstrap instance: cannot allocate a public IP as needed: could not find an external network in availability zone [] not found

model defaults:
ubuntu@production-cpe-07d59674-95e9-4406-9d05-8f25282eb455:~/project$ cat generated/model_defaults.yaml
apt-http-proxy: http://squid.internal:3128/
apt-https-proxy: http://squid.internal:3128/
automatically-retry-hooks: false
logging-config: <root>=DEBUG
network: cd2e264c-5c66-4fb7-89f7-9498fceff487
snap-http-proxy: http://squid.internal:3128/
snap-https-proxy: http://squid.internal:3128/
use-floating-ip: true

openstack network info, showing the network above is connected to a single external network with a router:
http://paste.ubuntu.com/p/yNkGwrph5c/

This setup worked on ussuri-ovs so I think it's a OVN related issue. Since there is only a single external network connected to the network given, juju should be able to discover it and use it.

It works if we provide the external network ID in model_defaults.yaml.

[Pen Gale (pengale)]

This feels like a dupe of #1891561, probably due to the same issue w/ Rocky+ accepting an empty string on searches.

That issue is fixed in the latest candidate. If you see this again, please re-open/unmark as a dupe.

[Jason Hobbs (jason-hobbs)]

We saw this in 2.8.5 https://solutions.qa.canonical.com/testruns/testRun/028033e4-e5e0-424c-9f95-d592ad164032

[Jason Hobbs (jason-hobbs)]

sub'd to field high as it has caused failures during release testing.

[Pen Gale (pengale)]

Triaging and adding to 2.8.7 milestone. Agree on the field high classification, and this is in our queue of next up things to work on.

[Heather Lanigan (hmlanigan)]

Juju does attempt to find an external network if none is specified. However it checks the network availability zones to match the network with the external network. Having an AZ list of 0 was not accounted for, working to resolve the problem.

[Heather Lanigan (hmlanigan)]

https://github.com/juju/juju/pull/12276