Cannot use MicroStack as a backing cloud with Juju

[Anthony Dillon]

This issue was originally filed on the website repo: https://github.com/canonical-web-and-design/microstack.run/issues/42.

## Summary

Bootstrap don't work on Ubuntu 20.4 and juju 2.8.0

## Process

Bootstrap step don't work.

## Current and expected result

ERROR failed to bootstrap model: cannot start bootstrap instance: no usable availability zones

## Screenshot

[If relevant, include a screenshot.]

## Browser details

11:37:45 INFO juju.cmd supercommand.go:91 running juju [2.8.0 0 d816abe62fbf6787974e5c4e140818ca08586e44 gc go1.14.4]
11:37:45 DEBUG juju.cmd supercommand.go:92 args: []string{"/snap/juju/12370/bin/juju", "bootstrap", "--bootstrap-series=bionic", "--metadata-source=~/simplestreams", "--model-default", "network=test", "--model-default", "external-network=external", "--model-default", "use-floating-ip=true", "--debug", "microstack", "msk"}
11:37:45 DEBUG juju.cmd.juju.commands bootstrap.go:1133 authenticating with region "" and credential "admin" ()
11:37:45 DEBUG juju.cmd.juju.commands bootstrap.go:1261 provider attrs: map[external-network:external network:test policy-target-group: use-default-secgroup:false use-floating-ip:true use-openstack-gbp:false]
11:37:46 INFO cmd authkeys.go:114 Adding contents of "/home/netipos/.local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
11:37:46 DEBUG juju.cmd.juju.commands bootstrap.go:1320 preparing controller with config: map[agent-metadata-url: agent-stream:released apt-ftp-proxy: apt-http-proxy: apt-https-proxy: apt-mirror: apt-no-proxy: authorized-keys:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDzk/7a76/xqP4a6mMy0vOiRgmnrGBl2mk5ZitgFUFZZASAIQNZRJ6qTaAfixcF98xQ+ZyjEYdi7Tov7SdMWdAoXx90ZbKBG7t18cz+O4oRfNh27vU/4t9+6YCGyIOniI+vtP0ZunMui0lN0km/8y+KLrmd5e/xjGbKA36EnIkzQjQwkRTiD9JoZx+iqIlCvrVp/2SMX0ThcOyy/JTLEdqxdDV3OYYhhyMOKpMyylwacBjPTQR5ua85/egcgVfcAVjGb+hizt2wzbNtzACLIKmiXhfEzgLJVoTSoqS4/mU/17QQVXOM0RFJeKb/aqDXl1TshGQVVc8Py8bu28W1JgP juju-client-key
 automatically-retry-hooks:true backup-dir: cloudinit-userdata: container-image-metadata-url: container-image-stream:released container-inherit-properties: container-networking-method: default-series:bionic default-space: development:false disable-network-management:false egress-subnets: enable-os-refresh-update:true enable-os-upgrade:true external-network:external fan-config: firewall-mode:instance ftp-proxy: http-proxy: https-proxy: ignore-machine-addresses:false image-metadata-url: image-stream:released juju-ftp-proxy: juju-http-proxy: juju-https-proxy: juju-no-proxy:127.0.0.1,localhost,::1 logforward-enabled:false logging-config: lxd-snap-channel:latest/stable max-action-results-age:336h max-action-results-size:5G max-status-history-age:336h max-status-history-size:5G name:controller net-bond-reconfigure-delay:17 network:test no-proxy:127.0.0.1,localhost,::1 policy-target-group: provisioner-harvest-mode:destroyed proxy-ssh:false resource-tags: snap-http-proxy: snap-https-proxy: snap-store-assertions: snap-store-proxy: snap-store-proxy-url: ssl-hostname-verification:true test-mode:false transmit-vendor-metrics:true type:openstack update-status-hook-interval:5m use-default-secgroup:false use-floating-ip:true use-openstack-gbp:false uuid:0fff4778-53c8-444c-8f98-29da2edf1704]
11:37:46 INFO juju.provider.openstack provider.go:166 opening model "controller"
11:37:47 DEBUG goose client.go:612 auth details: &{Token:gAAAAABe-IELXDIG9uejTqlSCIIf4V6X0tiv2jahdH_i_RY3hEGUjWw3FnBiqUML9_vVk3y3mpafvG6dUjsyWqXMYPeDCr4nosZjeH4AOY1V_s_2VcuV5h_4Ie7f0dnWu3hws9b_5HSna7d5B7xOHGA4CezZul0jUgPFe3FCD4HsvixMGjJ7GTc TenantId:c2ae47c09d374c788af1615ff8235923 TenantName:admin UserId:d3c0d8eb8f084b91901e3b30fb6571d7 Domain: RegionServiceURLs:map[microstack:map[compute:http://10.20.20.1:8774/v2.1 identity:http://10.20.20.1:5000/v3/ image:http://10.20.20.1:9292 network:http://10.20.20.1:9696 placement:http://10.20.20.1:8778]]}
11:37:47 INFO cmd bootstrap.go:762 Creating Juju controller "msk" on microstack/microstack
11:37:47 DEBUG goose apiversion.go:274 discovered API versions: [{Version:{Major:2 Minor:0} Links:[{Href:http://10.20.20.1:8774/v2/ Rel:self}] Status:SUPPORTED} {Version:{Major:2 Minor:1} Links:[{Href:http://10.20.20.1:8774/v2.1/ Rel:self}] Status:CURRENT}]
11:37:47 INFO juju.cmd.juju.commands bootstrap.go:831 combined bootstrap constraints:
11:37:47 DEBUG juju.environs.bootstrap bootstrap.go:305 model "controller" supports application/machine networks: true
11:37:47 DEBUG juju.environs.bootstrap bootstrap.go:307 network management by juju enabled: true
11:37:47 DEBUG juju.environs.bootstrap bootstrap.go:959 no agent directory found, using default agent metadata source: https://streams.canonical.com/juju/tools
11:37:47 DEBUG juju.environs.bootstrap bootstrap.go:984 setting default image metadata source: /home/netipos/simplestreams/images
11:37:47 DEBUG juju.environs imagemetadata.go:46 new user image datasource registered: bootstrap metadata
11:37:47 INFO juju.environs.bootstrap bootstrap.go:1018 custom image metadata added to search path
11:37:47 INFO cmd bootstrap.go:349 Loading image metadata
11:37:47 DEBUG juju.environs imagemetadata.go:126 obtained image datasource "bootstrap metadata"
11:37:47 DEBUG juju.environs imagemetadata.go:126 obtained image datasource "default ubuntu cloud images"
11:37:47 DEBUG juju.environs.bootstrap bootstrap.go:850 constraints for image metadata lookup &{{{microstack http://10.20.20.1:5000/v3} [saucy xenial yakkety zesty win10 precise groovy win2019 win2012hv win2012 genericlinux vivid centos7 opensuseleap eoan utopic cosmic disco win2016nano win7 win81 quantal wily focal win2012r2 raring artful bionic win2008r2 win2012hvr2 win2016 win2016hv win8 trusty] [amd64 i386 armhf arm64 ppc64el s390x] released}}
11:37:47 DEBUG juju.environs.bootstrap bootstrap.go:862 found 1 image metadata in bootstrap metadata
11:37:47 DEBUG juju.environs.simplestreams simplestreams.go:457 skipping index "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson" because of missing information: index file has no data for cloud {microstack http://10.20.20.1:5000/v3} not found
11:37:47 DEBUG juju.environs.bootstrap bootstrap.go:858 ignoring image metadata in default ubuntu cloud images: index file has no data for cloud {microstack http://10.20.20.1:5000/v3} not found
11:37:47 DEBUG juju.environs.bootstrap bootstrap.go:866 found 1 image metadata from all image data sources
11:37:47 INFO cmd bootstrap.go:426 Looking for packaged Juju agent version 2.8.0 for amd64
11:37:47 INFO juju.environs.bootstrap tools.go:72 looking for bootstrap agent binaries: version=2.8.0
11:37:47 DEBUG juju.environs.tools tools.go:102 finding agent binaries in stream: "released"
11:37:47 DEBUG juju.environs.tools tools.go:104 reading agent binaries with major.minor version 2.8
11:37:47 DEBUG juju.environs.tools tools.go:112 filtering agent binaries by version: 2.8.0
11:37:47 DEBUG juju.environs.tools tools.go:115 filtering agent binaries by series: bionic
11:37:47 DEBUG juju.environs.tools tools.go:118 filtering agent binaries by architecture: amd64
11:37:47 DEBUG juju.environs.tools urls.go:138 trying datasource "keystone catalog"
11:37:47 DEBUG juju.environs.simplestreams simplestreams.go:692 using default candidate for content id "com.ubuntu.juju:released:tools" are {20161007 mirrors:1.0 content-download streams/v1/cpc-mirrors.sjson []}
11:37:48 INFO juju.environs.bootstrap tools.go:74 found 1 packaged agent binaries
11:37:48 INFO cmd bootstrap.go:513 Starting new instance for initial controller
11:37:48 INFO cmd bootstrap.go:176 Launching controller instance(s) on microstack/microstack...
11:37:48 ERROR juju.cmd.juju.commands bootstrap.go:782 failed to bootstrap model: cannot start bootstrap instance: no usable availability zones
11:37:48 DEBUG juju.cmd.juju.commands bootstrap.go:783 (error details: [{/build/snapcraft-juju-03af7d/parts/juju/src/cmd/juju/commands/bootstrap.go:857: failed to bootstrap model} {/build/snapcraft-juju-03af7d/parts/juju/src/environs/bootstrap/bootstrap.go:609: } {/build/snapcraft-juju-03af7d/parts/juju/src/environs/bootstrap/bootstrap.go:517: } {/build/snapcraft-juju-03af7d/parts/juju/src/provider/common/bootstrap.go:57: } {/build/snapcraft-juju-03af7d/parts/juju/src/provider/common/bootstrap.go:219: cannot start bootstrap instance} {/build/snapcraft-juju-03af7d/parts/juju/src/provider/common/bootstrap.go:322: no usable availability zones}])
11:37:48 DEBUG juju.cmd.juju.commands bootstrap.go:1428 cleaning up after failed bootstrap
11:37:48 INFO juju.provider.common destroy.go:21 destroying model "controller"
11:37:48 INFO juju.provider.common destroy.go:32 destroying instances
11:37:48 DEBUG juju.provider.openstack provider.go:1641 terminating instances []
11:37:49 DEBUG goose client.go:612 auth details: &{Token:gAAAAABe-IENDjUVI0l3pqnZ6bFjS3-wcgriAoofW32PgmTd0BQlh1AhJtYJtfugVu44JTICwB-aXQk9LafpWNbNQOo-qlLNKKTs6rvxomdC62oU4klA6VS3HM4AaD0UoeK1OZjA4JMM2D7U1tOb8CdGF73TaP2xdbq0dOSNDONmCy1yxHkRBic TenantId:c2ae47c09d374c788af1615ff8235923 TenantName:admin UserId:d3c0d8eb8f084b91901e3b30fb6571d7 Domain: RegionServiceURLs:map[microstack:map[compute:http://10.20.20.1:8774/v2.1 identity:http://10.20.20.1:5000/v3/ image:http://10.20.20.1:9292 network:http://10.20.20.1:9696 placement:http://10.20.20.1:8778]]}
11:37:49 DEBUG goose apiversion.go:274 discovered API versions: [{Version:{Major:2 Minor:0} Links:[{Href:http://10.20.20.1:9696/v2.0/ Rel:self}] Status:CURRENT}]
11:37:49 INFO juju.provider.common destroy.go:56 destroying storage
11:37:49 DEBUG juju.provider.openstack cinder.go:684 endpoint "volumev3" not found for "microstack" region, trying "volumev2"
11:37:49 DEBUG juju.provider.openstack cinder.go:689 endpoint "volumev2" not found for "microstack" region, trying "volume"
11:37:49 DEBUG juju.provider.openstack cinder.go:684 endpoint "volumev3" not found for "microstack" region, trying "volumev2"
11:37:49 DEBUG juju.provider.openstack cinder.go:689 endpoint "volumev2" not found for "microstack" region, trying "volume"
11:37:49 INFO cmd supercommand.go:544 command finished

[Peter Matulis (petermatulis)]

I hit this today with the beta (ussuri). See attachment.

[Alvaro Uria (aluria)]

Snap revisions in use:
"""
$ snap list juju
Name Version Rev Tracking Publisher Notes
juju 2.9.4 16423 latest/candidate canonical✓ classic
$ snap list microstack
Name Version Rev Tracking Publisher Notes
microstack ussuri 233 latest/beta canonical✓ -
"""

I got the same error so I tried to specify the AZ listed below ("internal"):

$ microstack.openstack availability zone list -fjson | jq -c
[{"Zone Name":"internal","Zone Status":"available"}]

$ microstack.openstack compute service list -fjson | jq -c '.[] | [.ID, .Binary,.Zone,.Status,.State]'
[4,"nova-conductor","internal","enabled","up"]
[6,"nova-scheduler","internal","enabled","up"]

$ microstack.openstack network agent list -fjson | jq -c
[{"ID":"18ec4279-f197-4133-b674-78145dc98612","Agent Type":"OVN Controller Gateway agent","Host":"pluto","Availability Zone":"","Alive":false,"State":true,"Binary":"ovn-controller"}]

$ juju bootstrap --bootstrap-series=$OS_SERIES --metadata-source=$SIMPLESTREAMS --model-default network=test --model-default external-network=external --bootstrap-constraints="allocate-public-ip=true" --to zone=internal --debug microstack microstack

This command also failed with the following error:
"""
16:06:31 ERROR juju.cmd.juju.commands bootstrap.go:881 failed to bootstrap model: cannot start bootstrap instance: availability zone "internal" not valid
16:06:31 DEBUG juju.cmd.juju.commands bootstrap.go:882 (error details: [{/build/snapcraft-juju-35d6cf/parts/juju/src/cmd/juju/commands/bootstrap.go:981: failed to bootstrap model} {/build/snapcraft-juju-35d6cf/parts/juju/src/environs/bootstrap/bootstrap.go:651: } {/build/snapcraft-juju-35d6cf/parts/juju/src/environs/bootstrap/bootstrap.go:555: } {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/common/bootstrap.go:61: } {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/common/bootstrap.go:239: cannot start bootstrap instance} {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/common/bootstrap.go:334: } {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/openstack/provider.go:681: } {/build/snapcraft-juju-35d6cf/parts/juju/src/provider/openstack/provider.go:702: } {/build/snapcraft-juju-35d6cf/parts/juju/src/core/network/zone.go:33: availability zone "internal" not valid}])
"""

Since the Neutron agent missed an AZ, I started looking into the logs, and found:
"""
$ tail -50 /var/snap/microstack/common/log/neutron.log
2021-06-09 16:18:54.078 852600 INFO ovsdbapp.backend.ovs_idl.vlog [-] unix:/var/snap/microstack/common/run/ovn/ovnsb_db.sock: connecting...
2021-06-09 16:18:54.079 852600 INFO ovsdbapp.backend.ovs_idl.vlog [-] unix:/var/snap/microstack/common/run/ovn/ovnsb_db.sock: connected
2021-06-09 16:18:54.079 852601 INFO ovsdbapp.backend.ovs_idl.vlog [-] unix:/var/snap/microstack/common/run/ovn/ovnsb_db.sock: connecting...
2021-06-09 16:18:54.081 852601 INFO ovsdbapp.backend.ovs_idl.vlog [-] unix:/var/snap/microstack/common/run/ovn/ovnsb_db.sock: connected
2021-06-09 16:18:54.085 852602 INFO ovsdbapp.backend.ovs_idl.vlog [-] unix:/var/snap/microstack/common/run/ovn/ovnsb_db.sock: connecting...
2021-06-09 16:18:54.086 852602 INFO ovsdbapp.backend.ovs_idl.vlog [-]...

[Alvaro Uria (aluria)]

I have managed to make it work with the latest Juju 2.9.4 and microstack from beta channel, also passing "--devmode" (I don't think I missed this flag before, though).

The main difference I see is that there are 2 neutron agents and both are alive:
"""
$ microstack.openstack network agent list -fjson | jq
[
  {
    "ID": "135bac08-3e08-4895-b7b2-c4ce44e47f20",
    "Agent Type": "OVN Controller Gateway agent",
    "Host": "pluto",
    "Availability Zone": "",
    "Alive": true,
    "State": true,
    "Binary": "ovn-controller"
  },
  {
    "ID": "d10d6fd9-e19a-44f0-ae93-421c3f8a0493",
    "Agent Type": "OVN Metadata agent",
    "Host": "pluto",
    "Availability Zone": "",
    "Alive": true,
    "State": true,
    "Binary": "networking-ovn-metadata-agent"
  }
]
"""

FWIW, when I removed the snap and installed the classic (rocky) one, I also ran into network related. However, and as described above, this morning, after coming back from suspending my laptop, I managed to configure it and bootstrap a Juju controller.