openstack provider: if use-floating-ip=true, uses incorrect compute API endpoint to determine available floating IPs

[Florian Haas]

Trying to spin up a Juju controller in a public cloud with:

juju --debug bootstrap <name> \
  --config image-metadata-url=<swift-url>/simplestreams/images \
  --config network=<uuid> \
  --config use-floating-ip=true

Bootstrap fails.

Debug log shows these critical lines:

20:42:52 DEBUG juju.provider.openstack provider.go:938 allocating public IP address for openstack node
20:42:54 INFO juju.provider.common destroy.go:20 destroying model "controller"
20:42:54 INFO juju.provider.common destroy.go:31 destroying instances
20:42:54 DEBUG juju.provider.openstack provider.go:1068 terminating instances []
20:42:55 INFO juju.provider.common destroy.go:51 destroying storage
20:42:55 DEBUG juju.provider.openstack cinder.go:80 volume URL: https://<hostname>:8776/v2/<tenant-id>
20:42:58 ERROR cmd supercommand.go:458 failed to bootstrap model: cannot start bootstrap instance: cannot allocate a public IP as needed: failed to allocate a floating ip
caused by: Resource at https://<hostname>:8774/v2.1/<tenant-id>/os-floating-ips not found
caused by: request (https://<hostname>:8774/v2.1/<tenant-id>/os-floating-ips) returned unexpected status: 404; error info: {"itemNotFound": {"message": "Floating ip pool not found.", "code": 404}}

Now, compare to an API call for listing floating IPs:

nova --debug floating-ip-list

[...]
https://<hostname>:8774/v2/<tenant-id>/os-floating-ips

Compare the two API URLs, note they differ in v2.1 vs. v2.

"openstack catalog show compute" lists this publicURL:

publicURL: https://<hostname>:8774/v2/<tenant-id>

So the question is: where exactly does Juju dream up the v2.1 URL, which is then bound to fail with a 404?

[Andrew Wilkins (axwalk)]

There was a recent change to do API version discovery. In some calls to OpenStack, Juju will look for an API version matching "v2", which will list all of the available API versions and select the most recent one matching the major version 2. The v2.1 comes from the version list endpoint (GET https://<hostname>:8774/).

It looks like your version of OpenStack supports a microversion >= 2.35, at which point the network API proxy endpoints have been removed. We'll need to update Juju 2.0.x to either force the use of v2.0 (probably simplest), or specify a microversion.

There is work underway to move us off using the deprecated network APIs, and use the Neutron service directly. I think this won't be landing until Juju 2.1.

[Andrew Wilkins (axwalk)]

https://github.com/go-goose/goose/pull/29
https://github.com/juju/juju/pull/6535

[Florian Haas (fghaas)]

Thanks Andrew! Does this patch automatically roll into packages built for the juju-devel PPA, or is there another, better way for me to test this?

[Andrew Wilkins (axwalk)]

I was of the impression that the "edge" channel for the juju snap was built daily, but it appears not to be the case. I'm told that there will be a 2.0.2 release within a few days.

[Florian Haas (fghaas)]

OK, so that means use the 2.0.2 package from ppa:juju/stable then (for those of us who prefer APT over snaps)?

[Andrew Wilkins (axwalk)]

Correct.

[Florian Haas (fghaas)]

Will do, thank you!

[Andrew Wilkins (axwalk)]

Florian, just FYI we've found a couple of (unrelated) regressions that have held up 2.0.2. Sorry for the delay.

[Florian Haas (fghaas)]

Thanks for letting me know!

[Florian Haas (fghaas)]

All right, so I tested this with the 2.0.2 build from juju-proposed and now I'm confused:

09:21:39 DEBUG juju.provider.openstack provider.go:945 allocating public IP address for openstack node
09:21:40 INFO juju.provider.common destroy.go:20 destroying model "controller"
09:21:40 INFO juju.provider.common destroy.go:31 destroying instances
09:21:41 DEBUG juju.provider.openstack provider.go:1075 terminating instances []
09:21:42 INFO juju.provider.common destroy.go:51 destroying storage
09:21:42 DEBUG juju.provider.openstack cinder.go:80 volume URL: https://<hostname>:8776/v2/22f3992b3e2b4a07a7253199c93cd4f9
09:21:44 ERROR cmd supercommand.go:458 failed to bootstrap model: cannot start bootstrap instance: cannot allocate a public IP as needed: failed to allocate a floating ip
caused by: Resource at https://<hostname>:8774/v2/22f3992b3e2b4a07a7253199c93cd4f9/os-floating-ips not found
caused by: request (https://<hostname>:8774/v2/22f3992b3e2b4a07a7253199c93cd4f9/os-floating-ips) returned unexpected status: 404; error info: {"itemNotFound": {"message": "Floating ip pool not found.", "code": 404}}
09:21:44 DEBUG cmd supercommand.go:459 (error details: [{github.com/juju/juju/cmd/juju/commands/bootstrap.go:556: failed to bootstrap model} {github.com/juju/juju/provider/common/bootstrap.go:47: } {github.com/juju/juju/provider/common/bootstrap.go:178: cannot start bootstrap instance} {github.com/juju/juju/provider/openstack/provider.go:947: cannot allocate a public IP as needed} {failed to allocate a floating ip
caused by: Resource at https://<hostname>:8774/v2/22f3992b3e2b4a07a7253199c93cd4f9/os-floating-ips not found
caused by: request (https://<hostname>:8774/v2/22f3992b3e2b4a07a7253199c93cd4f9/os-floating-ips) returned unexpected status: 404; error info: {"itemNotFound": {"message": "Floating ip pool not found.", "code": 404}}}])

Compare this from "nova --debug floating-ip-list":

DEBUG (session:248) REQ: curl -g -i -X GET https://<hostname>:8774/v2/22f3992b3e2b4a07a7253199c93cd4f9/os-floating-ips -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: <token>"
DEBUG (connectionpool:388) "GET /v2/22f3992b3e2b4a07a7253199c93cd4f9/os-floating-ips HTTP/1.1" 200 20
DEBUG (session:277) RESP: [200] Date: Mon, 14 Nov 2016 09:22:54 GMT Content-Length: 20 Content-Type: application/json X-Compute-Request-Id: req-15cd6d3a-286a-4282-b2de-1143415870b9
RESP BODY: {"floating_ips": []}

+----+----+-----------+----------+------+
| Id | IP | Server Id | Fixed IP | Pool |
+----+----+-----------+----------+------+
+----+----+-----------+----------+------+

So now it's using exactly the URI that it should, but still produces a 404. What exactly is going on there?

[Andrew Wilkins (axwalk)]

That is baffling. You are using the exact same credentials and region?

Are you able to provide some more details about the OpenStack you're trying to bootstrap with? Based on the error message, it appears to be Liberty or older.

FWIW I get 404 with use-floating-ip=true when bootstrapping Rackspace, but I also get 404 when using "nova floating-ip-list" there. Both work with our internal testing OpenStack install (Grizzly, supposedly), and the deployment of Newton that I tested with on top of LXD.

[Florian Haas (fghaas)]

Yeah, color me baffled too. Is there any magic incantation of "juju bootstrap" that will allow me to see the headers of that request? Or, for that matter, the method? (Should be GET, but it just occurred to me that the Juju log actually doesn't say so.)

[Andrew Wilkins (axwalk)]

No, unfortunately there is not. I'll file a bug to add trace logging.

However, your question prompted me to look closer and there are two calls in that area of code: one GET (list existing floating IPs), and one POST (allocate a new floating IP).

The POST will return with 404 "Floating ip pool not found" if there exists no pool defined with the default pool name, because we don't specify one.

We're currently working on moving Juju over to using the Neutron APIs directly (https://github.com/juju/juju/pull/6552/files). Part of that work involves changing how floating IPs are allocated: Juju will default to taking a floating IP from any external network in the AZ of the server being started, or an explicitly specified external network if you pass "external-network" in config.

As a work-around in the interim, you should be able to create a floating IP manually, and Juju will find and use it.

[Andrew Wilkins (axwalk)]

Assigning to Heather as this should be fixed when her work lands.

[Florian Haas (fghaas)]

Thanks Andrew, your workaround did help — in that if there is an allocated but not associated floating IP, the controller just grabs that and then comes up OK.

That, unfortunately, ins't to say that the controller actually does what it says on the tin, but I guess that's for a couple of new bugs. :)