juju 2 beta3 can't bootstrap to lcy02

[Francis Ginther]

Attempts at bootstrapping to canonistack lcy02 run into "{"badRequest": {"message": "Can not find requested image", "code": 400}}". Full output of bootstrap is here: https://pastebin.canonical.com/152908/ (with --debug: https://pastebin.canonical.com/152909/)

Bootstrapping to lcy01 works. Notice in the pastebin that the request is made to "https://nova-lcy01.canonistack.canonical.com/v2/a1412d7034ce49b2a980b84b15c04372/servers". Should that maybe be "https://nova-lcy02.canonistack.canonical.com"? (I'm really just guessing here.)

juju version:
2.0-beta3-xenial-amd64

Here's my cloud and credential files:

[~/.local/share/juju/clouds.yaml]
clouds:
  canonistack:
    type: openstack
    auth-types: [userpass]
    endpoint: https://keystone.canonistack.canonical.com:443/v2.0/
    regions:
      lcy01: {}
      lcy02: {}

[~/.local/share/juju/credentials.yaml]
credentials:
  canonistack:
    default-region: lcy02
    default-credential: fginther
    fginther:
      auth-type: userpass
      password: [REDACTED]
      tenant-name: [REDACTED]
      username: fginther

[Eric Snow (ericsnowcurrently)]

I wasn't able to make any progress on this.

[Martin Packman (gz)]

Run against lcy02 in CI:

with master (fails): http://juju-ci.vapour.ws/job/canonistack-deploy-trusty-amd64/4237/
with 1.25 (passes): http://juju-ci.vapour.ws/job/canonistack-deploy-trusty-amd64/4238/

[Katherine Cox-Buday (cox-katherine-e)]

It looks like the regression was introduced in https://github.com/juju/juju/commit/d27ccf3

Still investigating root cause.

[Katherine Cox-Buday (cox-katherine-e)]

You were correct to suspect the reference to lcy01; it looks as though Canonistack is incorrectly returning references to lcy01 if you use the v3 Keystone endpoints.

I'm going to pull in some people more familiar with Canonistack and Openstack in general tomorrow to verify.

[cargonza (cargonza)]

confirmed. Keystone v3 support will land with the 16.04 Openstack charms.

[Katherine Cox-Buday (cox-katherine-e)]

So the consensus is that the bug is that the user specified a v2 endpoint in their config, but Juju discovered a v3 endpoint and went with that. We're going to look into having the client constrain its usage to the version specified by Juju.

[Ryan Beisner (1chb1n)]

In this case, the user has specifically asked for Keystone V2 via the "endpoint: https://keystone.canonistack.canonical.com:443/v2.0/" value, so I don't think Juju should be attempting Keystone V3 at all.

Existence of V3 in a curl query to Keystone does not necessarily mean that all services are configured and clear to use V3. This is because Keystone V3 support did not expressly exist in the OpenStack charms prior to the upcoming 16.04 charm release.

[Katherine Cox-Buday (cox-katherine-e)]

Fix is merging: https://github.com/juju/juju/pull/5049