Juju is not capable of connecting to Juju deployed Openstack with keystone v3 with access-key

Bug #1770835 reported by Florian Guitton
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
Wishlist
Unassigned

Bug Description

Hello everybody,

There seems to be an issue with bootstrapping against an Openstack cloud running keystone v3.
At the end, the output of the Juju bootstrap.

It seems juju is attempting to reach /v3/tokens to proceed with authentication. This is not supported by Keystone v3 as described in their API doc : https://developer.openstack.org/api-ref/identity/v3/#authentication-and-token-management.

Juju should attempt to query /v3/auth/tokens. This might be linked to the goose package but their tests (https://github.com/go-oose/goose/blob/v2/identity/v3userpass_test.go) suggests it shouldn't.

---------------------------------------------------------------------------

@maas-region:~$ juju bootstrap dsi-os/dsi-r1 dsi-osr1-juju --debug
13:16:50 INFO juju.cmd supercommand.go:56 running juju [2.4-rc1 gc go1.10]
13:16:50 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/4271/bin/juju", "bootstrap", "dsi-os/dsi-r1", "dsi-osr1-juju", "--debug"}
13:16:50 DEBUG juju.cmd.juju.commands bootstrap.go:851 authenticating with region "dsi-r1" and credential "admin" ()
13:16:50 DEBUG juju.cmd.juju.commands bootstrap.go:979 provider attrs: map[use-floating-ip:false use-default-secgroup:false network: external-network: use-openstack-gbp:false policy-target-group:]
13:16:50 INFO cmd authkeys.go:114 Adding contents of ".local/share/juju/ssh/juju_id_rsa.pub" to authorized-keys
13:16:50 INFO cmd authkeys.go:114 Adding contents of ".ssh/id_rsa.pub" to authorized-keys
13:16:50 DEBUG juju.cmd.juju.commands bootstrap.go:1035 preparing controller with config: map[]
13:16:50 INFO juju.provider.openstack provider.go:146 opening model "controller"
13:16:50 DEBUG juju.provider.openstack provider.go:805 authentication failed: authentication failed
caused by: Resource at http://keystone-admin.example.com:35357/v3/tokens not found
caused by: request (http://keystone-admin.example.com:35357/v3/tokens) returned unexpected status: 404; error info: Failed: 404 error: (http://keystone-admin.example.com:35357/v3/tokens): The resource could not be found.
ERROR authentication failed.

Please ensure the credentials are correct. A common mistake is
to specify the wrong tenant. Use the OpenStack "project" name
for tenant-name in your model configuration.
13:16:50 DEBUG cmd supercommand.go:459 error stack:
github.com/juju/juju/provider/openstack/provider.go:806: authentication failed.

Please ensure the credentials are correct. A common mistake is
to specify the wrong tenant. Use the OpenStack "project" name
for tenant-name in your model configuration.
github.com/juju/juju/environs/bootstrap/prepare.go:164:
github.com/juju/juju/environs/bootstrap/prepare.go:100:
github.com/juju/juju/cmd/juju/commands/bootstrap.go:486:

description: updated
description: updated
description: updated
Revision history for this message
Florian Guitton (f-guitton) wrote :

I must precise the configuration goes as follow :

maas-region:~$ cat .local/share/juju/clouds.yaml
clouds:
  dsi-os:
    type: openstack
    auth-types: [access-key]
    endpoint: http://keystone-admin.example.com:35357/v3/
    regions:
      dsi-r1:
        endpoint: http://keystone-admin.example.com:35357/v3/

maas-region:~$ cat .local/share/juju/credentials.yaml
credentials:
  dsi-os:
    florian:
      auth-type: access-key
      access-key: bae7651caeabc1ed876ffdb342bae23c
      secret-key: 9172bc91ae1c3df1787623ac12093bc0
      username: admin
      tenant-name: admin

Revision history for this message
Florian Guitton (f-guitton) wrote :

A little bit of fiddling around demonstrated that AccessKeyPair can only be used with Keystone in version 2. I believe the documentation should be improved to explicitly precise this. Right now it is misleading and demonstrates the addition of a v3 endpoint with both access-key and userpass auth types.

Generally is would be great to complement the Openstack section of the doc to address issues like : "the configured region 'x' does not allow access to all required services, namely: compute"

Revision history for this message
Anastasia (anastasia-macmood) wrote :

@Florian Guitton (f-guitton),

It looks like it might be a documentation improvement that will resolve all future confusion.

I have added doc issue [1] and am marking this report for Juju as Invalid.

[1] https://github.com/juju/docs/issues/2868

Changed in juju:
status: New → Invalid
Revision history for this message
Peter Matulis (petermatulis) wrote :

@Florian This is not quite clear to me. Which of the following scenarios are we talking about:

1. Access key pair is not supported upstream under Keystone v3.

2. Juju does not support access key pair with Keystone v3.

3. Your Keystone-v3-based cloud is not configured for access key pair.

This resource suggests that Keystone v3 authentication types are more wide-ranged than v2 types:

https://lists-new.canonical.com/mailman3/postorius/lists/product-docs.lists.canonical.com/members/subscriber/

Revision history for this message
Peter Matulis (petermatulis) wrote :

Sorry, messed up my URL there. :)

This is what I intended:

https://docs.openstack.org/keystoneauth/latest/authentication-plugins.html

Revision history for this message
Drew Freiberger (afreiberger) wrote :
Download full text (3.2 KiB)

I am receiving the same issue as well and can provide clarity.

I'm running this all with a focal-ussuri undercloud.

I have followed this process:
1. create new openstack project, tenant network with external router gateway to a floating-ip network.
2. Grant a user Admin and member privileges to the project.
3. Login as that user and create a focal ubuntu VM with an associated FIP
4. add inbound ssh rule in security group default
5. Generate application credentials in Horizon:
  In Horizon as that user under that project in top project selection menu:
  Click Identity -> Application Credentials -> Create Application Credentials
  Name something useful like: afreiberger-juju-credentials
  Multi-select Admin and member roles
  Click Create Application Credential, and save the ID and Secret for use later.
  download novarc for user (not app credentials) to put on jumphost

On the jumphost in the project (after setting up any necessary proxies),
6. sudo snap install juju --classic; sudo snap install openstackclients
7. source the project novarc
8. juju add-cloud openstack
answer prompts as:
openstack
<enter>
<enter>
access-key
<enter>
<enter>
n
9. add credentials:

ubuntu@jumpbox:~$ juju add-credential openstack
This operation can be applied to both a copy on this client and to the one on a controller.
No current controller was detected and there are no registered controllers on this client: either bootstrap one or register one.
Enter credential name: afreiberger-juju-credentials

Regions
  RegionOne

Select region [any region, credential is not region specific]: <enter>

Using auth-type "access-key".

Enter access-key: <id of credential>

Enter secret-key: <copied secret from earlier>

Enter tenant-name (optional): afreiberger-tenant

Enter tenant-id (optional): <enter>

Enter version (optional): <enter>

Credential "afreiberger-juju-credentials" added locally for cloud "openstack".

10. Run bootstrap:
juju bootstrap openstack openstack

# expected results are a new VM is deployed and bootstrapping occurs.
# Actual results are in the description of this ticket.

It appears that the issue Florian noted in comment #2 was that in their environment, they could get the access-key style juju credentials to work if they used a v2 endpoint instead of a v3 endpoint.

However, the issue in juju or it's upstream dependencies is that if using keystone v3 and application credentials, the URI must be /v3/auth/tokens, but the URI being accessed by the juju client's openstack methods for token generation via access-key is actually hitting a 404 /v3/tokens URI.

I believe the link to the openstack API was Florian demonstrating that the tokens API is not present under /v3/tokens.

It is my opinion that juju should very much support v3 keystone access-key authentication, as user-pass based authentication is prone to expiration and constantly changing passwords. Application credentials are the proper way for software clients to interact with the Openstack API. It does appear the juju client does not currently support a functioning way to use access-key credentials against a v3-only Keystone deployment. V2 endpoints are no longer available in modern Openstack, ...

Read more...

Changed in juju:
status: Invalid → Confirmed
summary: Juju is not capable of connecting to Juju deployed Openstack with
- keystone v3
+ keystone v3 with access-key
Revision history for this message
Drew Freiberger (afreiberger) wrote (last edit ):

@petermatulis:
tl;dr the answer to question in comment #4 is option 2. Juju doesn't support keystone v3 access-key auth.

Revision history for this message
Juan M. Tirado (tiradojm) wrote :

You're right at this moment Juju does not support v3 access-key auth. You can see the supported modes at https://github.com/juju/juju/blob/2.9/provider/openstack/client.go#L170-L206

Right now this is not in our road map so I will add it to the wishlist.

Changed in juju:
importance: Undecided → Wishlist
John A Meinel (jameinel)
Changed in juju:
status: Confirmed → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.