Authentication process bypasses oidc-provider-metadata-url

[Natalia Litvinova]

Hi team,

I have deployed the charm and configured it with the following:

keystone-openidc:
  oidc-client-id: <userid>
  oidc-client-secret: <secret>
 # oidc-redirect-uri: https://<keystone url>:443/v3/auth/OS-FEDERATION/identity_providers/keycloak/protocols/openid/websso/redirect_uri
  oidc-provider-metadata-url: https://<keycloak url>/realms/<realm>/.well-known/openid-configuration
  user-facing-name: Keycloak
  protocol_id: keycloak

And on the configuration side all seems well, but when I try to pick this Keycloak option from Horizon and authenticate with it I'm instantly getting redirected not to the Keycloak URL, but to keystone. And I get {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}. The URL I'm facing is https://keystone.<mydomain>/v3/auth/OS-FEDERATION/identity_providers/keycloak/protocols/openid/websso?origin=https://horizon.<mydomain>/auth/websso/

I've tried to trace the redirection to Keycloak, but it just doesn't get there.

The logs don't indicate anything of that nature (I've monitored on all 3 units at the same time):
$ sudo tail -f /var/log/juju/unit-keystone-openidc-3.log
2024-05-13 12:41:31 INFO unit.keystone-openidc/3.juju-log server.go:325 Status updated
2024-05-13 12:41:31 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)
2024-05-13 12:46:40 INFO unit.keystone-openidc/3.juju-log server.go:325 Updating status
2024-05-13 12:46:40 INFO unit.keystone-openidc/3.juju-log server.go:325 GETing content from https://<keycloak url>/realms/<realm>/.well-known/openid-configuration
2024-05-13 12:46:40 INFO unit.keystone-openidc/3.juju-log server.go:325 Status updated
2024-05-13 12:46:40 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)
2024-05-13 12:50:58 INFO unit.keystone-openidc/3.juju-log server.go:325 Updating status
2024-05-13 12:50:58 INFO unit.keystone-openidc/3.juju-log server.go:325 GETing content from https://<keycloak url>/realms/<realm>/.well-known/openid-configuration
2024-05-13 12:50:58 INFO unit.keystone-openidc/3.juju-log server.go:325 Status updated
2024-05-13 12:50:58 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)

$ sudo tail -f /var/log/keystone/keystone.log
(keystone.common.password_hashing): 2024-05-13 12:54:32,545 WARNING Truncating password to algorithm specific maximum length 72 characters.
(keystone.common.password_hashing): 2024-05-13 12:54:45,767 WARNING Truncating password to algorithm specific maximum length 72 characters.
(keystone.api.auth): 2024-05-13 12:54:50,609 ERROR https://horizon.<mydomain>/auth/websso/ is not a trusted dashboard host
(keystone.server.flask.application): 2024-05-13 12:54:50,612 WARNING Authorization failed. The request you have made requires authentication. from 10.1.0.71
(keystone.common.password_hashing): 2024-05-13 12:54:53,595 WARNING Truncating password to algorithm specific maximum length 72 characters.
(keystone.common.password_hashing): 2024-05-13 12:55:02,170 WARNING Truncating password to algorithm specific maximum length 72 characters.
(keystone.common.password_hashing): 2024-05-13 12:55:09,136 WARNING Truncating password to algorithm specific maximum length 72 characters.
(keystone.common.password_hashing): 2024-05-13 12:55:31,473 WARNING Truncating password to algorithm specific maximum length 72 characters.
(keystone.common.password_hashing): 2024-05-13 12:55:45,759 WARNING Truncating password to algorithm specific maximum length 72 characters.
(keystone.common.password_hashing): 2024-05-13 12:55:59,971 WARNING Truncating password to algorithm specific maximum length 72 characters.
(keystone.common.password_hashing): 2024-05-13 12:56:31,481 WARNING Truncating password to algorithm specific maximum length 72 characters.
(keystone.common.password_hashing): 2024-05-13 12:56:45,109 WARNING Truncating password to algorithm specific maximum length 72 characters.

Versions:
Openstack Yoga
Keystone yoga/stable rev 686
Keystone-openidc latest/edge rev 13

The steps to reproduce:
$ juju deploy --config keystone-oidc.yaml keystone-openidc --channel latest/edge --to lxd:0
$ juju relate keystone keystone-openidc
$ juju relate openstack-dashboard:websso-fid-service-provider keystone-openidc:websso-fid-service-provider
$ source deploy/generated/openstack/novarc
$ openstack identity provider create --remote-id https://<keycloak url>/realms/<realm> keycloak
+-------------------+------------------------------------------------+
| Field | Value |
+-------------------+------------------------------------------------+
| authorization_ttl | None |
| description | None |
| domain_id | edf2c8a97b0e4a3b9be77ff5eea66ba5 |
| enabled | True |
| id | keycloak |
| remote_ids | https://<keycloak url>/realms/<realm> |
+-------------------+------------------------------------------------+
$ openstack domain list
+----------------------------------+----------------------------------+---------+-----------------------------------------------------------------+
| ID | Name | Enabled | Description |
+----------------------------------+----------------------------------+---------+-----------------------------------------------------------------+
| 28c0be2ea2df44a680d41273977dfa3e | admin_domain | True | Created by Juju |
| 59f2a499cffe4597b5c893018e51271f | service_domain | True | Created by Juju |
| ccce5424b0a54b598bc7254e7f2ce679 | heat | True | Stack projects and users |
| default | Default | True | The default domain |
| edf2c8a97b0e4a3b9be77ff5eea66ba5 | edf2c8a97b0e4a3b9be77ff5eea66ba5 | True | Auto generated federated domain for Identity Provider: keycloak |
+----------------------------------+----------------------------------+---------+-----------------------------------------------------------------+
$ openstack domain set --name federated_domain edf2c8a97b0e4a3b9be77ff5eea66ba5
$ openstack group create --domain federated_domain federated_users
$ openstack project create --domain federated_domain federated_project
$ openstack mapping create --rules deploy/config/oidc/rules.json keycloak_mapping
$ openstack federation protocol create openid --mapping keycloak_mapping --identity-provider keycloak
$ openstack group list --domain federated_domain
+----------------------------------+-----------------+
| ID | Name |
+----------------------------------+-----------------+
| 8de48154fe514651abacbd5fe0990dfc | federated_users |
+----------------------------------+-----------------+
$ openstack role add --group 8de48154fe514651abacbd5fe0990dfc --project federated_project member

[Jadon Naas (jadonn)]

I replicated the behavior you described in your bug report on a minimal install of Keystone + OpenStack Dashboard on Yoga/stable with the keystone-openidc charm on latest/edge like you described. I also get redirected to the same kind of URL - http://10.20.0.83:5000/v3/auth/OS-FEDERATION/identity_providers/keycloak/protocols/openid/websso?origin=http://10.20.0.84/horizon/auth/websso/.

That is the URL that is rendered into the Apache configuration at /etc/apache2/openidc/openidc-location.openid.conf on the Keystone unit in my installation.

I'm not familiar enough with the Keystone OpenIDC integration to say what exactly is happening at this time, but I will continue to look into what is causing the behavior we are seeing. I will report back when I have more information.

[Gaetan Gouzi (ggouzi)]

Thank you,
That is good news if you are able to reproduce the issue on a lab setup.

Regarding openidc-location.openid.conf, this other issue https://bugs.launchpad.net/puppet-keystone/+bug/2002490 could be related ? Maybe it can help in your analysis.

Let us know if you require some additional information.

[Jadon Naas (jadonn)]

Thanks for the link to the other information! After further looking I have identified two problems in the keystone-openidc charm that break the charm in my testing. The first problem is that the charm does not add a block like the following in /etc/keystone/keystone.conf:

[federation]
trusted_dashboard = https://horizon.<mydomain>/auth/websso/

Replace <mydomain> with the actual domain. The workaround for this is to add a block like the above example to the Keystone charm template inside each Keystone unit at /var/lib/juju/agents/unit-keystone-<unit-num>/charm/templates/rocky/keystone.conf. Then, trigger a rebuild of the template, such as by triggering debugging on and off (that is, juju config keystone debug=true then juju config keystone debug=false). That should rebuild the template with the change and restart Keystone. This should resolve the 401 error associated with this error from the Keystone logs:

(keystone.api.auth): 2024-05-13 12:54:50,609 ERROR https://horizon.<mydomain>/auth/websso/ is not a trusted dashboard host

I noticed this error is in the logs sample Natalia provided when reporting this bug.

There is a second, more serious problem I found after fixing this error in my test setup. It appears the OIDC configuration for Apache is not actually included anywhere in Apache's configuration. This means the configration the keystone-openidc charm places at /etc/apache2/openidc/openidc-location.openid.conf is not applied to Apache. The workaround for this is to add the following line to the bottom of /etc/apache2/apache2.conf in each Keystone unit:

IncludeOptional openidc/*.conf

After adding this line, reload the Apache service's configuration, such as by running sudo systemctl reload apache2. That should activate the OIDC configuration, and Horizon should properly redirect to the OIDC provider, such as Keycloak, instead of the Keystone URL that returns the 401 error.

The keystone-openidc charm needs to do these things on its own. That will take more work, but I wanted to share back these workarounds in case they can help unblock you.

[Jadon Naas (jadonn)]

I looked into things further. It looks like there is an existing integration for the trusted_dashboard configuration between Keystone and Horizon/OpenStack Dashboard. it is called "websso-trusted-dashboard". Instead of manually adding the configuration to keystone.conf you should be able to do:

juju integrate keystone:websso-trusted-dashboard openstack-dashboard:websso-trusted-dashboard

Replace "integrate" with "relate" if you are on Juju 2.x. That should handle that configuration step.

[Jadon Naas (jadonn)]

For the second problem of the missing OIDC Apache configuration, there was a change (https://review.opendev.org/c/openstack/charm-keystone/+/850952) that added this configuration from stable/zed, but the change was never backported to stable/yoga. It should be a good candidate for a cherry-pick to stable/yoga. I will get that submitted.

[Jadon Naas (jadonn)]

Here is the cherry-pick I submitted to pull in the OIDC Apache configuration to stable/yoga: https://review.opendev.org/c/openstack/charm-keystone/+/920211.

[Gaetan Gouzi (ggouzi)]

Thank you for the provided patches and for your time.

In addition to the 2 provided patches, we had to:
- Remove the 2 extra `OIDCRedirectURI` directives that are overriding our `redirect_uri`
- Change the order of keycloak and openid in the path of the Location directives
e.g `/v3/OS-FEDERATION/identity_providers/openid/protocols/keycloak/auth` vs `/v3/OS-FEDERATION/identity_providers/keycloak/protocols/openid/auth`

Those remaining changes were not needed on your setup (which was working fine)
Sending you the full diff in MP for more details.

On customer side, the redirect_uri was also misconfigured. We changed that.

[Gaetan Gouzi (ggouzi)]

Feel free to lower the priority as we have a workaround now

[Gaetan Gouzi (ggouzi)]

Latest update: We can now access the authentication page thanks to the patches provided by Jadon (and the one in latest comment).
However, we receive a HTTP 500 from Apache when we log in.
```
[Thu Jun 06 07:14:43.481636 2024] [proxy:debug] [pid 768038:tid 139840041514560] proxy_util.c(2587): [client 10.1.5.50:35306] AH00944: connecting http://localhost:4980/v3/auth/OS-FEDERATION/identity_providers/keycloak/protocols/openid/websso?origin=https://horizon.XXX/auth/websso/ to localhost:4980, referer: https://keystone.XXX:5000/v3/auth/OS-FEDERATION/websso/keycloak
...
2024-06-06 07:14:43.539999 oidc_authenticate_user: the URL scheme (https) of the configured OIDCRedirectURI does not match the URL scheme of the URL being accessed (http): the "state" and "session" cookies will not be shared between the two!
```

This issue seems to be caused by mod_auth_openidc forwarding the request from the proxy to individual keystone units through http.

We can somehow workaround this by adding this directive to keep using https.
```
OIDCXForwardedHeaders X-Forwarded-Proto
```

However, we now get a HTTP 400. I don't want to dig into the rabbit hole with custom non-approved patches.
This is why we are requesting your assistance on how to continue troubleshooting this issue.

Thank you for your help

[Gaetan Gouzi (ggouzi)]

Hi Jadon,

Here is a feedback on the current patches we have and need to make keystone-openidc/Keycloak integration work on our setup (HAProxy behind 3 keystone units)

- Patch 1:
Add missing incude line in Apache config in /etc/apache/apache2.conf
IncludeOptional openidc/*.conf

- Patch 2:
Add missing trusted dashbaoard configuration in Keystone. This can be done through the existing, but not documented, relation
`juju relate openstack-dashboard:websso-trusted-dashboard keystone:websso-trusted-dashboard`

- Patch 3:
Change the order of /keycloak/ and /openidc/ in URI path of the 2 last Location directives, the ones using AuthType openid-connect.
`<Location /v3/auth/OS-FEDERATION/websso/keycloak>` changed to `<Location /v3/auth/OS-FEDERATION/websso/openid>` and `<Location /v3/auth/OS-FEDERATION/identity_providers/openid/protocols/keycloak/websso>` changed to `<Location /v3/auth/OS-FEDERATION/identity_providers/keycloak/protocols/openid/websso>`.
Not sure if this patch is really needed or if it comes from a misconfiguration on our side. In anycase, we receive the original HTTP 401 Natalia mentioned if we don't apply it.

- Patch 4:
Comment the 2 extra OIDCRedirectURI directives
openidc configuration file has 3 OIDCRedirectURI directives. The first one around the top of the file and 2 extra ones after the `# Support for websso from Horizon` line. Commenting those 2 extra ones was needed (why do we have this directive defined 3 times ?)
```
# Support for websso from Horizon
# OIDCRedirectURI "https://keystone.XXX.de:5000/v3/auth/OS-FEDERATION/identity_providers/openid/protocols/keycloak/websso"
# OIDCRedirectURI "https://keystone.XXX.de:5000/v3/auth/OS-FEDERATION/websso/keycloak"
```

- Patch 5:
Add HTTPS scheme forwarding to not break the HTTPS OIDC workflow with HTTP requests between HAProxy and individual Keystone units.
`OIDCXForwardedHeaders X-Forwarded-Proto`
Without this setting, we receive a HTTP 500 with logs sent in previous comment.

- Patch 6:
Do not use user-agent to generate state and session cookie. This will lead to a cookie mismatch between the request being made to Keysotne external endpoint and the request between HAproxy and the Keystone units.
Add the 2 following settings:
```
OIDCStateInputHeaders none
OIDCSessionType client-cookie:persistent
```

Is patch 6 going to be persistent or is it something to be changed in HAProxy (wouldn't forwarding the User-Agent be better?)

Thanks again for your quick help.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-keystone-openidc/+/921708

[Jadon Naas (jadonn)]

Thank you, Gaetan, for the detailed information and documentation!

In response to the patches you have outlined, I can report at this time:

- Patch 1:
We have merged a change to the Keystone charm that adds the missing Apache configuration for including the OpenIDC configuration directory. That change is at:

https://review.opendev.org/c/openstack/charm-keystone/+/920211.

- Patch 2:
I submitted a change to the README for the Keystone OpenIDC charm to include adding the relation between Keystone and Horizon for the websso-trusted-dashboard. That change is pending review at:

https://review.opendev.org/c/openstack/charm-keystone-openidc/+/921708

- Patch3 :
It looks like the charm code has some errors where the Identity provider ID and the protocol ID are exchanged or improperly defined. This will require a bit more thinking about how best to resolve these errors.

- Patch 4:
Based on the mod_auth_openidc documentation, it seems like OIDCRedirectURI should only be set once. I think the additional declarations should be removed.

- Patch 5:
Adding configuration for OIDCXForwardedHeaders is required for enabling the OIDC workflow when you do not have full HTTPS/TLS at every step of teh workflow. I believe we will need to add this capability.

- Patch 6:
We will need to manage mod_auth_openidc's behavior for calculating the session cookie if we plan to use multiple units of Keystone with load balancers or proxies. mod_auth_openidc can solve this problem with a shared cookie/session store like memcached, but I would prefer to not introduce an additional service if possible. The default behavior for mod_auth_oidc is to pass a session cookie to the web browser, and the cookie lasts for as long as the browser session lasts. Adding the "persistent" flag makes the cookie persist through multiple browser restarts. The cookie will still expire normally within the timeout configured in mod_auth_openidc. The default timeout is 300 seconds. While changing the Apache configuration is the simplest approach, this part of the work could use more thinking.

Thank you again, Gaetan, for your help and for carefully recording all of the changes to get the OIDC federation working.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-keystone-openidc/+/921708
Committed: https://opendev.org/openstack/charm-keystone-openidc/commit/ba5cd152fddf2208be98bb3138807a7fcb14fd73
Submitter: "Zuul (22348)"
Branch: master

commit ba5cd152fddf2208be98bb3138807a7fcb14fd73
Author: Jadon Naas <email address hidden>
Date: Mon Jun 10 16:40:48 2024 -0400

    Doc change: Add missing relation for Keystone OIDC configuration

    This change adds documentation on another relation you must have
    in order for you to configure Keystone and Horizon or logins
    using OpenIDC.

    Partial-Bug: 2065590
    Change-Id: I0deba42ef4869d5f10229ebe1aac08b8e68d3009

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-keystone-openidc/+/922049