Keystone produce error after trying to read application_credential even if not set

[Marcus Murwall]

If you are authed using application credentials and try to add a loadbalancer listener with TERMINATED_HTTPS, keystone produce an error causing a 500 internal error.

After digging through the code I found that it's caused by keystone adding application_credential as allowed method and then trying to read application_credential from auth payload, even when it is not set.
Octavia doesn't try to use application credentials in the payload, it tries to auth with method: ["token"]. It is keystone that adds application_credential as a method.

Octavia auth payload is created here:
https://opendev.org/openstack/octavia/src/branch/master/octavia/certificates/common/auth/barbican_acl.py#L87

The payload sent to keystone looks like this:
{
    "data": {
        "auth": {
            "identity": {
                "methods": ["token"],
                "token": {
                    "id": "<token id>"
                }
            },
            "scope": {
                "project": {
                    "id": "<project id>"
                }
            }
        }
    }
}

Keystone adds application_secret to allowed_methods here:
https://opendev.org/openstack/keystone/src/branch/master/keystone/api/_shared/authentication.py#L206

Keystone then tries to read the id of the application credential, which will fail as it is not included in the auth payload: https://opendev.org/openstack/keystone/src/branch/master/keystone/api/_shared/authentication.py#L210-L212

This cause a keystone error and you get a 500 internal error sent back to octavia.

Steps to reproduce:

1. Create an application credential with openstack application credential create.
2. Auth using the application credential
3. Try to add a terminated_https loadbalancer with openstack loadbalancer listener create

If you want to isolate the keystone auth failure without going through octavia you can do so with:
curl -H "Content-Type: application/json" https://<keystone url>/v3/auth/tokens -d '{"auth": {"identity": {"methods": ["token"], "token": {"id": "<token id>"}}, "scope": {"project": {"id": "<project id>"}}}}'

[Marcus Murwall (muran)]

I can create a patch that checks if application_credential is set and only then tries to read it.
But I'm not sure if the current enforcement of application_credential being set in the payload was deliberate or not.

[Michael Johnson (johnsom)]

I can reproduce this on yoga devstack.

Here is what I see going on:

1. User presents Octavia with an application token.
2. Octavia needs a token to talk to Barbican based on the presented token, so it calls keystone, via keystoneauth1 to get a token based on the app credential. https://github.com/openstack/octavia/blob/master/octavia/certificates/common/auth/barbican_acl.py#L87
3. This succeeds, Octavia is issued a token based on the app credential.
4. Octavia turns around and attempts to use this new token to access barbican (get the service catalog, etc. Note, it has not yet connected to barbican API).
5. Keystone fails as "application_credential" is listed in the method_name list, but what it has is just a token and no longer an application credential.
  a. method_names has ['token', 'application_credential']
  b. token_auth only has ['methods', 'token'] as we only have a token here.
  c. This blows up: https://github.com/openstack/keystone/blob/b0318a455ac5654adf6b5c6e3bc6fbabba1e5b39/keystone/api/_shared/authentication.py#L210 because it's just a token and not an application_credential.

So, I think one of a few things is wrong in keystone:
1. Something is mistakenly adding the "application_credential" to the auth_context "method_names".
2. The check for 'application_credential' in method_names should also check if there is actually a application_credential in the token_auth. (Note, I tried this and it resolves the issue)
3. keystoneauth1 is building/getting the token for Octavia incorrectly.

[Dmitriy Rabotyagov (noonedeadpunk)]

To have that said, other projects like cinder that have common usecase with barbican, just pass context and don't produce extra token: https://opendev.org/openstack/cinder/src/branch/master/cinder/volume/volume_utils.py#L1018-L1019

However when I tried same approach with octavia and used castellan RequestContext this changed nothing and same stack trace is produced from keystone side.

[Kristi Nikolla (knikolla)]

To add to the list of the 3 things that may be going wrong.

4. Keystone is stripping out the application_credential_id when the token is issued and then another token is issued based on that token. I'm getting a general feeling that this might be the issue.

We need to keep track from which application credential a token was created, for the 'restricted' flag.

[Dmitriy Rabotyagov (noonedeadpunk)]

I've pushed https://review.opendev.org/c/openstack/keystone/+/851979 to evaluate this bug report

[Marcus Murwall (muran)]

Where do we stand on this now then?
Reading discussions in https://review.opendev.org/c/openstack/keystone/+/851979 and https://review.opendev.org/c/openstack/keystone/+/727808 am I to interpret it as it is not something that will be fixed in keystone but instead something that needs to be fixed in octavia?
Or are we deciding that you can't use application credentials for this use case? That is, creating a https_terminated loadbalancer?

[Dmitriy Rabotyagov (noonedeadpunk)]

Well, from what I got, code refactoring is required for Octavia, as creating token from application credentials token was never supported by keystone.

There's a bug report for Octavia for the related topic https://storyboard.openstack.org/#!/story/2007619

While it seems to be worked on, I think it needs another round

[Rishat Azizov (razizov)]

Hello!

Bug fixed from octavia side: https://review.opendev.org/c/openstack/octavia/+/854563

https://storyboard.openstack.org/#!/story/2007619

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Dmitriy Rabotyagov <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/keystone/+/851979