client-keystone-auth fails with valid domain-scoped novarc or with valid project-scoped novarc with different domains for the project or user

[Nikolay Vinogradov]

Hi,

I've deployed K8s 1.20 on top of OpenStack and I needed to enable keystone LDAP integration as described in [1]. The problematic tool is client-keystone-auth [2], which is used in ~/.kube/config as the keystone token generator. The tricky part in my case was domains configuration in the underlying keystone. There I had 2 domains:
1) "K8s domain", which contained the project I used to deploy Charmed K8s;
2) "LDAP domain", which contained the LDAP users that must be granted the access to the K8s cluster, by means of adding a role assignments of these users to the K8s project in the K8s domain. This domain doesn't have any projects that the LDAP users have access to.

Keystone supports: a) domain scoped tokens and b) project scoped tokens [4]. It seemed logical to use domain scoped tokens for K8s LDAP authentication, so we started with option a). If we prepare a valid domain-scoped novarc that can be used openstack CLI, client-keystone-auth would not allow us to use that, because it will prompt for the project name unless we specify it. With Keystone we could also use option b) with different OS_USER_DOMAIN_NAME and OS_PROJECT_DOMAIN_NAME, that would also be accepted by OpenStack CLI, but client-keystone-auth won't allow that as it doesn't support OS_USER_DMAIN_NAME as it can be seen in [3].

The workaround is to use 'openstack credentials create <cred_name>' based on the valid project scoped novarc, as client-keystone-auth supports working with the app. credentials and the creds inherit all the roles of the creating user [3]:
OS_APPLICATION_CREDENTIAL_SECRET=<SECRET_HASH>
OS_APPLICATION_CREDENTIAL_ID=<SECRET_ID>
OS_APPLICATION_CREDENTIAL_NAME=kubectl

Since the charm provides the .kube/config that uses the snapped tool to generate keystone token, it is not clear from [1] how to deal with the token scoping.

[1] https://ubuntu.com/kubernetes/docs/ldap
[2] https://snapcraft.io/install/client-keystone-auth/ubuntu
[3] https://github.com/kubernetes/cloud-provider-openstack/blob/master/cmd/client-keystone-auth/main.go
[4] https://docs.openstack.org/keystone/latest/admin/tokens-overview.html

[George Kraft (cynerva)]

Targeting this against CK bundles for now, until we have a proper place to file issues for the client-keystone-auth snap.