k8s-keystone-auth pod is failing with domain-scoped token

[Alexander Litvinov]

I am following
https://ubuntu.com/kubernetes/docs/troubleshooting#troubleshooting-keystoneldap-issues

After relating keystone and kubernetes-master, the kube-keystone.sh file generated on master unit is based on
https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/master/templates/kube-keystone.sh
and the generated token is domain scoped, for my admin user looks like this:

export OS_PROJECT_NAME=admin
export OS_DOMAIN_NAME=admin_domain
export OS_USERNAME=admin
export OS_PASSWORD=password

If I use this TOKEN with k8s-keystone-auth pod

(Image ID: rocks.canonical.com:443/cdk/k8scloudprovider/k8s-keystone-auth@sha256:d8e8b9200a94bbea11bf24d5f7bc3189d675165fc52220b6993eb5410be0ea5e)

this way

$ cat <<EOF | curl -ks -XPOST -d @- https://keystone:8443/webhook | python -mjson.tool
> {
> "apiVersion": "authentication.k8s.io/v1beta1",
> "kind": "TokenReview",
> "metadata": {
> "creationTimestamp": null
> },
> "spec": {
> "token": "TOKEN"
> }
> }
> EOF
Response
--------
No JSON object could be decoded
--------

and the pod fails with the following error in the log:

$ kubectl -n kube-system logs pod/k8s-keystone-auth-f6bcff8c4-j94jj

I0827 10:23:20.066076 1 keystone.go:186] Authorization policy updated.
I0827 10:24:37.540965 1 log.go:181] http2: panic serving 10.1.50.0:45430: runtime error: invalid memory address or nil pointer dereference
goroutine 13 [running]:
net/http.(*http2serverConn).runHandler.func1(0xc0000c22d0, 0xc0003d1f8e, 0xc000266000)
 /usr/local/go/src/net/http/h2_bundle.go:5711 +0x16f
panic(0x148b7a0, 0x2083ce0)
 /usr/local/go/src/runtime/panic.go:969 +0x175
k8s.io/cloud-provider-openstack/pkg/identity/keystone.(*Keystoner).GetTokenInfo(0xc000186670, 0xc000045920, 0x20, 0xc0003c02e0, 0x0, 0x147a500)
 /home/zuul/src/k8s.io/cloud-provider-openstack/pkg/identity/keystone/keystone.go:263 +0x4e8
I0827 10:30:12.156739 1 log.go:181] http2: panic serving 10.1.50.0:45566: runtime error: invalid memory address or nil pointer dereference
-------

I believe the issue is go client just returns nil instead of project is case if token is domain scoped
https://sourcegraph.com/github.com/sodafoundation/nbp/-/blob/vendor/github.com/gophercloud/gophercloud/openstack/identity/v3/tokens/results.go#L138

After I generate the token with project scope the following way instead of generated kube-keystone.sh
The pod works ok and has a response from keystone, nothing is failing in logs

curl -si -d @token-request.json -H "Content-type: application/json" http://<keystone-ip>:5000/v3/auth/tokens | awk '/X-Subject-Token/ {print $2}'

cat token-request.json
{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
                "user": {
                    "domain": {
                        "name": "admin_domain"
                    },
                    "name": "admin",
                    "password": "openstack"
                }
            }
        },
        "scope": {
            "project": {
                "domain": {
                    "name": "admin_domain"
                },
                "name": "admin"
            }
        }
    }
}

Some logs: https://pastebin.canonical.com/p/b5t2jD9Pfk/

Expected result: charm generates kube-keystone.sh which generates usable tokens for k8s-keystone-auth
(and possibly k8s-keystone-auth is not failing with invalid memory address or nil pointer dereference but that's other topic)

[Alexander Litvinov (alitvinov)]

subscribed ~field-high

[Nobuto Murata (nobuto)]

According to Alex, this is a possibly related change in upstream:
https://github.com/kubernetes/cloud-provider-openstack/commit/448033f0696f2b45703232382853c8dc728b6c7d#diff-1f3b752885c9183c0c998d65f9595298R78
Which is committed as a part of 1.18 and might explain why charmed k8s needs an equivalent update.

[Felipe Reyes (freyes)]

We have been bitten by this one in the field, we are generating tokens for the dashboard using the json suggested in the bug description without issues so far.

I have a script to reproduce a whole kerberos+ldap+keystone+k8s (keystone queens, k8s 1.17/stable) at https://gist.github.com/freyes/39c7c59183140cb8a3ed341b6ac1cea1 .

Is there some configuration that needs to be tested before committing a patch with the updated json?, otherwise I can propose a patch.

[George Kraft (cynerva)]

It looks like this occurs when getting a token for a user that doesn't have a default project set, and only when using the get_keystone_token function to obtain a token.

Updating the get_keystone_token function to use the json from the bug description looks like it would be a complete and proper fix to me.

> Is there some configuration that needs to be tested before committing a patch with the updated json?, otherwise I can propose a patch.

Not that I'm aware of. A patch would be much appreciated!

[Km olsen (km-phones)]

I came across the same thing and found out that the kubernetes-master charm needs to generate kube-keystone.sh with the project scope. ( the client-keystone-auth snap includes project scope data with its request )

To fix this, the kube-keystone.sh get_keystone_token function needs to includes the project scope ( which needs the domain id ).

This now works after updating the function manually by adding scope and inserting the domain ID and project name. Here is a working example:

get_keystone_token() {
  data='{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "'"${OS_USERNAME}"'",
          "domain": { "name": "'"${OS_DOMAIN_NAME}"'" },
          "password": "'"${OS_PASSWORD}"'"
        }
      }
    },
    "scope": {
      "project": {
        "domain": {
          "id": "2761f5ce781f42008494648c750bd077"
         },
         "name": "myproject"
      }
    }
  }
}'

The guide I used for keystone is here:
https://docs.openstack.org/keystone/latest/getting-started/architecture.html

[Felipe Reyes (freyes)]

Pull request https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/128

[Felipe Reyes (freyes)]

What's the work needed in cdk-addons?, I believe the task should be set to 'invalid'.