RBAC: K8S Master charm hangs trying to deploy kube-system pods

[Pedro Guimarães]

Running:
k8s-master version 700
k8s-worker version 552
calico version 641
Bundle: https://drive.google.com/open?id=1lsq_5b_W5g8UcbsFF9qopgVhss2x7elI

If I start deployment with authorization-mode set to "AlwaysAllow" and then move to "RBAC,Node" later, it works. Although there will probably consequences later, such as when we try to add new workers. Setting authorization-mode="RBAC,Node" on the bundle triggers the issue below.

RBAC enabled with:
$ juju config kubernetes-master authorization-mode
RBAC,Node

I am seeing something similar to: https://github.com/kubernetes-sigs/kubespray/issues/3980

Nodes are hanging with: Warning FailedCreatePodSandBox 31s (x510 over 113m) kubelet, juju-2a829d-kubernetes-5 (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "34b02a4845859abe91965f3a99d83d0d481ac95e79d97d1961a5dd008b6afcfe": namespaces "kube-system" is forbidden: User "system:node:juju-2a829d-kubernetes-5" cannot get resource "namespaces" in API group "" in the namespace "kube-system"

More complete log: https://pastebin.canonical.com/p/5jYDy4Q92f/

I've tried to run:
kubectl apply -f https://docs.projectcalico.org/v3.6/manifests/rbac-etcd-calico.yaml

As instructed on https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/integration#role-based-access-control-rbac
Which essentially creates kube-calico-controller role.

I can see some RBAC / Calico related issues, but I see no particular relation to these bugs:
https://bugs.launchpad.net/charm-kubernetes-master/+bug/1816856
https://bugs.launchpad.net/charm-calico/+bug/1809357

[Pedro Guimarães (pguimaraes)]

Logs from collect.py on: https://drive.google.com/open?id=18sOa4qR569_QArOC2IikkX6UjRn6M3vV

[Pedro Guimarães (pguimaraes)]

Defining system:node:juju-CODE-NUM on Cluster Role Binding can resolve this issue, such as:

for n in $(kubectl get nodes | awk '{print $1}' | tail -n +2); do cp test_binding.yaml.template $n.yaml; sed -i "s/#REPLACEME/$n/g" $n.yaml; kubectl apply -f "$n.yaml"; done

And then, we have a template for clusterrolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node:#REPLACEME
subjects:
- kind: User
  name: system:node:#REPLACEME
  apiGroup: rbac.authorization.k8s.io
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node

[Pedro Guimarães (pguimaraes)]

However, after doing so, I see pods randomly failing because of "dial i/o timeout" trying to reach for API Server or egress traffic

[Pedro Guimarães (pguimaraes)]

This is a blocker for current deployment as customer is demanding RBAC enabled. Raising to field-critical

[Pedro Guimarães (pguimaraes)]

Work-around for the ContainerCreating missing rights is:

More organized view here:
https://pastebin.ubuntu.com/p/9FKRXF8F9N/

(Printing here so we do not lose it with pastebin later)
kubectl get clusterrole -o yaml system:node > system_node.yaml
sed -i '/creationTimestamp:/d' system_node.yaml; sed -i '/uid:/d' system_node.yaml; sed -i '/selfLink:/d' system_node.yaml

tee -a ./system_node.yaml << EOF
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch
EOF
kubectl delete clusterrole system:node
kubectl apply -f ./system_node.yaml

tee ./test_binding.yaml.template << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node:#REPLACEME
subjects:
- kind: User
  name: system:node:#REPLACEME
  apiGroup: rbac.authorization.k8s.io
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
EOF

for n in $(kubectl get nodes | awk '{print $1}' | tail -n +2); do cp test_binding.yaml.template $n.yaml; sed -i "s/#REPLACEME/$n/g" $n.yaml; kubectl apply -f "$n.yaml"; done

[Pedro Guimarães (pguimaraes)]

However, this only solves the container creation issue. Just after, I get:
https://pastebin.canonical.com/p/mcMjbJnQxh/

For example:

2019-07-22T11:06:12.820Z [ERROR] plugin/errors: 2 8158258259395781619.3127874540815363623. HINFO: read udp 192.168.45.134:40172->46.49.128.122:53: i/o timeout
E0722 11:06:31.819208 1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:260: Failed to list *v1.Namespace: Get https://REDACTED:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp REDACTED:443: i/o timeout
E0722 11:06:31.819274 1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:253: Failed to list *v1.Service: Get https://REDACTED:443/api/v1/services?limit=500&resourceVersion=0: dial tcp REDACTED:443: i/o timeout
E0722 11:06:31.819285 1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:255: Failed to list *v1.Endpoints: Get https://REDACTED:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp REDACTED:443: i/o timeout

This does not happen if I set kubernetes-master with authorization-mode="AlwaysAllow"

[Pedro Guimarães (pguimaraes)]

Full logs here: https://drive.google.com/open?id=1qHqocEfM9BgQ9mpgvdonR7xth6X-8mjd

[Pedro Guimarães (pguimaraes)]

Default system:node with RBAC + Calico: https://pastebin.ubuntu.com/p/hwQkwZjtPn/

[Joseph Borg (joeborg)]

Have reproduced the first bug; the pod being denied creation. It appears to be an issue with Calico and CoreDNS, where Calico needs the `namespaces` permission, which wasn't the case. I will roll up the workaround Pedro mentioned into the charm.

I cannot currently reproduce the CoreDNS timeouts, so working with Pedro to see if we can find what I'm missing.

[Joseph Borg (joeborg)]

The patch can be achieved without needing to apply to every node (thus we can scale):

```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:node
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node
subjects:
- kind: User
  name: system:node
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
```

[Joseph Borg (joeborg)]

https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/38

[Pedro Guimarães (pguimaraes)]

Regarding timeouts post first fix, I believe this is certificate-related. I can see logs such as on CoreDNS: https://pastebin.canonical.com/p/HgpwSyXSQs/

Trying to wget those URLs from kubernetes-master, I get: https://pastebin.canonical.com/p/F5qFFqFMTS/

It is still unclear to me why does this issue happens only with RBAC enabled. I have no specific settings on Calico (as you can see on my bundle, above in the description)

[Joseph Borg (joeborg)]

Thanks for the improvements, have added them to the PR.

Moving onto the timeouts, I've tried with CoreDNS 1.5.2, with no success. I have also tried this fix https://stackoverflow.com/questions/53075796/coredns-pods-have-crashloopbackoff-or-error-state, which doesn't work in our case either. Most recorded issues seem to point to DNS config, but my feeling is that we're missing some RBAC policy and it's getting denied access.

[Joseph Borg (joeborg)]

Yep, if you curl the API request:

  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "endpoints is forbidden: User \"system:anonymous\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "endpoints"
  },
  "code": 403
}

[Joseph Borg (joeborg)]

^ Probably due to curl permissions.

Seeing a lot of

Forbidden: "/api/v1/namespaces?limit=500&resourceVersion=0", Reason: ""

in the kube-api log

[Joseph Borg (joeborg)]

I believe I've gotten to the bottom of the second issue. The timeouts are there, whether or not RBAC is set, but, with RBAC, CoreDNS fails Readiness; which I think is the actual issue.

The reason for this is RBAC permissions on Calico. Quick fix is to apply this over the top of the charm provided manifests: https://raw.githubusercontent.com/projectcalico/calico/master/v3.3/getting-started/kubernetes/installation/rbac.yaml (I also deleted the calico-kube-controllers pod and let the deployment re-fulfill it, but I don't think this is necessary).

I will go through the policy that the Calico charm has currently and see what's missing.

[Joseph Borg (joeborg)]

https://github.com/charmed-kubernetes/layer-calico/pull/43

[Pedro Guimarães (pguimaraes)]

Added Calico and Canal since we've seen both falling with this bug

[Pedro Guimarães (pguimaraes)]

Workaround: after kubernetes-master reaches status "waiting" with message "Waiting for X pods to start", do:

1) download the yaml file that I've uploaded on this comment and copy to k8s master/0
2) juju ssh kubernetes-master/0
3) kubectl apply -f reset-roles.yaml
4) Wait (maybe a minute)
5) Delete calico-kube-controllers pod: kubectl delete po -n kube-system calico-kube-controllers-XXXX-XXXX

Wait until kubectl get po -n kube-system sets down
Deployment is ready for use

[Pedro Guimarães (pguimaraes)]

^^ That solution applies to Calico case

[Joseph Borg (joeborg)]

Fixes moved entirely to CNI charm https://github.com/charmed-kubernetes/layer-calico/pull/43

[Pedro Guimarães (pguimaraes)]

Since this bug only affects kubernetes when deploying Calico or other CNIs, but not Flannel; fix was moved to those CNI-related charms and kubernetes-master is marked as "Won't Fix"

[Kevin W Monroe (kwmonroe)]

Fixes cherry-picked to stable:

https://github.com/charmed-kubernetes/layer-canal/commit/370d3d193673de26d1004a68516443932285ebfd
https://github.com/charmed-kubernetes/layer-calico/commit/71c65e10139ecc50c9646246fbb6bb55431e76f4
https://github.com/charmed-kubernetes/layer-tigera-secure-ee/commit/bc06699decccfc04896a3e06856bb70bbc5f997b

[Alexey Tsitsin (cicin)]

Hello, have this issue been tested with K8s 1.21.1 and calico 3.10.1?
It seems that we've faced the same bug coredns can not start:

 Warning FailedCreatePodSandBox 26m kubelet Failed to create pod sandbox: rpc error: code = Unknown des
c = failed to setup network for sandbox "55196a9be8653c48e315c0bacbdd31d8acef322eb6ad6296d8fab1298ff62415": namespaces "kube-system"
is forbidden: User "system:node:juju-a3aebe-k8s-8" cannot get resource "namespaces" in API group "" in the namespace "kube-system"

[Billy Olsen (billy-olsen)]

@cicin This bug was fixed released awhile ago. If you are seeing a similar issue, can you please raise a new bug with relevant information in it?

[Alexey Tsitsin (cicin)]

https://bugs.launchpad.net/charm-calico/+bug/1939386

[Alexey Tsitsin (cicin)]

Problem persists with Calico CNI.
Flannel CNI works well.