It appears to be that service account called calico-policy-controller in kube-system namespace does not have proper permissions to list (and maybe do other actions after listing) for pods, namespaces and networkpolicies.
juju status: http://paste.ubuntu.com/p/ZMXbYYRVTm/
bundle: http://paste.ubuntu.com/p/N8YvFGQ9VY/
kubectl logs -n kube-system calico-policy-controller-675499888b-6sxsb | grep cannot
http://paste.ubuntu.com/p/c2KMw74rxr/
E0220 19:43:16.246620 1 reflector.go:201] github.com/projectcalico/kube-controllers/pkg/controllers/namespace/namespace_controller.go:151: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:calico-policy-controller" cannot list resource "namespaces" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "system:basic-user" not found, clusterrole.rbac.authorization.k8s.io "calico-policy-controller" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found]
E0220 19:43:16.253464 1 reflector.go:201] github.com/projectcalico/kube-controllers/pkg/controllers/pod/pod_controller.go:201: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:calico-policy-controller" cannot list resource "pods" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "system:basic-user" not found, clusterrole.rbac.authorization.k8s.io "calico-policy-controller" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found]
E0220 19:53:03.880187 1 reflector.go:201] github.com/projectcalico/kube-controllers/pkg/controllers/networkpolicy/policy_controller.go:192: Failed to list *extensions.NetworkPolicy: networkpolicies.extensions is forbidden: User "system:serviceaccount:kube-system:calico-policy-controller" cannot list resource "networkpolicies" in API group "extensions" at the cluster scope
It's strange though, because the role and the binding was created before the first log message in the policy controller.
kubectl logs -n kube-system calico-policy-controller-675499888b-6sxsb | head -n1
2019-02-20 13:52:49.022 [INFO][1] main.go 66: Loaded configuration from environment config=&config.Config{LogLevel:"info", ReconcilerPeriod:"5m", EnabledControllers:"policy,profile,workloadendpoint", WorkloadEndpointWorkers:1, ProfileWorkers:1, PolicyWorkers:1, NodeWorkers:1, Kubeconfig:""}
kubectl logs -n kube-system calico-policy-controller-675499888b-6sxsb | grep 'Failed to list' | head -n1
E0220 13:55:51.792831 1 reflector.go:201] github.com/projectcalico/kube-controllers/pkg/controllers/namespace/namespace_controller.go:151: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:calico-policy-controller" cannot list resource "namespaces" in API group "" at the cluster scope
kubectl get clusterrole -o yaml -n kube-system calico-policy-controller
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRole","metadata":{"annotations":{},"name":"calico-policy-controller"},"rules":[{"apiGroups":["","extensions"],"resources":["pods","namespaces","networkpolicies"],"verbs":["watch","list"]}]}
creationTimestamp: "2019-02-20T13:51:42Z"
name: calico-policy-controller
resourceVersion: "414"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/calico-policy-controller
uid: a74f9d97-3516-11e9-b52d-78e7d124d998
rules:
- apiGroups:
- ""
- extensions
resources:
- pods
- namespaces
- networkpolicies
verbs:
- watch
- list
kubectl get clusterrolebinding -o yaml -n kube-system calico-policy-controller
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"calico-policy-controller"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"calico-policy-controller"},"subjects":[{"kind":"ServiceAccount","name":"calico-policy-controller","namespace":"kube-system"}]}
creationTimestamp: "2019-02-20T13:51:42Z"
name: calico-policy-controller
resourceVersion: "415"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/calico-policy-controller
uid: a750cf3c-3516-11e9-b52d-78e7d124d998
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: calico-policy-controller
subjects:
- kind: ServiceAccount
name: calico-policy-controller
namespace: kube-system
