Kubernetes Control Plane Charm

  1. Bug #1867645
  2. Comment #3

Comment 3 for bug 1867645

Revision history for this message
Km olsen (km-phones) wrote :

Same problem on kubernetes-master/1:

unable to load configmap based request-header-client-ca-file: Unauthorized

Cannot reproduce as it appears to be part of day to day running of Juju deployed CDK, perhaps as part of general snap upgrades?

(I am running vault instead of easyrsa)

juju status

--snipped--
kubernetes-master/0* active idle 8 192.168.70.25 6443/tcp Kubernetes master running.

kubernetes-master/1 blocked idle 9 192.168.70.12 6443/tcp Stopped services: kube-controller-manager

On kube-master/1
snap list
Name Version Rev Tracking Publisher Notes
cdk-addons 1.17.7 2655 1.17/stable canonical✓ in-cohort
core 16-2.45 9289 latest/stable canonical✓ core
kube-apiserver 1.17.7 1683 1.17/stable canonical✓ in-cohort
kube-controller-manager 1.17.7 1587 1.17/stable canonical✓ in-cohort
kube-proxy 1.17.7 1579 1.17/stable canonical✓ classic,in-cohort
kube-scheduler 1.17.7 1558 1.17/stable canonical✓ in-cohort
kubectl 1.17.7 1544 1.17/stable canonical✓ classic,in-cohort

juju debug-log -i unit-kubernetes-master-1 --replay --tail
--snipped--
unit-kubernetes-master-1: 09:16:05 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: reactive/kubernetes_master.py:2229:send_cluster_tag
unit-kubernetes-master-1: 09:16:06 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: reactive/kubernetes_master.py:2450:setup_keystone_user
unit-kubernetes-master-1: 09:16:06 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: reactive/kubernetes_master.py:2470:keystone_config
unit-kubernetes-master-1: 09:16:06 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: reactive/vault_kv.py:40:clear_ready
unit-kubernetes-master-1: 09:16:06 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: hooks/relations/openstack-integration/requires.py:84:remove_ready:openstack
unit-kubernetes-master-1: 09:16:06 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:kube-api-endpoint
unit-kubernetes-master-1: 09:16:06 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: hooks/relations/aws-integration/requires.py:106:remove_ready:aws
unit-kubernetes-master-1: 09:16:06 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: hooks/relations/vault-kv/requires.py:32:broken:vault-kv
unit-kubernetes-master-1: 09:16:06 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: hooks/relations/azure-integration/requires.py:114:remove_ready:azure
unit-kubernetes-master-1: 09:16:06 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: hooks/relations/kubernetes-cni/provides.py:10:changed:cni
unit-kubernetes-master-1: 09:16:07 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: hooks/relations/gcp-integration/requires.py:116:remove_ready:gcp
unit-kubernetes-master-1: 09:16:07 INFO unit.kubernetes-master/1.juju-log Invoking reactive handler: hooks/relations/tls-certificates/requires.py:79:joined:certificates
unit-kubernetes-master-1: 09:16:08 DEBUG unit.kubernetes-master/1.update-status active
unit-kubernetes-master-1: 09:16:08 DEBUG unit.kubernetes-master/1.update-status activating
unit-kubernetes-master-1: 09:16:08 DEBUG unit.kubernetes-master/1.update-status active
unit-kubernetes-master-1: 09:16:08 DEBUG unit.kubernetes-master/1.update-status active
unit-kubernetes-master-1: 09:16:08 INFO unit.kubernetes-master/1.juju-log status-set: blocked: Stopped services: kube-controller-manager
unit-kubernetes-master-1: 09:16:08 INFO juju.worker.uniter.operation ran "update-status" hook
unit-kubernetes-master-1: 09:16:08 INFO juju.util.exec run result: exit status 1

On the kubernetes-master/1 server

systemctl status snap.kube-controller-manager.daemon.service
● snap.kube-controller-manager.daemon.service - Service for snap application kube-controller-manager.daemon
   Loaded: loaded (/etc/systemd/system/snap.kube-controller-manager.daemon.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/snap.kube-controller-manager.daemon.service.d
           └─always-restart.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2020-07-01 09:19:13 UTC; 8s ago
  Process: 15532 ExecStart=/usr/bin/snap run kube-controller-manager.daemon (code=exited, status=1/FAILURE)
 Main PID: 15532 (code=exited, status=1/FAILURE)

journalctl -r -u snap.kube-controller-manager.daemon.service
-- Logs begin at Mon 2020-06-29 06:01:57 UTC, end at Wed 2020-07-01 09:20:00 UTC. --
Jul 01 09:19:58 juju-9afcf0-9 systemd[1]: snap.kube-controller-manager.daemon.service: Failed with result 'exit-code'.
Jul 01 09:19:58 juju-9afcf0-9 systemd[1]: snap.kube-controller-manager.daemon.service: Main process exited, code=exited, status=1/FAILURE
Jul 01 09:19:58 juju-9afcf0-9 kube-controller-manager.daemon[16007]: unable to load configmap based request-header-client-ca-file: Unauthorized
Jul 01 09:19:57 juju-9afcf0-9 kube-controller-manager.daemon[16007]: W0701 09:19:57.992781 16007 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserve
Jul 01 09:19:57 juju-9afcf0-9 kube-controller-manager.daemon[16007]: W0701 09:19:57.992705 16007 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserve
Jul 01 09:19:57 juju-9afcf0-9 kube-controller-manager.daemon[16007]: I0701 09:19:57.179531 16007 flags.go:33] FLAG: --vmodule=""
Jul 01 09:19:57 juju-9afcf0-9 kube-controller-manager.daemon[16007]: I0701 09:19:57.179524 16007 flags.go:33] FLAG: --version="false"
Jul 01 09:19:57 juju-9afcf0-9 kube-controller-manager.daemon[16007]: I0701 09:19:57.179519 16007 flags.go:33] FLAG: --v="2"
Jul 01 09:19:57 juju-9afcf0-9 kube-controller-manager.daemon[16007]: I0701 09:19:57.179515 16007 flags.go:33] FLAG: --use-service-account-cred

(I'm working on Getting a crashdump)