That function is automatically retried and succeeds later in the log, so that error is a red herring. (Perhaps we could improve the charm code so that it doesn't bother attempting that step until after the kube config has been written?) The actual issue is that the API service seems to have been running for about 10 minutes just fine and then is SIGTERM killed seemingly out of nowhere (see below). I have not seen that before and don't know what might have caused it.
---
Apr 03 09:05:39 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:39.844792 115586 httplog.go:89] "HTTP" verb="GET" URI="/api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=965&timeout=8m43s&timeoutSeconds=523&watch=true" latency="10.000475255s" userAgent="kube-controller-manager/v1.21.0 (linux/amd64) kubernetes/6b526c5" srcIP="192.168.33.168:49464" resp=0
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:46.849764 115586 client.go:360] parsed scheme: "passthrough"
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:46.849839 115586 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://192.168.33.172:2379 <nil> 0 <nil>}] <nil> <nil>}
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:46.849858 115586 clientconn.go:948] ClientConn switching balancer to "pick_first"
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:46.849999 115586 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc011e8f090, {CONNECTING <nil>}
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:46.866723 115586 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc011e8f090, {READY <nil>}
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:46.868046 115586 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:51.278603 115586 client.go:360] parsed scheme: "passthrough"
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:51.278681 115586 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://192.168.33.173:2379 <nil> 0 <nil>}] <nil> <nil>}
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:51.278700 115586 clientconn.go:948] ClientConn switching balancer to "pick_first"
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:51.278826 115586 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc011c924e0, {CONNECTING <nil>}
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:51.296269 115586 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc011c924e0, {READY <nil>}
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.daemon[115586]: I0403 09:05:51.297634 115586 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: State 'stop-sigterm' timed out. Killing.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: Killing process 115586 (kube-apiserver) with signal SIGKILL.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: Killing process 115611 (kube-apiserver) with signal SIGKILL.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: Killing process 115612 (n/a) with signal SIGKILL.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: Killing process 115620 (kube-apiserver) with signal SIGKILL.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: Killing process 115625 (n/a) with signal SIGKILL.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: Killing process 115627 (n/a) with signal SIGKILL.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: Killing process 115632 (n/a) with signal SIGKILL.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: Killing process 115746 (n/a) with signal SIGKILL.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: Main process exited, code=killed, status=9/KILL
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-apiserver.daemon.service: Failed with result 'timeout'.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: Stopped Service for snap application kube-apiserver.daemon.
That function is automatically retried and succeeds later in the log, so that error is a red herring. (Perhaps we could improve the charm code so that it doesn't bother attempting that step until after the kube config has been written?) The actual issue is that the API service seems to have been running for about 10 minutes just fine and then is SIGTERM killed seemingly out of nowhere (see below). I have not seen that before and don't know what might have caused it.
---
Apr 03 09:05:39 juju-8dd1b7-4-lxd-1 kube-apiserver. daemon[ 115586] : I0403 09:05:39.844792 115586 httplog.go:89] "HTTP" verb="GET" URI="/api/ v1/namespaces/ kube-system/ configmaps? allowWatchBookm arks=true& fieldSelector= metadata. name%3Dextensio n-apiserver- authentication& resourceVersion =965&timeout= 8m43s&timeoutSe conds=523& watch=true" latency= "10.000475255s" userAgent= "kube-controlle r-manager/ v1.21.0 (linux/amd64) kubernetes/6b526c5" srcIP=" 192.168. 33.168: 49464" resp=0 daemon[ 115586] : I0403 09:05:46.849764 115586 client.go:360] parsed scheme: "passthrough" daemon[ 115586] : I0403 09:05:46.849839 115586 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https:/ /192.168. 33.172: 2379 <nil> 0 <nil>}] <nil> <nil>} daemon[ 115586] : I0403 09:05:46.849858 115586 clientconn.go:948] ClientConn switching balancer to "pick_first" daemon[ 115586] : I0403 09:05:46.849999 115586 balancer_ conn_wrappers. go:78] pickfirstBalancer: HandleSubConnSt ateChange: 0xc011e8f090, {CONNECTING <nil>} daemon[ 115586] : I0403 09:05:46.866723 115586 balancer_ conn_wrappers. go:78] pickfirstBalancer: HandleSubConnSt ateChange: 0xc011e8f090, {READY <nil>} daemon[ 115586] : I0403 09:05:46.868046 115586 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing" daemon[ 115586] : I0403 09:05:51.278603 115586 client.go:360] parsed scheme: "passthrough" daemon[ 115586] : I0403 09:05:51.278681 115586 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https:/ /192.168. 33.173: 2379 <nil> 0 <nil>}] <nil> <nil>} daemon[ 115586] : I0403 09:05:51.278700 115586 clientconn.go:948] ClientConn switching balancer to "pick_first" daemon[ 115586] : I0403 09:05:51.278826 115586 balancer_ conn_wrappers. go:78] pickfirstBalancer: HandleSubConnSt ateChange: 0xc011c924e0, {CONNECTING <nil>} daemon[ 115586] : I0403 09:05:51.296269 115586 balancer_ conn_wrappers. go:78] pickfirstBalancer: HandleSubConnSt ateChange: 0xc011c924e0, {READY <nil>} daemon[ 115586] : I0403 09:05:51.297634 115586 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing" apiserver. daemon. service: State 'stop-sigterm' timed out. Killing. apiserver. daemon. service: Killing process 115586 (kube-apiserver) with signal SIGKILL. apiserver. daemon. service: Killing process 115611 (kube-apiserver) with signal SIGKILL. apiserver. daemon. service: Killing process 115612 (n/a) with signal SIGKILL. apiserver. daemon. service: Killing process 115620 (kube-apiserver) with signal SIGKILL. apiserver. daemon. service: Killing process 115625 (n/a) with signal SIGKILL. apiserver. daemon. service: Killing process 115627 (n/a) with signal SIGKILL. apiserver. daemon. service: Killing process 115632 (n/a) with signal SIGKILL. apiserver. daemon. service: Killing process 115746 (n/a) with signal SIGKILL. apiserver. daemon. service: Main process exited, code=killed, status=9/KILL apiserver. daemon. service: Failed with result 'timeout'. daemon.
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:46 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:05:51 juju-8dd1b7-4-lxd-1 kube-apiserver.
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: snap.kube-
Apr 03 09:06:04 juju-8dd1b7-4-lxd-1 systemd[1]: Stopped Service for snap application kube-apiserver.