Master services constantly restarting due to cert change

[Andrey Grebennikov]

In CDK-like deployment with latest versions of the charms and EasyRSA as the cert authority I can see that master components restarting every minute:

2019-04-27 03:15:24 INFO juju-log certificates:7: Certificate information changed, restarting api server
2019-04-27 03:15:25 INFO juju-log certificates:7: status-set: maintenance: Restarting snap.kube-apiserver.daemon service
--
2019-04-27 03:16:24 INFO juju-log certificates:7: Certificate information changed, restarting api server
2019-04-27 03:16:24 INFO juju-log certificates:7: status-set: maintenance: Restarting snap.kube-apiserver.daemon service
--
2019-04-27 03:17:17 INFO juju-log certificates:7: Certificate information changed, restarting api server
2019-04-27 03:17:17 INFO juju-log certificates:7: status-set: maintenance: Restarting snap.kube-apiserver.daemon service

On the side of EasyRSA charm it looks like the handler of "client-relation-changed" is called every 30 seconds or even more often:
https://pastebin.ubuntu.com/p/ZBsDqG37xf/

[Tim Van Steenburgh (tvansteenburgh)]

From the bug template:

"When reporting a Charmed Kubernetes problem, please run CDK Field Agent (https://github.com/juju-solutions/cdk-field-agent) and include the resulting tarball as an attachment to your bug."

[Andrey Grebennikov (agrebennikov)]

Tim,
Can you please elaborate on this script a little bit?
I ran it and it got stuck on the step "juju debug-log --replay" - it seems this option let the command run until it is interrupted, but if I do so - remaining commands from the script don't run obviously...

Though I uploaded the archive with the logs collected during this and previous commands.

[Mike Wilson (knobby)]

Andrey,

The script takes some time to complete and this tar isn't a complete output. Can you let it run to completion and also provide your bundle? I don't see an integrator application, is this deployed on Openstack?

[Mike Wilson (knobby)]

I would also note that on a newly deployed cluster on AWS I am not seeing this thrashing of the cert. More information for reproduction would be very useful here so we can help.

[George Kraft (cynerva)]

Per IRC, it appears that on every hook, kubernetes-master is slightly changing the SANs on its certificate request.

The likely culprit is here:

https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/1a02cd35d04cc336fbd63f88bdff1a3c40a574ad/reactive/kubernetes_master.py#L875-L876
https://github.com/juju-solutions/interface-public-address/blob/c878f946af0e868cd1431da26e789c60b3a7eeaf/requires.py#L21

[George Kraft (cynerva)]

We don't see this in typical deployments because the kubernetes-master:loadbalancer interface only has a single unit related to it. Andrey's bundle has 4 units attached to it: 3 units of kubeapi-load-balancer, 1 unit of keepalived.

[George Kraft (cynerva)]

PR from cory_fu: https://github.com/juju-solutions/interface-tls-certificates/pull/17

[Mike Wilson (knobby)]

https://github.com/juju-solutions/interface-public-address/pull/5

[Mike Wilson (knobby)]

More from cory_fu: https://github.com/charmed-kubernetes/charm-kubernetes-worker/pull/9 and https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/18

[Mike Wilson (knobby)]

Fixes available in kubernetes-master revision 665 and greater and kubernetes-worker revision 526 and greater.

[George Kraft (cynerva)]

PRs to stable branches:
https://github.com/charmed-kubernetes/interface-public-address/pull/1
https://github.com/charmed-kubernetes/interface-tls-certificates/pull/1
https://github.com/charmed-kubernetes/charm-kubernetes-worker/pull/15
https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/23