Certs problem with all worker nodes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
EasyRSA Charm |
New
|
Undecided
|
Unassigned |
Bug Description
My cluster has been running fine for two weeks.
But out of a sudden don't know what exactly happened on all work nodes these CA problem just pop out.
```
ychuang@
NAME STATUS ROLES AGE VERSION
bubnicki NotReady <none> 19d v1.13.2
karpinski NotReady <none> 31d v1.13.1
```
On worker node's /var/log/syslog
```
Feb 17 16:15:42 Bubnicki kubelet.
```
I have searched on line and found this post. https:/
Saying this means my client's certificate is invalid because either:
The server doesn't trust the client's signing certificate authority
The client doesn't trust the server's signing certificate authority
The certificate's DN doesn't match the hostname
Below is my kube-apiserver start parameters.
```
root@kubernetes
/snap/kube-
--advertise-
--min-request-
--etcd-
--etcd-
--etcd-
--etcd-servers=https:/
--storage-
--tls-cert-
--tls-private-
--insecure-
--insecure-
--audit-
--audit-
--audit-
--audit-
--basic-
--client-
--requestheader
--requestheader
--requestheader
--requestheader
--requestheader
--service-
--token-
--authorization
--admission-
--allow-privileged
--enable-
--kubelet-
--kubelet-
--kubelet-
--kubelet-
--proxy-
--proxy-
--service-
--logtostderr
--v=4
```
I have checked on all my certs for time and they all seem legit.
```
root@kubernetes
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
Signature Algorithm: sha256WithRSAEn
Issuer: CN = 172.29.100.186
Validity
Not Before: Jan 16 20:16:29 2019 GMT
Not After : Jan 13 20:16:29 2029 GMT
```
Cann someone point it to me what else I should check? and how do I fix this issue?
More info.
```
root@Bubnicki:
* Trying 172.29.100.185...
* TCP_NODELAY set
* Connected to 172.29.100.185 (172.29.100.185) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=kubernetes-
* start date: Jan 16 20:26:12 2019 GMT
* expire date: Jan 13 20:26:12 2029 GMT
* subjectAltName: host "172.29.100.185" matched cert's IP address!
* subjectAltName: host "172.29.100.185" matched cert's IP address!
* issuer: CN=172.29.100.186
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5563237f0900)
> GET /api/v1/nodes HTTP/2
> Host: 172.29.100.185:6443
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT
< HTTP/2 403
< audit-id: 0de66be1-
< content-type: application/json
< x-content-
< content-length: 308
< date: Mon, 18 Feb 2019 09:49:37 GMT
<
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "nodes is forbidden: User \"kubernetes-
"reason": "Forbidden",
"details": {
"kind": "nodes"
},
"code": 403
* Connection #0 to host 172.29.100.185 left intact
```
So I have create a service account and bind it with user "kubernetes- worker_ 3".
after binding curl to api-server is working now.
One simple question, can anyone tell me who to restart Kubelet in CDK's way?
``` /172.29. 100.185: 6443/api/ v1/nodes --cert /root/cdk/ server. crt --key /root/cdk/ server. key
root@Bubnicki:~# curl -v https:/
{ rsion": "4518487"
"kind": "NodeList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/nodes",
"resourceVe
},
...
```