wrong etcd_connection_string on kubernetes-master charm
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
High
|
Joseph Phillips | ||
2.8 |
Fix Released
|
High
|
Joseph Phillips |
Bug Description
Hello
This is case on manual deployment environment.
1. bootstrap & deploy juju 2.3.7 & k8s with specific revision
2. I got issue like below, but it is resolved when I upgrade juju to 2.4.7
- subprocess.
3. so, I upgraded juju controller and model to 2.4.7
4. There are different issue.
- I analyzed it and found out that there are wrong IP in kube-apiserver args ( etcd-servers )
- --etcd-servers="https:/
- /var/snap/
5. I also needed to change canal argument manually like you did before.
- It points 127.0.1.1 as master as well
6. I modified them manually, k8s cluster worked fine.
7. then, I tried to upgrad charm to lastest(etcd, kubernetes-master)
8. kubernetes-master's configuration reverted to wrong IP
- --etcd-servers="https:/
I checked code quickly and found out below function gets this info
etcd.get_
Some info is here
#######
juju run --unit etcd/0 unit-get public-address
node-02.maas
juju run --unit etcd/0 unit-get private-address
node-02.maas
#######
cat /var/snap/
--advertise-
--min-request-
--etcd-
--etcd-
--etcd-
--etcd-servers="https:/
--storage-
--tls-cert-
--tls-private-
--insecure-
--insecure-
--audit-
--audit-
--audit-
--audit-
--basic-
--client-
--requestheader
--requestheader
--requestheader
--requestheader
--requestheader
--service-
--token-
--authorization
--admission-
--allow-
--enable-
--kubelet-
--kubelet-
--kubelet-
--kubelet-
--proxy-
--proxy-
--service-
--v="4"
#######
juju status
tnode01: Tue Jun 4 18:12:06 2019
Model Controller Cloud/Region Version SLA Timestamp
maas maas maas 2.4.7 unsupported 18:12:07+09:00
App Version Status Scale Charm Store Rev OS Notes
canal 0.10.0/2.6.12 active 5 canal jujucharms 604 ubuntu
easyrsa 3.0.1 active 1 easyrsa jujucharms 231 ubuntu
etcd 3.2.10 active 3 etcd jujucharms 426 ubuntu
kubeapi-
kubernetes-master 1.12.8 blocked 2 kubernetes-master jujucharms 678 ubuntu
kubernetes-worker 1.12.8 active 3 kubernetes-worker jujucharms 536 ubuntu
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 node-01.maas Certificate Authority connected.
etcd/0 active idle 1 node-02.maas 2379/tcp Healthy with 3 known peers
etcd/1* active idle 2 node-03.maas 2379/tcp Healthy with 3 known peers
etcd/2 active idle 3 node-04.maas 2379/tcp Healthy with 3 known peers
kubeapi-
kubernetes-master/0 maintenance idle 5 node-06.maas 6443/tcp Writing kubeconfig file.
canal/4 active idle node-06.maas Flannel subnet 172.19.88.1/24
kubernetes-
canal/3 active idle node-07.maas Flannel subnet 172.19.62.1/24
kubernetes-
canal/0* active idle node-08.maas Flannel subnet 172.19.63.1/24
kubernetes-worker/1 active executing 8 node-09.maas Kubernetes worker running.
canal/2 active idle node-09.maas Flannel subnet 172.19.19.1/24
kubernetes-worker/2 active executing 9 node-10.maas Kubernetes worker running.
canal/1 active idle node-10.maas Flannel subnet 172.19.57.1/24
Entity Meter status Message
model amber user verification pending
Machine State DNS Inst id Series AZ Message
0 started node-01.maas manual:node-01.maas xenial Manually provisioned machine
1 started node-02.maas manual:node-02.maas xenial Manually provisioned machine
2 started node-03.maas manual:node-03.maas xenial Manually provisioned machine
3 started node-04.maas manual:node-04.maas xenial Manually provisioned machine
4 started node-05.maas manual:node-05.maas xenial Manually provisioned machine
5 started node-06.maas manual:node-06.maas xenial Manually provisioned machine
6 started node-07.maas manual:node-07.maas xenial Manually provisioned machine
7 started node-08.maas manual:node-08.maas xenial Manually provisioned machine
8 started node-09.maas manual:node-09.maas xenial Manually provisioned machine
9 started node-10.maas manual:node-10.maas xenial Manually provisioned machine
tags: | added: sts |
no longer affects: | charm-etcd |
no longer affects: | charm-kubernetes-master |
Changed in juju: | |
status: | New → Triaged |
importance: | Undecided → Low |
tags: | added: network |
Changed in juju: | |
milestone: | none → 2.7.7 |
importance: | Low → High |
Changed in juju: | |
status: | Triaged → In Progress |
assignee: | nobody → Joseph Phillips (manadart) |
Changed in juju: | |
status: | In Progress → Fix Committed |
Changed in juju: | |
status: | Fix Committed → Fix Released |
the connection string comes from etcd charm itself, it sets it in the relation with the key "connection_ string" , this string is built with the information coming from get_ingress_ addresses( )[0] which relies on network_get() and fallbacks to unit_private_ip(), so I'm adding a task for the etcd charm as well.
[0] https:/ /github. com/juju- solutions/ layer-etcd/ blob/master/ lib/etcd_ lib.py# L4