API server cannot connect to Metrics service
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Kubernetes Control Plane Charm |
Triaged
|
Medium
|
Unassigned | ||
Kubernetes Worker Charm |
Triaged
|
Medium
|
Unassigned |
Bug Description
Removed old deployment and did a fresh install just to be sure.
Cloud environment: Ubuntu MaaS using KVM
Juju: 2.7.5
Used bundle:
https:/
Reduced number of nodes and limited CPU/RAM so all environment can fit on a single host
Deployment completed successfully and Kubernetes is operational, i.e. I can do deployments.
However "kubectl top" does not work. After half a day of reading manuals, github issues I have stumbled upon the following command: kubectl describe apiservice v1beta1.
In the output I see:
-------
Status:
Conditions:
Last Transition Time: 2020-04-
Message: failing or missing response from https:/
Reason: FailedDiscovery
Status: False
Type: Available
Events: <none>
-------
However metrics server itself is working fine, if I proxy port and run Curl I do get a metrics service response. This IP address is reachable from user pod or worker node too. However it is not reachable on master node. It times out. On a master node I see calico running and routes to metrics service IP but it does not respond
-------------------
root@tops-calf:~# ip r l
default via 192.168.101.1 dev eth0 proto static
192.168.66.64/26 via 192.168.101.7 dev eth0 proto bird
192.168.101.0/24 dev eth0 proto kernel scope link src 192.168.101.5
192.168.201.128/26 via 192.168.101.6 dev eth0 proto bird
------------------
What calico or metrics server options do I miss so that API can communicate with metrics server?
PS. Crashdump is 200MB. Should I really attach that?
Changed in charmed-kubernetes-bundles: | |
importance: | Undecided → High |
Changed in charm-kubernetes-master: | |
importance: | Undecided → High |
Changed in charm-kubernetes-worker: | |
importance: | Undecided → High |
no longer affects: | charmed-kubernetes-bundles |
Changed in charm-kubernetes-master: | |
status: | New → Triaged |
Changed in charm-kubernetes-worker: | |
status: | New → Triaged |
Changed in charm-kubernetes-master: | |
importance: | High → Medium |
Changed in charm-kubernetes-worker: | |
importance: | High → Medium |
I'm not sure if this is the cause, but one major issue I'm seeing here is that the Calico charm's default CIDR is 192.168.0.0/16, which conflicts with the 192.168.101.0/24 network that you're using.
Can you try setting the Calico charm's cidr option to one that doesn't overlap with your 192.168.101.0/24 subnet? If you can, I recommend setting the cidr config at deploy time by using a bundle overlay - changing the config on a live cluster might not work.