© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
a60fb26
(Get the code!)
Short version:
Found the culprit. Seems to be working. Not sure who's to blame.
It was missing route to container IP subnet on KVM virtualization host.
Long version.
All Kubernetes hosts are VM's on my workstation, deployed via Juju on MaaS. So networking was "automagically" created by MaaS and it seems to be working. I can communicate between nodes and maas/juju deployments works as expected.
sysctl net.ipv4.ip_forward shows 1 as expected.
Juju cluster machines:
Machine State DNS Inst id Series AZ Message
0 started 192.168.101.8 tight-gar bionic default Deployed
1 started 192.168.101.9 modest-aphid bionic default Deployed
2 started 192.168.101.10 live-hound bionic default Deployed
3 started 192.168.101.19 super-cub bionic default Deployed
4 started 192.168.101.17 fair-fish bionic default Deployed
5 started 192.168.101.18 happy-viper bionic default Deployed
Test Kubernetes pod (I have deployed a generic deployment in case metrics server being some kind of special).
bacila@juodas ~/Documents/
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deploymen
Regarding firewall, that was an idea worth checking. I was just not sure which corner should I kick into, and which corner should I listen for an echo to identify problem. ip a l listed 37 interfaces, so there were quite a few of them there :) So I started from good old tcpdump.
* SSH to worker node and run ping metrics_server_IP
* On the workstation run "tcpdump -i any icmp"
22:17:03.045019 IP 192.168.101.17 > 10.100.69.132: ICMP echo request, id 9430, seq 1, length 64
22:17:03.045067 IP 192.168.101.1 > 10.100.69.132: ICMP echo request, id 9430, seq 1, length 64
22:17:03.045299 IP 10.100.69.132 > 192.168.101.1: ICMP echo reply, id 9430, seq 1, length 64
So I see that ping reaches test pod and it sends reply to Juju network gateway but it does not reach back origin.
I have looked at main host route list and it lacked container network IP range, which means that ICMP response is sent to default GW.
root@juodas:
default via 192.168.4.254 dev enp8s0 proto dhcp metric 100
169.254.0.0/16 dev enp8s0 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.18.0.0/16 dev br-03ebb1003d68 proto kernel scope link src 172.18.0.1 linkdown
192.168.4.0/24 dev enp8s0 proto kernel scope link src 192.168.4.2 metric 100
192.168.100.0/24 dev virbr1 proto kernel scope link src 192.168.100.1 linkdown
192.168.101.0/24 dev virbr2 proto kernel scope link src 192.168.101.1
192.168.123.0/24 dev virbr0 proto kernel scope link src 192.168.123.1 linkdown
So I have added a route to container IP range via juju cluster GW.
root@juodas:
Voila! ping responds and metrics apiservice no longer shows an error and I have my metrics.
I have also added a service IP range so requests to service IP now works
ip r a 10.152.0.0/16 via 192.168.101.1
So it was missing routes on virtualization host. Why they are missing it is a question. Juju is run via non privileged user, so juju cannot update host level routing. So should it be done somehow via MaaS? Or simply install instructions updated with some manual post deployment tasks.