Master needs iptables target to reach cluster services
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Kubernetes Control Plane Charm |
Fix Released
|
Undecided
|
Mike Wilson |
Bug Description
Problem with iptables, if kubelets are not running on the same host as the master.
We have the following setup
api-loadbalancer <-> (2x) kubernetes-master <-> number of workers
For authentication keystone is used.
The problem experienced was that we could not do token authentication for the k8s cluster, while all components appeared to be installed and configured correctly, and the syslog on the k8s-masters where full with lines like:
Jun 5 06:25:10 juju-1b688f-
We finally got this setup running by manually creating a KUBE-MARK-DROP rule with the commands:
sudo iptables -t NAT -N KUBE-MARK-DROP
sudo iptables -t NAT -A KUBE-MARK-DROP -j MARK --set-mark 0x8000/0x8000
When checking the code we found that this line should be inserted via the function in pkg/kubelet/
As we are not running any kubelet on the kubernetes-master this code is not called, so the source of the origin is clear, but how can we prevent this problem from happening in the future, or after a host restart?
kubernetes running is stable/1.13 snap.
On the master:
# snap list
Name Version Rev Tracking Publisher Notes
canonical-livepatch 9.3.0 77 stable canonical✓ -
cdk-addons 1.13.5 875 1.13/edge canonical✓ -
core 16-2.39 6964 stable canonical✓ core
kube-apiserver 1.13.6 984 1.13/edge canonical✓ -
kube-controller
kube-proxy 1.13.6 992 1.13/edge canonical✓ classic
kube-scheduler 1.13.6 985 1.13/edge canonical✓ -
kubectl 1.13.6 991 1.13/edge canonical✓ classic
On the workers:
sudo snap list
Name Version Rev Tracking Publisher Notes
canonical-livepatch 9.3.0 77 stable canonical✓ -
core 16-2.39 6964 stable canonical✓ core
kube-proxy 1.13.6 992 1.13 canonical✓ classic
kubectl 1.13.6 991 1.13 canonical✓ classic
kubelet 1.13.6 995 1.13 canonical✓ classic
All managed by charms:
canal 0.10.0/2.6.12 active 8 canal jujucharms 610 ubuntu
kubeapi-
kubernetes-master 1.13.6 active 2 kubernetes-master jujucharms 646 ubuntu
kubernetes-worker 1.13.6 active 6 kubernetes-worker jujucharms 519 ubuntu exposed
openstack-
(edit by afreiberger to update the KUBE-MARK-DROP table --set-mark option to that of comments #1 and #3)
tags: | added: canonical-bootstack |
description: | updated |
Changed in charm-kubernetes-master: | |
milestone: | none → 1.15+ck1 |
summary: |
- Master will not function without kubelet being active + Master needs iptables target to reach cluster services |
Changed in charm-kubernetes-master: | |
assignee: | nobody → Mike Wilson (knobby) |
Changed in charm-kubernetes-master: | |
status: | Fix Committed → Fix Released |
not trying to nitpick, but for clarity, the commands should be:
sudo iptables -t nat -N KUBE-MARK-DROP
sudo iptables -t nat -A KUBE-MARK-DROP -j MARK --set-mark 0x8000