HWE Kernel causes incompatable behavior with Kubernetes

[Stefan Lasiewski]

This is a follow up from a bug that I filed for Kubernetes: https://github.com/kubernetes/kubernetes/issues/95409

**What happened**:

I'm running Kubernetes on my Ubuntu 18.04 systems. Kubernetes is installed via Rancher (RKE).

My Kubernetes 1.17 cluster (and Kubernetes 1.16 cluster) has 40-50 firewall rules that say `[unsupported revision]`. I am concerned this will cause problems with

```
# iptables-save
...
-A KUBE-SEP-AAAREDACTED1 -s 10.99.99.27/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-AAAREDACTED1 -p tcp -m tcp -j DNAT [unsupported revision]
-A KUBE-SEP-AAAREDACTED2 -s 10.99.99.40/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-AAAREDACTED2 -p tcp -m tcp -j DNAT [unsupported revision]
-A KUBE-SEP-AAAREDACTED3 -s 10.99.99.27/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-AAAREDACTED3 -p tcp -m tcp -j DNAT [unsupported revision]
```

**What you expected to happen**:

I expect these firewall rules to be valid, like so:

```
# iptables-save
...
-A KUBE-SEP-AAAREDACTED1 -s 10.99.99.27/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-AAAREDACTED1 -p tcp -m tcp -j DNAT --to-destination 10.99.99.27:9153
-A KUBE-SEP-AAAREDACTED2 -s 10.99.99.40/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-AAAREDACTED2 -p tcp -m tcp -j DNAT --to-destination 10.99.99.40:3306
-A KUBE-SEP-AAAREDACTED3 -s 10.99.99.27/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-AAAREDACTED3 -p tcp -m tcp -j DNAT --to-destination 10.99.99.27:53
```

**How to reproduce it (as minimally and precisely as possible)**:

1. Allocate a worker node
1. Install Ubuntu 18.04.5
2. Install Kubernetes with Canal
  * I'm using Rancher & RKE, and I assume this happens with vanilla versions of Kubernetes as well.
3. Install the Ubuntu LTS Hardware Enablement (HWE) kernel via https://wiki.ubuntu.com/Kernel/LTSEnablementStack#Server
4. Reboot
5. When the system comes back online & Docker is running, look for invalid iptables rules as shown above.

** Environment **

- 18.04.5 LTS (Bionic Beaver)
- Kernel - Linux cntest13 5.4.0-48-generic #52~18.04.1-Ubuntu SMP Thu Sep 10 12:50:22 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux\
- Kubernetes 1.17.11 - Installed via RKE
- Canal version: rancher/calico-node:v3.13.4 (Not sure how to tell the other version numbers)
- Docker version: 18.9.9

Happens with both Bare Metal and VM systems:
- Bare metal nodes - AMD EPYC 7452 32-Core Processor, large memory, multiple NICs to different networks
- VMs on VMware vSphere

Default iptables version:

```
# iptables --version
iptables v1.6.1
# ls -ld `which iptables`
lrwxrwxrwx 1 root root 13 Nov 12 2017 /sbin/iptables -> xtables-multi
#
```

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-generic-hwe-18.04 5.4.0.48.52~18.04.42
ProcVersionSignature: Ubuntu 5.4.0-48.52~18.04.1-generic 5.4.60
Uname: Linux 5.4.0-48-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.18
Architecture: amd64
Date: Tue Oct 13 19:19:20 2020
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US
 SHELL=/bin/bash
SourcePackage: linux-meta-hwe-5.4
UpgradeStatus: No upgrade log present (probably fresh install)

[Stefan Lasiewski (stefanlasiewski)]

This appears to be a bug with Ubuntu, not Kubernetes. Kube-proxy is reponsible for managing these rules.

The rules inside the container appear fine even though they show an error on the host.

On the host:

```
root@docker1:~# iptables-save | grep AAAREDACTED1
:KUBE-SEP-AAAREDACTED1 - [0:0]
-A KUBE-SEP-AAAREDACTED1 -s 10.99.99.190/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-AAAREDACTED1 -p tcp -m tcp -j DNAT [unsupported revision]
-A KUBE-SVC-123REDACTEDABC -j KUBE-SEP-AAAREDACTED1
root@docker1:/#
```

Inside the container:

```
root@docker1:~# docker exec -it kube-proxy bash
root@docker1:/# iptables-save | grep AAAREDACTED1
:KUBE-SEP-AAAREDACTED1 - [0:0]
-A KUBE-SEP-AAAREDACTED1 -s 10.99.99.190/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-AAAREDACTED1 -p tcp -m tcp -j DNAT --to-destination 10.99.99.190:24231
-A KUBE-SVC-123REDACTEDABC -j KUBE-SEP-AAAREDACTED1
root@docker1:/#
```

[Stefan Lasiewski (stefanlasiewski)]

This does not happen with the standard Linux kernel:

```
root@docker1:~# uname -a
Linux cntest13 4.15.0-118-generic #119-Ubuntu SMP Tue Sep 8 12:30:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@docker1:~# iptables-save |grep unsupported
root@docker1:~#
```

As a workaround, we have deinstalled the HWE kernel.

[Stefan Lasiewski (stefanlasiewski)]

One real question here:

Why does iptables behave differently when the HWE kernel v5 is installed instead of the standard Kernel v4. Is it possible to use a configuration option to prevent the `[unsupported revision]` errors?

[Andreas Hasenack (ahasenack)]

I know it's been a long time, but can you still get to the version of iptables that's running inside the k8s pod that is inserting these rules? I assume it was kube-router?

[Andreas Hasenack (ahasenack)]

I reproduced it in bionic with hwe kernel, and a privileged docker container with iptables 1.8.3, with host networking. Didn't try other versions yet.

[Stefan Lasiewski (stefanlasiewski)]

It looks like the version was iptables v1.8.3 according to my comment at [1]

```
root@kube-proxy:/# iptables-save --version
iptables-save v1.8.3 (legacy)
root@kube-proxy:/#
```

[1] https://github.com/kubernetes/kubernetes/issues/95409#issuecomment-706472275