[bionic/18.04 only - fixed in newer ubuntu] iptables doesn't provide --random-fully flag

[Paul D]

Hello. This isn't strictly a bug, but more of an upgrade-request on the iptables package. Normally i wouldn't be inclined to submit such a bug report, but a user on the ubuntu-devel-discuss mailing list encouraged me to submit this anyway [1]. For our production systems, we're running into a kernel race condition bug, for which a workaround has been made available. The fix boils down to iptables having a new flag which it passes down to the kernel, to enable the workaround. However, the version of iptables in Ubuntu (v1.6.1) doesn't support that kernel feature yet. Specifically, it's introduced in this commit on the iptables codebase: https://git.netfilter.org/iptables/commit/?id=8b0da2130b8af3890ef20afb2305f11224bb39ec.

The feature we need from that commit is part of the v1.6.2 and newer iptables releases, but it looks like the Bionic, Cosmic, and Disco releases of Ubuntu all include v1.6.1 without that patch, so for now we're going to have to build iptables from source on our production machines. That shouldn't pose any huge issues, but of course, we'd prefer to be able to use the package from package management, or perhaps a backported package from a newer Ubuntu release.

So to summarise, this might be an invalid bug report, but consider it a vote to upgrade the packaged version of iptables. If this bug report is entirely inappropriate, then I apologise.

1. Link to thread on ubuntu-devel-discuss where I describe the problem and Nish suggests I file this bug report: https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2018-November/018181.html

Ubuntu version we're using:
Description: Ubuntu 18.04.1 LTS
Release: 18.04

$ apt-cache policy iptables
iptables:
  Installed: 1.6.1-2ubuntu2
  Candidate: 1.6.1-2ubuntu2
  Version table:
 *** 1.6.1-2ubuntu2 500
        500 http://us-west-2.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

Thanks for your time,

Paul

[Seth Arnold (seth-arnold)]

On Wed, Nov 28, 2018 at 02:47:10AM -0000, Paul D wrote:
> feature yet. Specifically, it's introduced in this commit on the
> iptables codebase:
> https://git.netfilter.org/iptables/commit/?id=8b0da2130b8af3890ef20afb2305f11224bb39ec.

I think this relies upon this kernel feature:

commit 34ce324019e76f6d93768d68343a0e78f464d754
Author: Daniel Borkmann <email address hidden>
Date: Fri Dec 20 22:40:29 2013 +0100

    netfilter: nf_nat: add full port randomization support

Given the date I'm optimistic that this should be supported in our
kernels, but some confirmation would be nice.

The iptables patch looks pretty simple.

It seems like a good candidate for an SRU to me.

Thanks

[Paul D (paul-rb)]

First off, wow, thank you for the speedy response! I am pretty confident that the kernel we're running supports this feature. Here's why:

$ uname -a
Linux ip-172-18-45-20 4.15.0-1025-aws #25-Ubuntu SMP Wed Oct 10 14:23:49 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Subsequently I found the patch in the kernel tree which I think introduces support for the feature I'm after:

https://github.com/torvalds/linux/commit/34ce324019e76f6d93768d68343a0e78f464d754#diff-b0b059e2ea29367b52cec84db46f33a3R11

If I read the list of tags on that page correctly, this has been part of kernels released since 3.14, which is quite a while ago now. In particular, I see that the kernel I'm running is in that list too. Although for absolute transparency I'm using the Ubuntu released on AWS EC2 instances, but I'm guessing this shouldn't make any difference.

Thank you again.

$ apt-cache policy linux-image-4.15.0-1025-aws
linux-image-4.15.0-1025-aws:
  Installed: 4.15.0-1025.25
  Candidate: 4.15.0-1025.25
  Version table:
 *** 4.15.0-1025.25 500
        500 http://us-west-2.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status

$ apt-cache policy linux-base
linux-base:
  Installed: 4.5ubuntu1
  Candidate: 4.5ubuntu1
  Version table:
 *** 4.5ubuntu1 500
        500 http://us-west-2.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sebastien Bacher (seb128)]

For doing an SRU the bug would need a description of the impact and a test case / description of the regression potential (see https://wiki.ubuntu.com/StableReleaseUpdates). Could someone provide those informations (and ideally a debdiff for the update)

[Todd B (spam-buiten)]

I found this thread because I ran into a problem with a brand-new installation of Kubernetes (K8s) running in AWS that was failing a large number of browser requests being serviced by the K8s cluster. There is a ton of detail about this problem at https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02. To make a very long story short, we need the 1.6.2+ version of iptables on Ubuntu because it supports the --random-fully flag. Without this, any K8s cluster created on Ubuntu is pretty useless if you use local DNS to resolve cluster services by name (e.g. http://my-backend-microservice), which is what we do to support namespaces for reverse proxies (nginx).

I manually built iptables 1.6.2 using the instructions at http://www.linuxfromscratch.org/blfs/view/8.2/postlfs/iptables.html and my problem appears to be solved. It would be great if the change could be backported into bionic, but at the minimum getting this into the next LTS then that would be great. If it makes any difference, iptables in Debian buster is at 1.8.2-4. They have a planned release in, oh look at that, two days :-)

As for test cases, in this particular instance it's sufficient that when using the --random-fully flag to set up a NAT masquerading rule that the NF_NAT_RANGE_PROTO_RANDOM_FULLY flag is set. I can't say what the regression potential is, but since it's a minor release then I'd expect it to be minimal. The diffs are at https://www.netfilter.org/projects/iptables/files/patch-iptables-1.6.1-1.6.2.bz2

Here are some details of my installation:

ubuntu@kubernetes-master:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

ubuntu@kubernetes-master:~$ uname -a
Linux kubernetes-master 4.15.0-1043-aws #45-Ubuntu SMP Mon Jun 24 14:07:03 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[VinodKumar (vinod79)]

Hi all,
I manually built iptables 1.6.2 using the instructions at http://www.linuxfromscratch.org/blfs/view/8.2/postlfs/iptables.html and my problem is if I use
#iptables -V
output is 1.6.2
and if I use
#apt-cache policy iptables
it is showing 1.6.0 only. i went through following process. can anybody suggest/correct me wr is the mistake.

1 downloaded and extracted iptables-1.6.2.tar.bz2

2 ./configure --prefix=/usr \
            --sbindir=/sbin \
            --disable-nftables \
            --enable-libipq \
            --with-xtlibdir=/lib/xtables &&
make

3 make install &&
ln -sfv ../../sbin/xtables-multi /usr/bin/iptables-xml &&

for file in ip4tc ip6tc ipq iptc xtables
do
  mv -v /usr/lib/lib${file}.so.* /lib &&
  ln -sfv ../../lib/$(readlink /usr/lib/lib${file}.so) /usr/lib/lib${file}.so
done

4 downloaded and extracted blfs-bootscripts-20180105

5 make install-iptables

Please suggest how to get the updated info in apt-cache policy iptables

[Jason (jasmas)]

Should this be higher priority?
https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02

[Oibaf (oibaf)]

I updated the title to better reflect the status.
Note that cosmic and disco, noted in the first post, are already in End of Life.