Embedded DNS server for user-defined network and netfilter conntrack

[Serhii Popovych]

Hello,

docker behavior for name resolution different for user-defined networks (i.e. --network=mynet):
it unconditionally configures Embedded DNS server for this.

However this have side effect on netfilter conntrack. Since Linux kernel commit
  https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=ba3fbe663635ae7b33a2d972c5d2def036258e42

conntrack hooks registered only when explicitly requested by nft ruleset or enable_hooks modparam).

Above output shows embedded DNS server redirect rules
configured per container with user-defined network:
-----------------------------------------------------

# docker run -idt --name='test1' --hostname='test1' --network='nft-isolated' --rm ubuntu:jammy bash
# docker inspect test1 | grep '"Pid":'
            "Pid": 3334,
# ln -sf /proc/3334/ns/net /var/run/netns/test1
# ip netns exec test1 iptables -L -nv -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
    0 0 DOCKER_OUTPUT all -- * * 0.0.0.0/0 127.0.0.11

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target prot opt in out source destination
    0 0 DOCKER_POSTROUTING all -- * * 0.0.0.0/0 127.0.0.11

Chain DOCKER_OUTPUT (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DNAT tcp -- * * 0.0.0.0/0 127.0.0.11 tcp dpt:53 to:127.0.0.11:34563
    0 0 DNAT udp -- * * 0.0.0.0/0 127.0.0.11 udp dpt:53 to:127.0.0.11:34392

Chain DOCKER_POSTROUTING (1 references)
 pkts bytes target prot opt in out source destination
    0 0 SNAT tcp -- * * 127.0.0.11 0.0.0.0/0 tcp spt:34563 to::53
    0 0 SNAT udp -- * * 127.0.0.11 0.0.0.0/0 udp spt:34392 to::53

-----------------------------------------------------

This ruleset effectively enables conntrack for container:

# docker exec -it test1 cat /proc/sys/net/netfilter/nf_conntrack_count
15
# ip netns exec test1 conntrack -C
15

Counters aren't zero, thus conntrack enabled, and conntrack -L shows entries when executed
with ip-netns(8) exec command.

Since connection tracking hash table isn't per network namespace and having single container
with thousands of conntrack entries can effectively affect other containers and host performance
also causing packet loss:

   # dmesg | grep 'nf_conntrack'
   nf_conntrack: table full, dropping packet
   ...

Is there a way to disable embedded DNS server for user-defined networks to avoid conntrack table overflows?

Thanks,
Serhii

[Serhii Popovych (spopovyc)]

Alternative to disabling embedded DNS server could be an option that forces it to listen on default
port 53/udp and 53/tcp so libc resolver able to contact it using 127.0.0.11 directly w/o firewall rules.

If that option is on (not default) - do not configure ruleset and thus conntrack remains disabled.

Listening on port 53 by default in container would make embedded DNS server to conflict with container
software listening on same port (e.g. BIND, dnsmasq, etc). Thus such option should be off by default.

[Serhii Popovych (spopovyc)]

Temporary workaround, just if someone curious about it, can be found at

https://github.com/serhepopovych/docker-host/blob/master/ubuntu/edge/usr/local/sbin/iptables-dockerd

It could be installed on host with
  sudo install -D -m 0755 -u root -g root ~/Downloads/iptables-dockerd '/usr/local/sbin/iptables-dockerd'
  sudo update-alternatives --install '/usr/sbin/iptables' 'iptables' '/usr/local/sbin/iptables-dockerd'

Note that containers, using user-defined network should be (re)created and tested afterwards with
   docker exec -it <name> cat /proc/sys/net/netfilter/nf_conntrack_count

It should show 0.