dhcpd does not respect ip_local_port _range or ip_local_reserved_ports

[Brian Candler]

When isc-dhcp-server starts up, in addition to listening on port 67, it binds to a random UDP port on an IPv4 socket and another on an IPv6 socket:

# netstat -naup | grep dhcp
udp 0 0 0.0.0.0:11075 0.0.0.0:* 8188/dhcpd
udp 0 0 0.0.0.0:67 0.0.0.0:* 8188/dhcpd
udp6 0 0 :::10800 :::* 8188/dhcpd
#

(I am guessing this is for making outbound DNS queries?) However, this prevented a later application of mine from working, as it wanted to bind to port 11075 for accepting incoming data.

Simply doing "service isc-dhcp-server restart" makes it choose new ports, but this problem may occur again in the future.

In the default configuration, I believe ephemeral ports should only use 32768 and above:

# cat /proc/sys/net/ipv4/ip_local_port_range
32768 60999
# cat /proc/sys/net/ipv4/ip_local_reserved_ports

#

I also tried setting a reservation, and this was not respected either.

# sysctl net.ipv4.ip_local_reserved_ports="10000-59999"
net.ipv4.ip_local_reserved_ports = 10000-59999

After restarting dhcpd:

# netstat -naup | grep dhcp
udp 0 0 0.0.0.0:50610 0.0.0.0:* 4592/dhcpd
udp 0 0 0.0.0.0:67 0.0.0.0:* 4592/dhcpd
udp6 0 0 :::28891 :::* 4592/dhcpd

I can find no way to tell isc-dhcp-server which port range to use. Setting "omapi-port" in dhcpd.conf makes it listen for *TCP* connections on the given port, and does not affect the UDP behaviour.

I don't know if this is a problem with the application (explicitly picking a local port), the resolver library (ditto), or the kernel (ignoring its own ip_local_port_range)

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: isc-dhcp-server 4.3.3-5ubuntu12.6
ProcVersionSignature: Ubuntu 4.4.0-64.85-generic 4.4.44
Uname: Linux 4.4.0-64-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
Date: Mon Mar 6 09:30:29 2017
DhServerLeases:

InstallationDate: Installed on 2017-03-04 (2 days ago)
InstallationMedia: Ubuntu-Server 16.04.2 LTS "Xenial Xerus" - Release amd64 (20170215.8)
ProcEnviron:
 SHELL=/bin/bash
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US
 LANGUAGE=en_US:
SourcePackage: isc-dhcp
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.dhcp.dhcpd.conf: 2017-03-04T09:46:07.987046

[Brian Candler (b-candler)]

strace shows that dhcpd is binding to explicit ports which it has chosen itself:

setsockopt(20, SOL_SOCKET, SO_TIMESTAMP, [1], 4) = 0
setsockopt(20, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
getsockopt(20, SOL_SOCKET, SO_RCVBUF, [212992], [4]) = 0
setsockopt(20, SOL_IP, IP_RECVTOS, [1], 4) = 0
bind(20, {sa_family=AF_INET, sin_port=htons(15101), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
socket(PF_INET6, SOCK_DGRAM, IPPROTO_UDP) = 5
fcntl(5, F_DUPFD, 20) = 21
close(5) = 0
fcntl(21, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(21, F_SETFL, O_RDWR|O_NONBLOCK) = 0
setsockopt(21, SOL_IPV6, IPV6_MTU, [1280], 4) = 0
setsockopt(21, SOL_SOCKET, SO_TIMESTAMP, [1], 4) = 0
setsockopt(21, SOL_IPV6, IPV6_RECVPKTINFO, [1], 4) = 0
setsockopt(21, SOL_IPV6, IPV6_MTU_DISCOVER, [0], 4) = 0
getsockopt(21, SOL_SOCKET, SO_RCVBUF, [212992], [4]) = 0
setsockopt(21, SOL_IPV6, IPV6_RECVTCLASS, [1], 4) = 0
setsockopt(21, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
bind(21, {sa_family=AF_INET6, sin6_port=htons(11709), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0

Aha - from man dhcpd:

When DDNS is enabled at compile time (see includes/site.h)
the server will open both a v4 and a v6 UDP socket on
random ports. These ports are opened even if DDNS is disabled
in the configuration file.

In the source code, the README says:

A fully-featured implementation of dynamic DNS updates is included in
this release. It uses libraries from BIND and, to avoid issues with
different versions, includes the necessary BIND version. The appropriate
BIND libraries will be compiled and installed in the bind subdirectory
as part of the make step.

However, debian/patches/system-bind.patch causes it to link against the system bind library; this seems to be -ldns-export from package libdns-export162

apt-get source libdns-export162 pulls in bind9. The CHANGES file says:

4109. [port] linux: support reading the local port range from
                        net.ipv4.ip_local_port_range. [RT # 39379]

However, the strace output from dhcpd shows no attempt to read ip_local_port_range.

I see the bind code includes these fallback values:

#define ISC_NET_PORTRANGELOW 1024
#define ISC_NET_PORTRANGEHIGH 65535

ISTM that a safer bound would be 49152-65535 (IANA dynamic port range), or at least the linux default of "32768 60999"

But the fundamental problem here is that for some reason, when dhcpd links into this code, it isn't making use of the detection of available Linux port range.

[Brian Candler (b-candler)]

WORKAROUND: if the other application which needs to bind to a UDP port can be configured to bind to a specific interface, then you can make isc-dhcp-server bind to a different interface in dhcpd.conf. e.g.

ddns-local-address4 127.0.0.1;
ddns-local-address6 ::1;

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Norman Henderson (norm-audrey)]

IMHO this is an important bug because it randomly interferes with other applications - lots of which use defined ports above 1024.

My recent case caused an OpenVPN instance to fail to start. More seriously it created a security risk since the port in question was of course open on the firewall for purposes of the VPN, and an outsider could have used it to fire data at dhcpd with who knows what results.

There is the same issue with isc-dhcp-client; per https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1176046 it seems the folks at ISC are unwilling to respect the defined dynamic port range, and they should be persuaded. Rather than allowing the kernel to assign a random port number like most applications, they want to do it "by self".

The solution for that bug was to split isc-dhcp-client into two versions, one compiled with and one without ddns support. That could also be done with dhcpd, however, in my opinion it's an ugly solution.

If we are going to have to just live with random ports starting from 1024, it would make a LOT more sense to alter the effect of ddns-update-style none (and ddns-updates off) so that dhcpd does NOT bind to random ports when those config parameters dictate that the random ports are never going to be used anyway.

[Rajesh Nataraja (rnataraja)]

Has this ever been fixed? This is pretty nasty, and brings down the whole Docker swarm cluster.