Ubuntu
isc-dhcp package

Bug #1926139
Comment #9

Comment 9 for bug 1926139

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2023-01-19:

Reproducer based on GDB and DHCP noise injection.

It uses 3 veth pairs (DHCP server/client/injector,
the latter two under namespaces) on a linux bridge.

LXD VM:

$ lxc launch ubuntu:focal lp1926139-focal --vm
$ lxc shell lp1926139-focal

Network Setup:

# ip link add br0 type bridge
# ip link set br0 up

# ip link add veth0 type veth peer name veth0br
# ip link set veth0 up
# ip link set veth0br up master br0

# ip netns add ns1
# ip link add veth1 netns ns1 type veth peer name veth1br
# ip -n ns1 link set veth1 up
# ip link set veth1br up master br0

# ip netns add ns2
# ip link add veth2 netns ns2 type veth peer name veth2br
# ip -n ns2 link set veth2 up
# ip link set veth2br up master br0

Network Check:

# ip link show type veth | grep veth
5: veth0br@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
6: veth0@veth0br: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
7: veth1br@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
8: veth2br@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000

# ip -n ns1 link show type veth | grep veth
2: veth1@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

# ip -n ns2 link show type veth | grep veth
2: veth2@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

DHCP Server Setup:

# apt install -y isc-dhcp-server

# ip addr add 192.168.42.1/24 dev veth0

# echo 'INTERFACESv4="veth0"' >>/etc/default/isc-dhcp-server

# cat <<EOF >>/etc/dhcp/dhcpd.conf
subnet 192.168.42.0 netmask 255.255.255.0 {
range 192.168.42.100 192.168.42.200;
}
EOF

# systemctl restart isc-dhcp-server.service
# systemctl status isc-dhcp-server.service | grep Active:
Active: active (running) since Thu 2023-01-19 02:06:18 UTC; 19s ago

# ss -nlp | grep 0.0.0.0:67
udp UNCONN 0 0 0.0.0.0:67 0.0.0.0:* users:(("dhcpd",pid=3279,fd=9))

DHCP Server Check:

# ip netns exec ns1 \
dhclient -v veth1
...
DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 3 (xid=0xd147ab17)
DHCPOFFER of 192.168.42.100 from 192.168.42.1
DHCPREQUEST for 192.168.42.100 on veth1 to 255.255.255.255 port 67 (xid=0x17ab47d1)
DHCPACK of 192.168.42.100 from 192.168.42.1 (xid=0xd147ab17)
bound to 192.168.42.100 -- renewal in 245 seconds.

# ip netns exec ns1 \
dhclient -v veth1 -r
...
DHCPRELEASE of 192.168.42.100 on veth1 to 192.168.42.1 port 67 (xid=0x1cd4aacf)

DHCP Noise Setup:

# ip -n ns2 addr add 192.168.42.2/24 dev veth2

# ip netns exec ns2 \
/bin/sh -c 'while sleep 0.1; do echo; done | nc -u -v -b -s 192.168.42.2 -p 67 255.255.255.255 68' &
Connection to 255.255.255.255 68 port [udp/bootpc] succeeded!

i.e., every 0.1 seconds, broadcast a message as DHCP (port 67) to DHCP client receive (port 68).

DHCP Noise Check:

# tcpdump -i veth0 -n 'udp and host 255.255.255.255' -c 10
...
02:13:26.993233 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
02:13:27.098317 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
02:13:27.205879 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
02:13:27.314234 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
02:13:27.424486 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
02:13:27.532431 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
02:13:27.639614 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
02:13:27.747633 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
02:13:27.864037 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
02:13:27.977402 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
...

GDB Reproducer (original package):
==============

# apt install -y gdb

Capture DHCP Server's UDP packets for reference:

# tcpdump -i veth0 -n 'udp and host 192.168.42.1' -w veth0-udp-192-168-42-1.pcap & pid=$!

Debug symbols:

# wget https://launchpad.net/ubuntu/+archive/primary/+files/isc-dhcp-client-dbgsym_4.4.1-2.1ubuntu5.20.04.4_amd64.ddeb
# apt install -y ./isc-dhcp-client-dbgsym_4.4.1-2.1ubuntu5.20.04.4_amd64.ddeb

Source code line numbers (for breakpoint):

198 isc_result_t omapi_register_io_object (omapi_object_t *h,
...
260 status = isc_socket_fdwatchcreate(dhcp_gbl_ctx.socketmgr,
...
278 /* Find the last I/O state, if there are any. */
279 for (p = omapi_io_states.next;

Reproduce the issue with a delay introduced via breakpoint on line 279:

# ip netns exec ns1 \
gdb -ex 'set target-async on' -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' -q dhclient

(gdb) break omapip/dispatch.c:279
(gdb) commands
shell sleep 0.2
continue
end
(gdb) run -v -d veth1
...
Listening on LPF/veth1/ea:7a:1d:d1:53:59
Sending on LPF/veth1/ea:7a:1d:d1:53:59

Thread 1 "dhclient" hit Breakpoint 1, omapi_register_io_object ...
Sending on Socket/fallback

Thread 1 "dhclient" hit Breakpoint 1, omapi_register_io_object ...
279 in dispatch.c
DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 3 (xid=0xe3b19607)
DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 8 (xid=0xe3b19607)
DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 13 (xid=0xe3b19607)
DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 18 (xid=0xe3b19607)
^C
...
(gdb) quit

The tcpdump confirms the DHCP Server _sent_ DHCP Offer packets,
not handled by the DHCP Client.

# kill $pid
4 packets captured
4 packets received by filter
0 packets dropped by kernel
[2]+ Done tcpdump -i veth0 -n 'udp and host 192.168.42.1' -w veth0-udp-192-168-42-1.pcap

# tcpdump -i veth0 -n 'udp and host 192.168.42.1' -r veth0-udp-192-168-42-1.pcap -v
...
     192.168.42.1.67 > 192.168.42.100.68: BOOTP/DHCP, Reply, length 300, xid 0xe3b19607, Flags [none]
    Your-IP 192.168.42.100
...
      DHCP-Message Option 53, length 1: Offer
...
     192.168.42.1.67 > 192.168.42.100.68: BOOTP/DHCP, Reply, length 300, xid 0xe3b19607, secs 4, Flags [none]
    Your-IP 192.168.42.100
...
      DHCP-Message Option 53, length 1: Offer
...
     192.168.42.1.67 > 192.168.42.100.68: BOOTP/DHCP, Reply, length 300, xid 0xe3b19607, secs 12, Flags [none]
    Your-IP 192.168.42.100
...
      DHCP-Message Option 53, length 1: Offer
...
     192.168.42.1.67 > 192.168.42.100.68: BOOTP/DHCP, Reply, length 300, xid 0xe3b19607, secs 25, Flags [none]
    Your-IP 192.168.42.100
...
      DHCP-Message Option 53, length 1: Offer
...

GDB Reproducer (patched package):
==============

Client & Debug symbols:

# sudo apt install \
./isc-dhcp-client_4.4.1-2.1ubuntu5.20.04.4+lp1926139.1_amd64.deb \
./isc-dhcp-client-dbgsym_4.4.1-2.1ubuntu5.20.04.4+lp1926139.1_amd64.ddeb

Source code line numbers (for breakpoint):

  253 isc_result_t omapi_register_io_object (omapi_object_t *h,
...
  324 status = isc_socket_fdwatchcreate(dhcp_gbl_ctx.socketmgr,
...
  343 /* Find the last I/O state, if there are any. */
  344 for (p = omapi_io_states.next;

Attempt to reproduce the issue again, the same way,
with a delay introduced via breakpoint on line 344:

# ip netns exec ns1 \
gdb -ex 'set target-async on' -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' -q dhclient

(gdb) break omapip/dispatch.c:344
(gdb) commands
shell sleep 0.2
continue
end
(gdb) run -v -d veth1
...
Listening on LPF/veth1/ea:7a:1d:d1:53:59
Sending on LPF/veth1/ea:7a:1d:d1:53:59

Thread 1 "dhclient" hit Breakpoint 1, omapi_register_io_object ...
Waiting for object registration to finish...
(This can be disabled with: <VAR>/<cmdline>)
Sending on Socket/fallback

Object registration finished.
Thread 1 "dhclient" hit Breakpoint 1, omapi_register_io_object ...
344 in dispatch.c
DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 3 (xid=0x13d35e3b)
DHCPOFFER of 192.168.42.100 from 192.168.42.1
DHCPREQUEST for 192.168.42.100 on veth1 to 255.255.255.255 port 67 (xid=0x3b5ed313)
DHCPACK of 192.168.42.100 from 192.168.42.1 (xid=0x13d35e3b)
[Detaching after fork from child process 15283]
bound to 192.168.42.100 -- renewal in 252 seconds.
^C
...
(gdb) quit

The issue did not happen!

The DHCP client successfully acquired a DHCP address (above).
It can even be released later, outside of GDB (below).

# ip netns exec ns1 \
dhclient -v veth1 -r
...
DHCPRELEASE of 192.168.42.100 on veth1 to 192.168.42.1 port 67 (xid=0x70f6c778)

Reproducer based on GDB and DHCP noise injection.

It uses 3 veth pairs (DHCP server/client/injector,
the latter two under namespaces) on a linux bridge.

LXD VM:

$ lxc launch ubuntu:focal lp1926139-focal --vm
	$ lxc shell lp1926139-focal

Network Setup:

# ip link add br0 type bridge
	# ip link set br0 up

# ip link add veth0 type veth peer name veth0br
	# ip link set veth0 up
	# ip link set veth0br up master br0

# ip netns add ns1
	# ip link add veth1 netns ns1 type veth peer name veth1br
	# ip -n ns1 link set veth1 up
	# ip link set veth1br up master br0

# ip netns add ns2
	# ip link add veth2 netns ns2 type veth peer name veth2br
	# ip -n ns2 link set veth2 up
	# ip link set veth2br up master br0

Network Check:

# ip link show type veth | grep veth
	5: veth0br@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
	6: veth0@veth0br: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
	7: veth1br@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
	8: veth2br@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000

# ip -n ns1 link show type veth | grep veth
	2: veth1@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

# ip -n ns2 link show type veth | grep veth
	2: veth2@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000

DHCP Server Setup:

# apt install -y isc-dhcp-server

# ip addr add 192.168.42.1/24 dev veth0

# echo 'INTERFACESv4="veth0"' >>/etc/default/isc-dhcp-server

# cat <<EOF >>/etc/dhcp/dhcpd.conf
	subnet 192.168.42.0 netmask 255.255.255.0 {
	  range 192.168.42.100 192.168.42.200;
	}
	EOF

# systemctl restart isc-dhcp-server.service
	# systemctl status isc-dhcp-server.service | grep Active:
	     Active: active (running) since Thu 2023-01-19 02:06:18 UTC; 19s ago

# ss -nlp | grep 0.0.0.0:67
	udp     UNCONN   0        0                                             0.0.0.0:67                                                 0.0.0.0:*                     users:(("dhcpd",pid=3279,fd=9))

DHCP Server Check:

# ip netns exec ns1 \
	  dhclient -v veth1
	...
	DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 3 (xid=0xd147ab17)
	DHCPOFFER of 192.168.42.100 from 192.168.42.1
	DHCPREQUEST for 192.168.42.100 on veth1 to 255.255.255.255 port 67 (xid=0x17ab47d1)
	DHCPACK of 192.168.42.100 from 192.168.42.1 (xid=0xd147ab17)
	bound to 192.168.42.100 -- renewal in 245 seconds.

# ip netns exec ns1 \
	  dhclient -v veth1 -r
	...
	DHCPRELEASE of 192.168.42.100 on veth1 to 192.168.42.1 port 67 (xid=0x1cd4aacf)

DHCP Noise Setup:

# ip -n ns2 addr add 192.168.42.2/24 dev veth2

# ip netns exec ns2 \
	  /bin/sh -c 'while sleep 0.1; do echo; done | nc -u -v -b -s 192.168.42.2 -p 67 255.255.255.255 68' &
	Connection to 255.255.255.255 68 port [udp/bootpc] succeeded!

i.e., every 0.1 seconds, broadcast a message as DHCP (port 67) to DHCP client receive (port 68).

DHCP Noise Check:

# tcpdump -i veth0 -n 'udp and host 255.255.255.255' -c 10
	...
	02:13:26.993233 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
	02:13:27.098317 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
	02:13:27.205879 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
	02:13:27.314234 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
	02:13:27.424486 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
	02:13:27.532431 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
	02:13:27.639614 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
	02:13:27.747633 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
	02:13:27.864037 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
	02:13:27.977402 IP 192.168.42.2.67 > 255.255.255.255.68: BOOTP/DHCP, unknown (0x0a) [|bootp]
	...

GDB Reproducer (original package):
==============

# apt install -y gdb
	
Capture DHCP Server's UDP packets for reference:
	
	# tcpdump -i veth0 -n 'udp and host 192.168.42.1' -w veth0-udp-192-168-42-1.pcap & pid=$!

Debug symbols:
	
	# wget https://launchpad.net/ubuntu/+archive/primary/+files/isc-dhcp-client-dbgsym_4.4.1-2.1ubuntu5.20.04.4_amd64.ddeb
	# apt install -y ./isc-dhcp-client-dbgsym_4.4.1-2.1ubuntu5.20.04.4_amd64.ddeb

Source code line numbers (for breakpoint):
	
	198 isc_result_t omapi_register_io_object (omapi_object_t *h,
	...
	260                 status = isc_socket_fdwatchcreate(dhcp_gbl_ctx.socketmgr,
	...
	278         /* Find the last I/O state, if there are any. */
	279         for (p = omapi_io_states.next;

Reproduce the issue with a delay introduced via breakpoint on line 279:

# ip netns exec ns1 \
	  gdb -ex 'set target-async on' -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' -q dhclient

(gdb) break omapip/dispatch.c:279
	(gdb) commands
	shell sleep 0.2
	continue
	end
	(gdb) run -v -d veth1
	...
	Listening on LPF/veth1/ea:7a:1d:d1:53:59
	Sending on   LPF/veth1/ea:7a:1d:d1:53:59

Thread 1 "dhclient" hit Breakpoint 1, omapi_register_io_object ...
	Sending on   Socket/fallback

Thread 1 "dhclient" hit Breakpoint 1, omapi_register_io_object ...
	279	in dispatch.c
	DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 3 (xid=0xe3b19607)
	DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 8 (xid=0xe3b19607)
	DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 13 (xid=0xe3b19607)
	DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 18 (xid=0xe3b19607)
	^C
	...
	(gdb) quit

The tcpdump confirms the DHCP Server _sent_ DHCP Offer packets,
not handled by the DHCP Client.

# kill $pid
	4 packets captured
	4 packets received by filter
	0 packets dropped by kernel
	[2]+  Done                    tcpdump -i veth0 -n 'udp and host 192.168.42.1' -w veth0-udp-192-168-42-1.pcap

# tcpdump -i veth0 -n 'udp and host 192.168.42.1' -r veth0-udp-192-168-42-1.pcap -v
	...
	    192.168.42.1.67 > 192.168.42.100.68: BOOTP/DHCP, Reply, length 300, xid 0xe3b19607, Flags [none]
		  Your-IP 192.168.42.100
	...
		    DHCP-Message Option 53, length 1: Offer
	...
	    192.168.42.1.67 > 192.168.42.100.68: BOOTP/DHCP, Reply, length 300, xid 0xe3b19607, secs 4, Flags [none]
		  Your-IP 192.168.42.100
	...
		    DHCP-Message Option 53, length 1: Offer
	...
	    192.168.42.1.67 > 192.168.42.100.68: BOOTP/DHCP, Reply, length 300, xid 0xe3b19607, secs 12, Flags [none]
		  Your-IP 192.168.42.100
	...
		    DHCP-Message Option 53, length 1: Offer
	...
	    192.168.42.1.67 > 192.168.42.100.68: BOOTP/DHCP, Reply, length 300, xid 0xe3b19607, secs 25, Flags [none]
		  Your-IP 192.168.42.100
	...
		    DHCP-Message Option 53, length 1: Offer
	...

GDB Reproducer (patched package):
==============

Client & Debug symbols:

# wget \
	  https://launchpad.net/~mfo/+archive/ubuntu/lp1926139/+files/isc-dhcp-client_4.4.1-2.1ubuntu5.20.04.4+lp1926139.1_amd64.deb \
	  https://launchpad.net/~mfo/+archive/ubuntu/lp1926139/+files/isc-dhcp-client-dbgsym_4.4.1-2.1ubuntu5.20.04.4+lp1926139.1_amd64.ddeb
	  
	# sudo apt install \
	  ./isc-dhcp-client_4.4.1-2.1ubuntu5.20.04.4+lp1926139.1_amd64.deb \
	  ./isc-dhcp-client-dbgsym_4.4.1-2.1ubuntu5.20.04.4+lp1926139.1_amd64.ddeb

Source code line numbers (for breakpoint):

253 isc_result_t omapi_register_io_object (omapi_object_t *h,
	...
	 324                 status = isc_socket_fdwatchcreate(dhcp_gbl_ctx.socketmgr,
	...
	 343         /* Find the last I/O state, if there are any. */
	 344         for (p = omapi_io_states.next;

Attempt to reproduce the issue again, the same way,
with a delay introduced via breakpoint on line 344:

# ip netns exec ns1 \
	  gdb -ex 'set target-async on' -ex 'set non-stop on' -ex 'set pagination off' -ex 'set confirm off' -q dhclient

(gdb) break omapip/dispatch.c:344
	(gdb) commands
	shell sleep 0.2
	continue
	end
	(gdb) run -v -d veth1
	...
	Listening on LPF/veth1/ea:7a:1d:d1:53:59
	Sending on   LPF/veth1/ea:7a:1d:d1:53:59

Thread 1 "dhclient" hit Breakpoint 1, omapi_register_io_object ...
	Waiting for object registration to finish...
	(This can be disabled with: <VAR>/<cmdline>)
	Sending on   Socket/fallback

Object registration finished.
	Thread 1 "dhclient" hit Breakpoint 1, omapi_register_io_object ...
	344	in dispatch.c
	DHCPDISCOVER on veth1 to 255.255.255.255 port 67 interval 3 (xid=0x13d35e3b)
	DHCPOFFER of 192.168.42.100 from 192.168.42.1
	DHCPREQUEST for 192.168.42.100 on veth1 to 255.255.255.255 port 67 (xid=0x3b5ed313)
	DHCPACK of 192.168.42.100 from 192.168.42.1 (xid=0x13d35e3b)
	[Detaching after fork from child process 15283]
	bound to 192.168.42.100 -- renewal in 252 seconds.
	^C
	...
	(gdb) quit

The issue did not happen!

The DHCP client successfully acquired a DHCP address (above).
It can even be released later, outside of GDB (below).

# ip netns exec ns1 \
	  dhclient -v veth1 -r
	...
	DHCPRELEASE of 192.168.42.100 on veth1 to 192.168.42.1 port 67 (xid=0x70f6c778)

Ubuntuisc-dhcp package

Comment 9 for bug 1926139

Ubuntu
isc-dhcp package