dhclient hang when tons of ip addresses exist

[Dongdong Wang]

Ubuntu
======

No LSB modules are available.
Description: Ubuntu 24.04.2 LTS
Release: 24.04

Package
=======

isc-dhcp-client:
  Installed: 4.4.3-P1-4ubuntu2
  Candidate: 4.4.3-P1-4ubuntu2
  Version table:
 *** 4.4.3-P1-4ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages
        100 /var/lib/dpkg/status

Steps to Reproduce
==================

# ip l add dev dummy type dummy
# ip l set dev dummy up
# for i in {1..50}; do for j in {1..50}; do ip a a 192.168.$i.$j/24 dev dummy; done; done
# ip -o a show dev dummy | wc -l
2500
# dhclient -v dummy

Result
======

After the operations above, we could see the dhclient hanging.
The top command showed the dhclient's memory was increasing,
seemed a memory leak happened. Also CPU usage was pretty high.

top - 02:13:48 up 12 days, 23:59, 0 user, load average: 2.99, 2.62, 2.43
Tasks: 5 total, 2 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.8 us, 1.9 sy, 0.0 ni, 96.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 772016.8 total, 614554.4 free, 111778.8 used, 50785.8 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 660238.0 avail Mem

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
   2870 root 20 0 283.4m 282.3m 2.3m R 100.0 0.0 5:05.29 dhclient
      1 root 20 0 4.5m 3.8m 3.2m S 0.0 0.0 0:00.00 bash
     10 root 20 0 4.6m 4.1m 3.4m S 0.0 0.0 0:00.59 bash
   2869 root 20 0 3.2m 1.3m 1.2m S 0.0 0.0 0:00.00 dhclient
   2882 root 20 0 8.7m 5.0m 2.9m R 0.0 0.0 0:00.01 top

[Dongdong Wang (wangdongdong0501)]

After a long time of debugging, I think I found the root cause of the hang.

The bug has been firstly introduced by the patch[1], see [2] for details.
The patch aimed to solve the multi-ip-addr-per-interface issue, but mistakely introduced another issue.

It mistakely calls getifaddrs() in the "while (next_iface(&info, &err, &ifaces))" loop.
As we reproduced above, there are 2500 ip addresses in the system, which will cause the loop runs 2500 x 2500 times, and totally calls getifaddrs() 2500 times too.

The getifaddrs() is not free, it allocates memories which need to call freeifaddrs() to free them. But the patch[1] does not do that, causing the dhclient's memory increases very fast, like a memory leak.

To fix the BUG, I propose a patch (the attachment), which reverts the patch[1] and append a tiny fix.
The fix is pretty simple and already exists in the source code of another next_iface() routine.
It just trims the "eth0:1" to "eth0" in the next_iface() routine, and returns the "eth0" to the caller, no need to loop 2500 times for each address.

After the fix, the dhclient hang issue disappears.

[1] https://launchpadlibrarian.net/66470806/dhcpd.workaround.patch
[2] https://bugs.launchpad.net/eucalyptus/+bug/717166

[Dongdong Wang (wangdongdong0501)]

Add debug patch for dhclient 4.4.3-P1.

[Dongdong Wang (wangdongdong0501)]

Add debug patch for dhclient 4.4.3-P1, just modify the version.

[Dongdong Wang (wangdongdong0501)]

Add debug patch for dhclient 4.4.1.

[Dongdong Wang (wangdongdong0501)]

Add debug patch for dhclient 4.4.1, just modify the version.

[Dongdong Wang (wangdongdong0501)]

Add debug patch for dhclient 4.3.5.

[Dongdong Wang (wangdongdong0501)]

Add debug patch for dhclient 4.3.5, just modify the version.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jian Wen (wenjianhn)]

isc-dhcp-client is no longer supported by upstream.
The preferred dhcp client is dhcpcd For 24.04.
See https://github.com/canonical/cloud-init/pull/4912

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "dhclient-fix-hang.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Dongdong Wang (wangdongdong0501)]

Hi Jian Wen (wenjianhn):

Thanks for the reply.

This bug is not introduced by upstream, but by the ubuntu community[1]. So I assume it should be fixed by ubuntu community either.

[1] https://bugs.launchpad.net/eucalyptus/+bug/717166