DHCPv6 server crashes regularly (bionic)

[Marian Rainer-Harbach]

The isc-dhcp-server crashes regularly on bionic, sometimes directly after boot, sometimes later.
The version installed is 4.3.5-3ubuntu7.

journalctl shows:
Jul 14 09:35:11 <hostname> dhcpd[1543]: Solicit message from fe80::18eb:dfc7:17e5:c8d7 port 546, transaction ID 0x7E8EC00
Jul 14 09:35:11 <hostname> dhcpd[1543]: Advertise NA: address <subnet>::1998 to client with duid 00:01:00:01:21:9f:3a:02:d4:a3:3d:bf:17:e9 iaid = 0 valid for 8
Jul 14 09:35:11 <hostname> dhcpd[1543]: Sending Advertise to fe80::18eb:dfc7:17e5:c8d7 port 546
Jul 14 09:35:12 <hostname> dhcpd[1543]: Request message from fe80::18eb:dfc7:17e5:c8d7 port 546, transaction ID 0x65FADB00
Jul 14 09:35:12 <hostname> dhcpd[1543]: Reply NA: address <subnet>::1998 to client with duid 00:01:00:01:21:9f:3a:02:d4:a3:3d:bf:17:e9 iaid = 0 valid for 86400
Jul 14 09:35:12 <hostname> dhcpd[1543]: Sending Reply to fe80::18eb:dfc7:17e5:c8d7 port 546
Jul 14 09:35:53 <hostname> dhcpd[1543]: Confirm message from fe80::725a:b6ff:fea2:6120 port 546, transaction ID 0x5105F400
Jul 14 09:35:53 <hostname> dhcpd[1543]: Sending Reply to fe80::725a:b6ff:fea2:6120 port 546
Jul 14 09:35:53 <hostname> dhcpd[1543]: Rebind message from fe80::725a:b6ff:fea2:6120 port 546, transaction ID 0x1FEA7E00
Jul 14 09:35:53 <hostname> dhcpd[1543]: Reply NA: address <subnet>::1992 to client with duid 00:04:c2:47:10:e8:8b:dc:d4:a1:0a:1d:21:f2:be:20:e8:a0 iaid = -1230
Jul 14 09:35:53 <hostname> sh[1543]: ../../../lib/isc/heap.c:251: REQUIRE(idx >= 1 && idx <= heap->last) failed, back trace
Jul 14 09:35:53 <hostname> sh[1543]: #0 0x7efc458a6417 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #1 0x7efc458a636a in ??
Jul 14 09:35:53 <hostname> sh[1543]: #2 0x7efc458ad4ea in ??
Jul 14 09:35:53 <hostname> sh[1543]: #3 0x55d9ee65d571 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #4 0x55d9ee658701 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #5 0x55d9ee65ab05 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #6 0x55d9ee65bff3 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #7 0x55d9ee65cafc in ??
Jul 14 09:35:53 <hostname> sh[1543]: #8 0x55d9ee678402 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #9 0x55d9ee667463 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #10 0x55d9ee696476 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #11 0x7efc458dd73b in ??
Jul 14 09:35:53 <hostname> sh[1543]: #12 0x7efc458ccf9e in ??
Jul 14 09:35:53 <hostname> sh[1543]: #13 0x7efc458d1e60 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #14 0x7efc458d2325 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #15 0x55d9ee6696b0 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #16 0x55d9ee61d519 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #17 0x7efc454c6b97 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #18 0x55d9ee61de0a in ??
Jul 14 09:35:54 <hostname> systemd[1]: isc-dhcp-server6.service: Main process exited, code=dumped, status=6/ABRT
Jul 14 09:35:54 <hostname> systemd[1]: isc-dhcp-server6.service: Failed with result 'core-dump'.

The bug was reported to Debian independently, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=896122.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Radek Zajic (radek-zajic)]

The bug is not present in 16.10 (yakkety, isc-dhcp-server=4.3.3-5ubuntu15.2). It is present in 17.10, 18.04 (isc-dhcp-server=4.3.5-3ubuntu7) and 18.10 (isc-dhcp-server=4.3.5-3ubuntu9). Not sure about 17.04.

[Steffen Sledz (sledz)]

Problem occurs on all our systems after dist upgrade to Ubuntu 18.04.1 LTS (isc-dhcp-server 4.3.5-3ubuntu7). :(

[Radek Zajic (radek-zajic)]

Apparently the issue is caused by this commit https://gitlab.isc.org/wpk/bind9/commit/65a483106e45704e19781bfe4f4634db4f77562e which fixes a bug in the ISC heap library.

There seems to be a double-deletion of the released addresses in the dhcpv6 code.

This quick-and-dirty patch helps (on ubuntu 18.04.1).

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "quick-and-dirty-patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Steffen Sledz (sledz)]

Is there now a forecast when a fix will be available in Ubuntu?

[Ivo Steinmann (isteinmann)]

Still not fixed :-(

[Marian Rainer-Harbach (marianrh)]

To those waiting for a fix: I'm using a workaround by setting systemd to restart the service when it crashes. This works well for me:

$ cat /etc/systemd/system/isc-dhcp-server6.service.d/override.conf
[Service]
Restart=on-abort
RestartSec=1

Run systemctl daemon-reload after creating/changing the file.

[pdf (pdffs)]

Yes, the problem is certainly exacerbated by the service not having a Restart parameter, but this needs some real attention. The quick-and-dirty-patch does reduce the crashes, but results in a different crash (albeit less frequently).

[Radek Zajic (radek-zajic)]

@pdf: would you please mind posting the different crash report? It can be related to the dirty patch, or there can be another issue. Thanks.

ad restart parameter: that does not help in all cases, for example my UEFI test machines booting via UEFI PXE+iPXE just hang when they do not receive the proper DHCPv6 message in time (e.g. they receive advertise, but not reply).

[pdf (pdffs)]

@radek-zajic I'm not running the patch currently, but I only saw that crash while running with the patch. If I rebuild using the patch again I'll grab a stack trace (I've just replaced the machine running DHCP).

[Andreas Hasenack (ahasenack)]

Has anybody filed an upstream bug about this at https://www.isc.org/community/report-bug/ ?

[Radek Zajic (radek-zajic)]

#12: I have, just now. RT #48804.

[Andreas Hasenack (ahasenack)]

Thanks, I linked it to this bug here. Soon someone at isc will unblock it so it's public.

[Sebastien Bacher (seb128)]

There is no debug symbol here but if it's the same issue than the redhat report then the error tracker bucket for the issue is
https://errors.ubuntu.com/problem/5c6b31248447aaecc5e91304bfd49f40be9eee7b

[Radek Zajic (radek-zajic)]

Thank you, Sebastien. Unfortunately that link is behind a restrictive access control, so if there indeed is a ISC DHCP bug tracker ID there, no one has told us here. (ISC can for sure close my ticket as duplicate, if they are already working on this.)

[Thomas Markwalder (tmark)]

Hello:

This was corrected in ISC DHCP 4.4.1 under:

 https://bugs.isc.org/Ticket/Display.html?id=46719#

4.3.* is EOL at this point.

[Steffen Sledz (sledz)]

It is really a shame that the fix of this bug in this basic network infrastructure package has not made it into the distributions (especially Ubuntu) after months. :(

[Alex Murray (alexmurray)]

This looks like a possible use-after-free so likely has a security impact (at a minimum it is a denial of service due to the crash, especially if it can be triggered remotely) - I've reported it to ISC as such who will hopefully assign a CVE and then we can fix it as a security update. For future reference, the RT #48804 contains a patch that should likely be fine for Bionic https://bugs.isc.org/Public/Ticket/Attachment/534989/331007/46719.v4_3.diff

[Andreas Hasenack (ahasenack)]

Thanks Alex, let's see if the patch changes after the issue is looked at from a security perspective.

[Alex Murray (alexmurray)]

This has been assigned CVE-2019-6470

[Marc Deslauriers (mdeslaur)]

Packages for this issue are now available in the security team test ppa here:

https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+packages

Could someone please try them out and make sure they resolve the issue?

Thanks!

[pdf (pdffs)]

I have the packages in for testing, though repro may take some time to occur organically. I also think it might be prudent to add a Restart=on-failure directive to the systemd units regardless of whether the fix for this issue is effective, so that future bugs don't take down users' networks.

[Morten Gade Sørensen (mgs)]

I have tested the -server package on Bionic and I haven't had any crashes yet. Typically in my environment it would have crashed after one hour or two hours the latest.

[Marian Rainer-Harbach (marianrh)]

Thanks for your work, Alex, Andreas and Marc! I can confirm that isc-dhcp-server 4.3.5-3ubuntu7.1 has been running for two days while the previous version crashed every few hours in my environment.

[Marc Deslauriers (mdeslaur)]

Great, thanks for testing, I'll release these early next week!

[Launchpad Janitor (janitor)]

This bug was fixed in the package isc-dhcp - 4.3.5-3ubuntu7.1

---------------
isc-dhcp (4.3.5-3ubuntu7.1) bionic-security; urgency=medium

  * SECURITY UPDATE: DoS via change in bind behaviour (LP: #1781699)
    - debian/patches/CVE-2019-6470.patch: use 0 instead of -1 to indicate
      empty heap index in includes/dhcpd.h, server/mdb6.c,
      server/tests/mdb6_unittest.c.
    - CVE-2019-6470

 -- Marc Deslauriers <email address hidden> Mon, 06 May 2019 09:00:01 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package isc-dhcp - 4.3.5-3ubuntu9.1

---------------
isc-dhcp (4.3.5-3ubuntu9.1) cosmic-security; urgency=medium

  * SECURITY UPDATE: DoS via change in bind behaviour (LP: #1781699)
    - debian/patches/CVE-2019-6470.patch: use 0 instead of -1 to indicate
      empty heap index in includes/dhcpd.h, server/mdb6.c,
      server/tests/mdb6_unittest.c.
    - CVE-2019-6470

 -- Marc Deslauriers <email address hidden> Mon, 06 May 2019 08:57:40 -0400