DHCPv6 server crashes regularly (bionic)

Bug #1781699 reported by Marian Rainer-Harbach on 2018-07-14
62
This bug affects 10 people
Affects Status Importance Assigned to Milestone
DHCP
Unknown
Unknown
isc-dhcp (Debian)
Fix Released
Unknown
isc-dhcp (Fedora)
Unknown
Unknown
isc-dhcp (Ubuntu)
Status tracked in Eoan
Bionic
Undecided
Marc Deslauriers
Cosmic
Undecided
Marc Deslauriers
Disco
Undecided
Marc Deslauriers
Eoan
High
Marc Deslauriers

Bug Description

The isc-dhcp-server crashes regularly on bionic, sometimes directly after boot, sometimes later.
The version installed is 4.3.5-3ubuntu7.

journalctl shows:
Jul 14 09:35:11 <hostname> dhcpd[1543]: Solicit message from fe80::18eb:dfc7:17e5:c8d7 port 546, transaction ID 0x7E8EC00
Jul 14 09:35:11 <hostname> dhcpd[1543]: Advertise NA: address <subnet>::1998 to client with duid 00:01:00:01:21:9f:3a:02:d4:a3:3d:bf:17:e9 iaid = 0 valid for 8
Jul 14 09:35:11 <hostname> dhcpd[1543]: Sending Advertise to fe80::18eb:dfc7:17e5:c8d7 port 546
Jul 14 09:35:12 <hostname> dhcpd[1543]: Request message from fe80::18eb:dfc7:17e5:c8d7 port 546, transaction ID 0x65FADB00
Jul 14 09:35:12 <hostname> dhcpd[1543]: Reply NA: address <subnet>::1998 to client with duid 00:01:00:01:21:9f:3a:02:d4:a3:3d:bf:17:e9 iaid = 0 valid for 86400
Jul 14 09:35:12 <hostname> dhcpd[1543]: Sending Reply to fe80::18eb:dfc7:17e5:c8d7 port 546
Jul 14 09:35:53 <hostname> dhcpd[1543]: Confirm message from fe80::725a:b6ff:fea2:6120 port 546, transaction ID 0x5105F400
Jul 14 09:35:53 <hostname> dhcpd[1543]: Sending Reply to fe80::725a:b6ff:fea2:6120 port 546
Jul 14 09:35:53 <hostname> dhcpd[1543]: Rebind message from fe80::725a:b6ff:fea2:6120 port 546, transaction ID 0x1FEA7E00
Jul 14 09:35:53 <hostname> dhcpd[1543]: Reply NA: address <subnet>::1992 to client with duid 00:04:c2:47:10:e8:8b:dc:d4:a1:0a:1d:21:f2:be:20:e8:a0 iaid = -1230
Jul 14 09:35:53 <hostname> sh[1543]: ../../../lib/isc/heap.c:251: REQUIRE(idx >= 1 && idx <= heap->last) failed, back trace
Jul 14 09:35:53 <hostname> sh[1543]: #0 0x7efc458a6417 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #1 0x7efc458a636a in ??
Jul 14 09:35:53 <hostname> sh[1543]: #2 0x7efc458ad4ea in ??
Jul 14 09:35:53 <hostname> sh[1543]: #3 0x55d9ee65d571 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #4 0x55d9ee658701 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #5 0x55d9ee65ab05 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #6 0x55d9ee65bff3 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #7 0x55d9ee65cafc in ??
Jul 14 09:35:53 <hostname> sh[1543]: #8 0x55d9ee678402 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #9 0x55d9ee667463 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #10 0x55d9ee696476 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #11 0x7efc458dd73b in ??
Jul 14 09:35:53 <hostname> sh[1543]: #12 0x7efc458ccf9e in ??
Jul 14 09:35:53 <hostname> sh[1543]: #13 0x7efc458d1e60 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #14 0x7efc458d2325 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #15 0x55d9ee6696b0 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #16 0x55d9ee61d519 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #17 0x7efc454c6b97 in ??
Jul 14 09:35:53 <hostname> sh[1543]: #18 0x55d9ee61de0a in ??
Jul 14 09:35:54 <hostname> systemd[1]: isc-dhcp-server6.service: Main process exited, code=dumped, status=6/ABRT
Jul 14 09:35:54 <hostname> systemd[1]: isc-dhcp-server6.service: Failed with result 'core-dump'.

The bug was reported to Debian independently, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=896122.

CVE References

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in isc-dhcp (Ubuntu):
status: New → Confirmed
Radek Zajic (radek-zajic) wrote :

The bug is not present in 16.10 (yakkety, isc-dhcp-server=4.3.3-5ubuntu15.2). It is present in 17.10, 18.04 (isc-dhcp-server=4.3.5-3ubuntu7) and 18.10 (isc-dhcp-server=4.3.5-3ubuntu9). Not sure about 17.04.

Changed in isc-dhcp (Debian):
status: Unknown → New
Steffen Sledz (sledz) wrote :

Problem occurs on all our systems after dist upgrade to Ubuntu 18.04.1 LTS (isc-dhcp-server 4.3.5-3ubuntu7). :(

Radek Zajic (radek-zajic) wrote :

Apparently the issue is caused by this commit https://gitlab.isc.org/wpk/bind9/commit/65a483106e45704e19781bfe4f4634db4f77562e which fixes a bug in the ISC heap library.

There seems to be a double-deletion of the released addresses in the dhcpv6 code.

This quick-and-dirty patch helps (on ubuntu 18.04.1).

The attachment "quick-and-dirty-patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Steffen Sledz (sledz) wrote :

Is there now a forecast when a fix will be available in Ubuntu?

Ivo Steinmann (isteinmann) wrote :

Still not fixed :-(

To those waiting for a fix: I'm using a workaround by setting systemd to restart the service when it crashes. This works well for me:

$ cat /etc/systemd/system/isc-dhcp-server6.service.d/override.conf
[Service]
Restart=on-abort
RestartSec=1

Run systemctl daemon-reload after creating/changing the file.

pdf (pdffs) wrote :

Yes, the problem is certainly exacerbated by the service not having a Restart parameter, but this needs some real attention. The quick-and-dirty-patch does reduce the crashes, but results in a different crash (albeit less frequently).

Radek Zajic (radek-zajic) wrote :

@pdf: would you please mind posting the different crash report? It can be related to the dirty patch, or there can be another issue. Thanks.

ad restart parameter: that does not help in all cases, for example my UEFI test machines booting via UEFI PXE+iPXE just hang when they do not receive the proper DHCPv6 message in time (e.g. they receive advertise, but not reply).

pdf (pdffs) wrote :

@radek-zajic I'm not running the patch currently, but I only saw that crash while running with the patch. If I rebuild using the patch again I'll grab a stack trace (I've just replaced the machine running DHCP).

Andreas Hasenack (ahasenack) wrote :

Has anybody filed an upstream bug about this at https://www.isc.org/community/report-bug/ ?

Radek Zajic (radek-zajic) wrote :

#12: I have, just now. RT #48804.

Andreas Hasenack (ahasenack) wrote :

Thanks, I linked it to this bug here. Soon someone at isc will unblock it so it's public.

Sebastien Bacher (seb128) wrote :

There is no debug symbol here but if it's the same issue than the redhat report then the error tracker bucket for the issue is
https://errors.ubuntu.com/problem/5c6b31248447aaecc5e91304bfd49f40be9eee7b

Changed in isc-dhcp (Ubuntu):
importance: Undecided → High
status: Confirmed → Triaged
Radek Zajic (radek-zajic) wrote :

Thank you, Sebastien. Unfortunately that link is behind a restrictive access control, so if there indeed is a ISC DHCP bug tracker ID there, no one has told us here. (ISC can for sure close my ticket as duplicate, if they are already working on this.)

Thomas Markwalder (tmark) wrote :

Hello:

This was corrected in ISC DHCP 4.4.1 under:

 https://bugs.isc.org/Ticket/Display.html?id=46719#

4.3.* is EOL at this point.

Steffen Sledz (sledz) wrote :

It is really a shame that the fix of this bug in this basic network infrastructure package has not made it into the distributions (especially Ubuntu) after months. :(

Alex Murray (alexmurray) wrote :

This looks like a possible use-after-free so likely has a security impact (at a minimum it is a denial of service due to the crash, especially if it can be triggered remotely) - I've reported it to ISC as such who will hopefully assign a CVE and then we can fix it as a security update. For future reference, the RT #48804 contains a patch that should likely be fine for Bionic https://bugs.isc.org/Public/Ticket/Attachment/534989/331007/46719.v4_3.diff

Andreas Hasenack (ahasenack) wrote :

Thanks Alex, let's see if the patch changes after the issue is looked at from a security perspective.

Alex Murray (alexmurray) wrote :

This has been assigned CVE-2019-6470

Changed in isc-dhcp (Ubuntu Bionic):
status: New → In Progress
assignee: nobody → Marc Deslauriers (mdeslaur)
Changed in isc-dhcp (Ubuntu Cosmic):
status: New → In Progress
assignee: nobody → Marc Deslauriers (mdeslaur)
Changed in isc-dhcp (Ubuntu Disco):
status: New → In Progress
assignee: nobody → Marc Deslauriers (mdeslaur)
Changed in isc-dhcp (Ubuntu Eoan):
status: Triaged → In Progress
assignee: nobody → Marc Deslauriers (mdeslaur)
Changed in isc-dhcp (Ubuntu Disco):
status: In Progress → Fix Released
Changed in isc-dhcp (Ubuntu Eoan):
status: In Progress → Fix Released
Marc Deslauriers (mdeslaur) wrote :

Packages for this issue are now available in the security team test ppa here:

https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+packages

Could someone please try them out and make sure they resolve the issue?

Thanks!

pdf (pdffs) wrote :

I have the packages in for testing, though repro may take some time to occur organically. I also think it might be prudent to add a Restart=on-failure directive to the systemd units regardless of whether the fix for this issue is effective, so that future bugs don't take down users' networks.

Morten Gade Sørensen (mgs) wrote :

I have tested the -server package on Bionic and I haven't had any crashes yet. Typically in my environment it would have crashed after one hour or two hours the latest.

Thanks for your work, Alex, Andreas and Marc! I can confirm that isc-dhcp-server 4.3.5-3ubuntu7.1 has been running for two days while the previous version crashed every few hours in my environment.

Marc Deslauriers (mdeslaur) wrote :

Great, thanks for testing, I'll release these early next week!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package isc-dhcp - 4.3.5-3ubuntu7.1

---------------
isc-dhcp (4.3.5-3ubuntu7.1) bionic-security; urgency=medium

  * SECURITY UPDATE: DoS via change in bind behaviour (LP: #1781699)
    - debian/patches/CVE-2019-6470.patch: use 0 instead of -1 to indicate
      empty heap index in includes/dhcpd.h, server/mdb6.c,
      server/tests/mdb6_unittest.c.
    - CVE-2019-6470

 -- Marc Deslauriers <email address hidden> Mon, 06 May 2019 09:00:01 -0400

Changed in isc-dhcp (Ubuntu Bionic):
status: In Progress → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package isc-dhcp - 4.3.5-3ubuntu9.1

---------------
isc-dhcp (4.3.5-3ubuntu9.1) cosmic-security; urgency=medium

  * SECURITY UPDATE: DoS via change in bind behaviour (LP: #1781699)
    - debian/patches/CVE-2019-6470.patch: use 0 instead of -1 to indicate
      empty heap index in includes/dhcpd.h, server/mdb6.c,
      server/tests/mdb6_unittest.c.
    - CVE-2019-6470

 -- Marc Deslauriers <email address hidden> Mon, 06 May 2019 08:57:40 -0400

Changed in isc-dhcp (Ubuntu Cosmic):
status: In Progress → Fix Released
Changed in isc-dhcp (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.