systemd-resolved crashed with SIGSEGV in dns_packet_is_reply_for()

[Daniele Dellafiore]

[Impact]

 * Null-pointer dereference in resolved, results in resolved crash and reports on launchpad and errors.

[Test Case]

 * Unknown steps to reproduce
 * Monitor the drop off in crashes on errors.ubuntu.com:
https://errors.ubuntu.com/problem/ea90aefe098653f44b46e56d72e2cc05ff980465

[Regression Potential]

 * The behavior is similar, instead of crashing resolved it returns an error in the relevant function. Whilst this may not result in correct dns resolution for the affected dns packets, it should not result in resolved crashes.

[Original Bug Report]

This is one of background errors that happens without any active app being involved.
For the records, I had open: Firefox, Slack, Franz and the Terminal

The Ubuntu Error Tracker has been receiving reports about a problem regarding systemd. This problem was most recently seen with package version 233-6ubuntu2, the problem page at https://errors.ubuntu.com/problem/ea90aefe098653f44b46e56d72e2cc05ff980465 contains more details, including versions of packages affected, stacktrace or traceback, and individual crash reports.
If you do not have access to the Ubuntu Error Tracker you can request it at http://forms.canonical.com/reports/.

ProblemType: Crash
DistroRelease: Ubuntu 16.10
Package: systemd 231-5
ProcVersionSignature: Ubuntu 4.4.0-9136.55-generic 4.4.16
Uname: Linux 4.4.0-9136-generic x86_64
ApportVersion: 2.20.3-0ubuntu7
Architecture: amd64
CrashCounter: 1
Date: Thu Sep 8 09:33:55 2016
ExecutablePath: /lib/systemd/systemd-resolved
InstallationDate: Installed on 2013-06-06 (1189 days ago)
InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Release amd64 (20130424)
MachineType: Dell Inc. Dell System XPS L322X
ProcCmdline: /lib/systemd/systemd-resolved
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-9136-generic root=UUID=2850be62-a05e-4ab9-af2b-5f1fd159ce5d ro quiet splash vt.handoff=7
Signal: 11
SourcePackage: systemd
StacktraceTop:
 ?? ()
 ?? ()
 ?? () from /lib/systemd/libsystemd-shared-231.so
 sd_event_dispatch () from /lib/systemd/libsystemd-shared-231.so
 sd_event_run () from /lib/systemd/libsystemd-shared-231.so
Title: systemd-resolved crashed with SIGSEGV in sd_event_dispatch()
UpgradeStatus: Upgraded to yakkety on 2016-09-03 (4 days ago)
UserGroups:

dmi.bios.date: 04/18/2013
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A08
dmi.board.name: 0PJHXN
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.chassis.version: 0.1
dmi.modalias: dmi:bvnDellInc.:bvrA08:bd04/18/2013:svnDellInc.:pnDellSystemXPSL322X:pvr:rvnDellInc.:rn0PJHXN:rvrA00:cvnDellInc.:ct8:cvr0.1:
dmi.product.name: Dell System XPS L322X
dmi.sys.vendor: Dell Inc.

[Apport retracing service (apport)]

StacktraceTop:
 dns_packet_is_reply_for (key=0x558fe28027b0, p=0x558fe27fdb50) at ../src/resolve/resolved-dns-packet.c:2267
 dns_transaction_process_reply (t=0x558fe28027e0, p=0x558fe27fdb50) at ../src/resolve/resolved-dns-transaction.c:1010
 on_dns_packet.lto_priv.85 (s=<optimized out>, fd=<optimized out>, revents=<optimized out>, userdata=0x558fe28027e0) at ../src/resolve/resolved-dns-transaction.c:1107
 source_dispatch.lto_priv.92 (s=0x558fe27f2ea0) at ../src/libsystemd/sd-event/sd-event.c:2267
 sd_event_dispatch (e=e@entry=0x558fe2790280) at ../src/libsystemd/sd-event/sd-event.c:2626

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

Marking as a security bug, since this shows a crash in the packet parsing code that can potentially be triggered remotely by a hostile DNS server or spoofed responses.

[Dimitri John Ledkov (xnox)]

Without understanding at all how the dns_packet structures work, it seems possible that a packet can pass DNS_PACKET_QR==1 check, yet when processed by dns_packet_extract fail the DNS_PACKET_QDCOUNT(p)>0, and hence end up with packet->question remain as NULL, resulting in bombing out as NULL pointer dereference.

Downloading a core dump to assert that above analysis is true, would be nice for sending this to upstream.

[Dimitri John Ledkov (xnox)]

Somebody else agrees with me https://github.com/systemd/systemd/pull/5998

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 233-6ubuntu3

---------------
systemd (233-6ubuntu3) artful; urgency=medium

  * resolved: fix null pointer dereference crash (LP: #1621396)

 -- Dimitri John Ledkov <email address hidden> Mon, 22 May 2017 09:29:22 +0100

[Tyler Hicks (tyhicks)]

I've requested a CVE from MITRE for this issue.

[Tyler Hicks (tyhicks)]

Lennart pointed out in the upstream pull request that systemd-resolved is respawned after crashing. Therefore, Ubuntu Security considers this security issue to be a low priority. To reduce the risk of regressions in security updates, our general rule is to only perform security updates that fix a medium or higher issue or wait until around five low issues have accumulated. The fix is simple and low risk but there's always inherent risk in building/publishing/installing new binaries. We'll include the fix in a future security update if there are new issues discovered in systemd.

[Tyler Hicks (tyhicks)]

@xnox you previously mentioned that you had some systemd SRUs to prepare. Feel free to include this fix in those SRUs to address the error tracker reports. It just doesn't quite make sense to do a standalone security update of the init daemon for a low priority security issue.

This bug is an interesting corner case of not being quite important enough to warrant a security update yet being enough of an annoyance that it warrants an SRU. My apologies for not noticing this fact earlier.

[Łukasz Zemczak (sil2100)]

Thank you for uploading this stable release update! To ease the SRU review process and later package validation, could you please update the bug description to include the relevant SRU information [1]? Especially the Regression Potential field that's missing here.

[1] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

[Brian Murray (brian-murray)]

Hello Daniele, or anyone else affected,

Accepted systemd into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/232-21ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dimitri John Ledkov (xnox)]

There is now one crash report in proposed. I'm not sure if this is because resolved was running, and has not been restarted yet. Or there is still this genuine crash present in the proposed package.....

_systemctl try-restart systemd-resolved.service || true

is called in the postinst, so the daemon should have been restarted.

[Brian Murray (brian-murray)]

Its worth noting that apport creates an initial .crash file without much information in it, after the user chooses to send the crash to the Error Tracker then some information gathering is performed. This information gathering stage includes adding the version of the package. Looking at the particular instance, https://errors.ubuntu.com/oops/3680877e-5046-11e7-89b7-fa163e54c21f, with the version of the package from -proposed a couple of things stand out.

Date: Thu Jun 8 12:35:23 2017

This is before the package was accepted into -proposed. Additionally there is this:

UnreportableReason:
Неполадка произошла с программой /lib/systemd/systemd-resolved, в которую были внесены изменения с момента её аварийного завершения работы.

So this is not a genuine crash with the version of the package from -proposed.

[Dimitri John Ledkov (xnox)]

After checking the tracker, there are no new crashes reported for the proposed version of the package (systemd amd64 232-21ubuntu4) marking as verification done for zesty.

[Apport retracing service (apport)]

Package: systemd 232-21ubuntu3
ProcCmdline: /lib/systemd/systemd-resolved

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 232-21ubuntu4

---------------
systemd (232-21ubuntu4) zesty; urgency=medium

  * Cherrypick upstream commit to enable system use kernel maximum limit for
    RLIMIT_NOFILE isntead of hard-coded (low) limit of 65536. (LP: #1686361)
  * debian/tests/root-unittests: disable execute and seccomp tests on arm
    test-seccomp and test-execute fail on arm64 kernels. Marking both tests as
    expected failures. An upstream bug report is filed to resolve these.
    (LP: #1672499)
  * Cherrypick upstream patch for platform predictable interface names.
    (LP: #1686784)
  * resolved: fix null pointer dereference crash (LP: #1621396)
  * Cherrypick core/timer downgrade message about random time addition
    (LP: #1692136)

 -- Dimitri John Ledkov <email address hidden> Wed, 24 May 2017 16:26:16 +0100

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Hello Daniele, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dimitri John Ledkov (xnox)]

As per tracker, we have not had any reports of this crash in resolved in 16.04. Thus this fix is mostly advisory / precautionary one for xenial. Marking as verified. There are now even less chances for resolved to crash on systems that use resolved

[Steve Langasek (vorlon)]

LP: #1704677 is reported as a regression in this SRU. Marking verification failed pending resolution.

[Adam Conrad (adconrad)]

Hello Daniele, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dimitri John Ledkov (xnox)]

No crashes visible at https://errors.ubuntu.com/problem/ea90aefe098653f44b46e56d72e2cc05ff980465

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-4ubuntu19

---------------
systemd (229-4ubuntu19) xenial; urgency=medium

  * debian/extra/units/systemd-resolved.service.d/resolvconf.conf: partially
    revert, by removing ExecStart|StopPost lines, as these are not needed on
    xenial and generate warnings in the journal. (LP: #1704677)

systemd (229-4ubuntu18) xenial; urgency=medium

  * debian/extra/units/systemd-resolved.service.d/resolvconf.conf: if resolved
    is going to be started, make sure this blocks network-online.target.
    (LP: #1673860)
  * networkd: cherry-pick support for setting bridge port's priority
    (LP: #1668347)
  * Cherrypick upstream commit to enable system use kernel maximum limit for
    RLIMIT_NOFILE isntead of hard-coded (low) limit of 65536. (LP: #1686361)
  * Cherrypick upstream patch for platform predictable interface names.
    (LP: #1686784)
  * resolved: fix null pointer dereference crash (LP: #1621396)
  * Cherrypick core/timer downgrade message about random time addition
    (LP: #1692136)
  * SECURITY UPDATE: Out-of-bounds write in systemd-resolved (LP: #1695546)
    - CVE-2017-9445
  * Cherry-pick subset of patches to introduce infinity value in logind.conf
    for UserTasksMax (LP: #1651518)

 -- Dimitri John Ledkov <email address hidden> Mon, 17 Jul 2017 17:00:42 +0100