systemd-resolved crashed with SIGSEGV in dns_transaction_cache_answer()

[dino99]

Got that crash with the newer network-manager 1.2.2-0ubuntu3 upgrade with a gnome-shell session.
Reinstalling network-manager also ends with that crash.

ProblemType: Crash
DistroRelease: Ubuntu 16.10
Package: systemd 230-1git1
ProcVersionSignature: Ubuntu 4.4.0-23.41-generic 4.4.10
Uname: Linux 4.4.0-23-generic x86_64
NonfreeKernelModules: nvidia_uvm nvidia_modeset nvidia
ApportVersion: 2.20.1-0ubuntu4
Architecture: amd64
Date: Mon May 30 11:58:58 2016
ExecutablePath: /lib/systemd/systemd-resolved
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 003 Device 002: ID 046d:c062 Logitech, Inc. M-UAS144 [LS1 Laser Mouse]
 Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: ASUSTEK COMPUTER INC P5W DH Deluxe
ProcCmdline: /lib/systemd/systemd-resolved
ProcEnviron:
 LANG=en_GB.UTF-8
 LANGUAGE=en_GB:en
 PATH=(custom, no user)
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-23-generic root=UUID=7c755ed6-51cc-4b75-88ac-9c75acf82749 ro
Signal: 11
SourcePackage: systemd
StacktraceTop:
 ?? ()
 ?? ()
 ?? ()
 ?? ()
 ?? ()
SystemdDelta:
 [EXTENDED] /etc/systemd/system/display-manager.service → /lib/systemd/system/display-manager.service.d/xdiagnose.conf
 [EXTENDED] /lib/systemd/system/systemd-timesyncd.service → /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf
 [EXTENDED] /lib/systemd/system/rc-local.service → /lib/systemd/system/rc-local.service.d/debian.conf

 3 overridden configuration files found.
Title: systemd-resolved crashed with SIGSEGV
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

dmi.bios.date: 07/22/2010
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 3002
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: P5W DH Deluxe
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3002:bd07/22/2010:svnASUSTEKCOMPUTERINC:pnP5WDHDeluxe:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP5WDHDeluxe:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: P5W DH Deluxe
dmi.product.version: System Version
dmi.sys.vendor: ASUSTEK COMPUTER INC

[Apport retracing service (apport)]

StacktraceTop:
 DNS_PACKET_SHALL_CACHE () at ../src/resolve/resolved-dns-packet.h:211
 dns_transaction_cache_answer (t=0x55d99b3b48e0) at ../src/resolve/resolved-dns-transaction.c:582
 dns_transaction_process_dnssec.lto_priv.425 (t=0x55d99b3b48e0) at ../src/resolve/resolved-dns-transaction.c:717
 dns_transaction_notify (source=0x55d99b3b71b0, t=0x55d99b3b48e0) at ../src/resolve/resolved-dns-transaction.c:2099
 dns_transaction_complete (t=0x55d99b3b71b0, state=<optimized out>) at ../src/resolve/resolved-dns-transaction.c:361

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Pitt (pitti)]

I can provoke a very similar crash with "systemd-resolve www.facebook.com":
www.facebook.com: resolve call failed: DNSSEC validation failed: failed-auxiliary

Mai 31 14:00:45 donald systemd-resolved[14605]: Using degraded feature set (UDP) for DNS server 192.168.2.1.
Mai 31 14:00:45 donald systemd-resolved[14605]: DNSSEC validation failed for question c10r.facebook.com IN SOA: failed-auxiliary
Mai 31 14:00:45 donald systemd-resolved[14605]: DNSSEC validation failed for question star-mini.c10r.facebook.com IN DS: failed-auxiliary
Mai 31 14:00:45 donald systemd-resolved[14605]: DNSSEC validation failed for question star-mini.c10r.facebook.com IN SOA: failed-auxiliary
Mai 31 14:00:45 donald systemd-resolved[14605]: DNSSEC validation failed for question star-mini.c10r.facebook.com IN AAAA: failed-auxiliary
Mai 31 14:00:45 donald systemd-resolved[14605]: DNSSEC validation failed for question star-mini.c10r.facebook.com IN A: failed-auxiliary
Mai 31 14:00:45 donald systemd-resolved[14605]: *** Error in `/lib/systemd/systemd-resolved': double free or corruption (top): 0x0000558e1a5feac0 ***

[Martin Pitt (pitti)]

Corresponding backtrace:

#0 0x00007f24559c2418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
        resultvar = 0
        pid = 14605
        selftid = 14605
#1 0x00007f24559c401a in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x66636666370a5d6b, sa_sigaction = 0x66636666370a5d6b}, sa_mask = {__val = {3256155514939455845, 7293972561931953719, 3255383588231721059, 3472328296227676272, 3472339291342909488, 2314885530818457632, 2314885530818453536, 2314885530818453536, 7022930802683944992, 7377853203759127922, 3256155514973010277, 7293972561931953719, 8659703141076316261, 3472328296227676272, 3472339291342909488, 2314885530818457632}}, sa_flags = 538976288, sa_restorer = 0x66}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00007f2455a0472a in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f2455b1d6b0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
        ap = <error reading variable ap (Attempt to dereference a generic pointer.)>
        fd = 2
        on_2 = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#3 0x00007f2455a0cf4a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7f2455b1d7a0 "double free or corruption (top)", action=3) at malloc.c:5007
        buf = "0000558e1a5feac0"
        cp = <optimized out>
        ar_ptr = <optimized out>
        str = 0x7f2455b1d7a0 "double free or corruption (top)"
        action = 3
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3868
        size = <optimized out>
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        errstr = <optimized out>
        locked = <optimized out>
#5 0x00007f2455a10abc in __GI___libc_free (mem=<optimized out>) at malloc.c:2969
        ar_ptr = <optimized out>
        p = <optimized out>
        hook = <optimized out>
#6 0x0000558e1a28e833 in dns_packet_free (p=0x558e1a5feac0) at ../src/resolve/resolved-dns-packet.c:177
No locals.
#7 dns_packet_unref (p=0x558e1a5feac0) at ../src/resolve/resolved-dns-packet.c:189
        __PRETTY_FUNCTION__ = "dns_packet_unref"
#8 0x0000558e1a24de56 in dns_transaction_free (t=t@entry=0x558e1a5e1a70) at ../src/resolve/resolved-dns-transaction.c:87
        __func__ = "dns_transaction_free"
        __PRETTY_FUNCTION__ = "dns_transaction_free"
#9 0x0000558e1a24e42b in dns_transaction_gc (t=0x558e1a5e1a70) at ../src/resolve/resolved-dns-transaction.c:148
        t = 0x558e1a5e1a70
        __PRETTY_FUNCTION__ = "dns_transaction_gc"
#10 0x0000558e1a24e57f in dns_transaction_complete (t=0x558e1a5e1a70, state=<optimized out>) at ../src/resolve/resolved-dns-transaction.c:365
        st = <optimized out>
        key_str = "\034'\001\000\000\000\000\000\377\000\000\000\000\000\000\000\r\000\000\000\000\000\000\000\000\n]\032\216U\000\000\001\000\000\000\000\000\000\000\000\240\235\376\374\...

[dino99 (9d9)]

Feedback after upgrading to systemd 230-2, and a cold reboot:

- still lot of systemd-resolved entries into journalctl (details joined)
- but reinstalling network-manager files & dependencies does not generate a crash as previously

- journalctl log a kernel segfault:
kernel: systemd-resolve[2087]: segfault at 5c ip 0000563086065ee7 sp 00007ffd6a3ffab0 error 4 in systemd-resolved[56308602c000+9f000]

[dino99 (9d9)]

Also logged:
systemd-resolved[3704]: Assertion '*_head == _item' failed at ../src/resolve/resolved-dns-transaction.c:94, function dns_transaction_free(). Aborting.

[dino99 (9d9)]

230-2 issue report: lp:1587743

[dino99 (9d9)]

Upgrade to 230-2git1 then reboot; and test:

systemd-resolve www.facebook.com
www.facebook.com: 179.60.192.36
                  (star-mini.c10r.facebook.com)

-- Information acquired via protocol DNS in 12.3ms.
-- Data is authenticated: no

[Ads20000 (ads20000)]

systemd-resolve www.facebook.com
www.facebook.com: 31.13.90.36
                  (star-mini.c10r.facebook.com)

-- Information acquired via protocol DNS in 310.7ms.
-- Data is authenticated: no

Seems to work fine for me in 230-2git1

[Ads20000 (ads20000)]

Has the issue been passed upstream/should it be passed upstream or is it an Ubuntu-specific issue? Having to patch it every time it's upgraded sounds annoying.

[Martin Pitt (pitti)]

Yes, it's reported upstream (https://github.com/systemd/systemd/issues/2942) and the package currently waiting in yakkety-proposed fixes this (or rather, works around this by disabling DNSSEC again for the time being).

[Martin Pitt (pitti)]

This is fixed/worked around in https://launchpad.net/ubuntu/+source/systemd/230-2git1, but I typoed the bug number.