eoan kernel does not contain "ipv6: do not free rt if FIB_LOOKUP_NOREF is set on suppress rule"

[Andrew Rennie]

[Impact]

An unprivileged local attacker could cause a denial of service, or possibly execute arbitrary code due to an ipv6 regression.

[Test Case]

An unpatched system will crash with the following command:

$ unshare -rUn sh -c 'ip link add dummy1 type dummy && ip link set dummy1 up && ip -6 route add default dev dummy1 && ip -6 rule add table main suppress_prefixlength 0 && ping -f 1234::1'

[Regression Potential]

Low. The change could theoretically introduce a memory leak but that would still be an improvement over immediate loss of system availability.

[Original Description]

Having recently upgraded to Eoan Ermine from Disco Dingo, my previously rock-solid wireguard now locks the system up shortly after I take the connection down with wg-quick down wg0.

Package:

wireguard:
  Installed: 0.0.20190913-1ubuntu1
  Candidate: 0.0.20190913-1ubuntu1
  Version table:
 *** 0.0.20190913-1ubuntu1 500
        500 http://gb.archive.ubuntu.com/ubuntu eoan/universe amd64 Packages
        500 http://gb.archive.ubuntu.com/ubuntu eoan/universe i386 Packages
        100 /var/lib/dpkg/status

Kernel:
5.3.0-13-generic

Snipped from /var/log/syslog:

kernel: [ 776.930804] BUG: unable to handle page fault for address: 0000000000001070
kernel: [ 776.930807] #PF: supervisor read access in kernel mode
kernel: [ 776.930808] #PF: error_code(0x0000) - not-present page
kernel: [ 776.930809] PGD 0 P4D 0
kernel: [ 776.930811] Oops: 0000 [#1] SMP NOPTI
kernel: [ 776.930813] CPU: 3 PID: 2598 Comm: Chrome_ChildIOT Tainted: G OE 5.3.0-13-generic #14-Ubuntu
kernel: [ 776.930813] Hardware name: Dell Inc. XPS 13 9380/0KTW76, BIOS 1.7.0 08/05/2019
kernel: [ 776.930817] RIP: 0010:ip6_sk_dst_store_flow+0x80/0xc0
kernel: [ 776.930819] Code: 48 8b 42 30 48 33 47 40 48 09 c1 0f b6 4f 12 b8 01 00 00 00 4d 0f 45 e9 31 db d3 e0 a9 bf ef ff ff 74 07 48 8b 9f f8 02 00 00 <48> 8b 46 70 31 d2 48 85 c0 74 0c 48 8b 40 10 48 85 c0 74 03 8b 50
kernel: [ 776.930820] RSP: 0018:ffffbeb841a9fcd8 EFLAGS: 00010202
kernel: [ 776.930821] RAX: 0000000000000080 RBX: ffffa0933c829360 RCX: 0000000000000007
kernel: [ 776.930822] RDX: ffffbeb841a9fd20 RSI: 0000000000001000 RDI: ffffa0933c828f00
kernel: [ 776.930823] RBP: ffffbeb841a9fcf0 R08: 0000000000000000 R09: 0000000000000000
kernel: [ 776.930823] R10: 0000000000000000 R11: ffffa093948fd800 R12: ffffa0933c829360
kernel: [ 776.930824] R13: ffffa0933c828f38 R14: 0000000000000001 R15: ffffa0933c829360
kernel: [ 776.930825] FS: 00007fbcd8a82700(0000) GS:ffffa0939e4c0000(0000) knlGS:0000000000000000
kernel: [ 776.930826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [ 776.930827] CR2: 0000000000001070 CR3: 000000049172a004 CR4: 00000000003606e0
kernel: [ 776.930828] Call Trace:
kernel: [ 776.930832] ip6_datagram_dst_update+0x15e/0x280
kernel: [ 776.930835] ? _raw_read_unlock_bh+0x20/0x30
kernel: [ 776.930837] __ip6_datagram_connect+0x1da/0x380
kernel: [ 776.930839] ip6_datagram_connect+0x2d/0x50
kernel: [ 776.930841] inet_dgram_connect+0x3f/0xc0
kernel: [ 776.930843] __sys_connect+0xf1/0x130
kernel: [ 776.930846] ? do_fcntl+0xe4/0x550
kernel: [ 776.930848] ? fput+0x13/0x15
kernel: [ 776.930849] __x64_sys_connect+0x1a/0x20
kernel: [ 776.930852] do_syscall_64+0x5a/0x130
kernel: [ 776.930854] entry_SYSCALL_64_after_hwframe+0x44/0xa9
kernel: [ 776.930855] RIP: 0033:0x7fbcde6324eb
kernel: [ 776.930856] Code: 83 ec 18 89 54 24 0c 48 89 34 24 89 7c 24 08 e8 ab fa ff ff 8b 54 24 0c 48 8b 34 24 41 89 c0 8b 7c 24 08 b8 2a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2f 44 89 c7 89 44 24 08 e8 e1 fa ff ff 8b 44
kernel: [ 776.930857] RSP: 002b:00007fbcd8a7ec90 EFLAGS: 00000293 ORIG_RAX: 000000000000002a
kernel: [ 776.930859] RAX: ffffffffffffffda RBX: 00000000ffffff94 RCX: 00007fbcde6324eb
kernel: [ 776.930859] RDX: 000000000000001c RSI: 00007fbcd8a7ecf0 RDI: 0000000000000022
kernel: [ 776.930860] RBP: 00007fbcd8a7edb0 R08: 0000000000000000 R09: 00007fbcd8a7edf8
kernel: [ 776.930861] R10: 00007fbcd8a7edf0 R11: 0000000000000293 R12: 0000250e77c19710
kernel: [ 776.930862] R13: 0000250e77c19900 R14: 00007fbcd8a7edc8 R15: 00007fbcd8a7edc8
kernel: [ 776.930863] Modules linked in: binfmt_misc wireguard(OE) ip6_udp_tunnel udp_tunnel ccm rfcomm uhid algif_hash algif_skcipher af_alg cmac bnep sof_pci_dev snd_sof_intel_hda_common snd_sof_intel_byt snd_sof_intel_ipc snd_sof snd_sof_nocodec snd_sof_xtensa_dsp snd_soc_skl snd_hda_codec_hdmi snd_soc_hdac_hda snd_hda_ext_core snd_soc_skl_ipc nls_iso8859_1 snd_soc_sst_ipc snd_soc_sst_dsp snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core snd_hda_codec_realtek snd_compress snd_hda_codec_generic ac97_bus snd_pcm_dmaengine ath10k_pci mei_hdcp snd_hda_intel intel_rapl_msr snd_hda_codec ath10k_core snd_hda_core snd_hwdep dell_laptop ath snd_pcm ledtrig_audio joydev mac80211 snd_seq_midi x86_pkg_temp_thermal snd_seq_midi_event intel_powerclamp coretemp snd_rawmidi kvm_intel uvcvideo btusb dell_wmi videobuf2_vmalloc kvm btrtl snd_seq videobuf2_memops btbcm irqbypass dell_smbios intel_cstate dcdbas btintel videobuf2_v4l2 intel_rapl_perf snd_seq_device bluetooth snd_timer input_leds snd serio_raw
kernel: [ 776.930888] wmi_bmof cfg80211 videobuf2_common intel_wmi_thunderbolt dell_wmi_descriptor ecdh_generic videodev rtsx_pci_ms soundcore processor_thermal_device mc mei_me libarc4 ecc ucsi_acpi hid_multitouch mei intel_rapl_common idma64 typec_ucsi memstick virt_dma intel_soc_dts_iosf intel_pch_thermal typec cdc_acm mac_hid int3403_thermal int340x_thermal_zone int3400_thermal intel_hid acpi_thermal_rel acpi_pad sparse_keymap sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 dm_crypt hid_generic crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 aesni_intel aes_x86_64 crypto_simd rtsx_pci_sdmmc cryptd i2c_algo_bit glue_helper drm_kms_helper psmouse syscopyarea nvme sysfillrect sysimgblt fb_sys_fops thunderbolt rtsx_pci nvme_core drm i2c_i801 intel_lpss_pci intel_lpss i2c_hid wmi hid pinctrl_cannonlake video pinctrl_intel
kernel: [ 776.930910] CR2: 0000000000001070
kernel: [ 776.930912] ---[ end trace a4cf4135f35abbbd ]---
kernel: [ 776.930913] RIP: 0010:ip6_sk_dst_store_flow+0x80/0xc0
kernel: [ 776.930915] Code: 48 8b 42 30 48 33 47 40 48 09 c1 0f b6 4f 12 b8 01 00 00 00 4d 0f 45 e9 31 db d3 e0 a9 bf ef ff ff 74 07 48 8b 9f f8 02 00 00 <48> 8b 46 70 31 d2 48 85 c0 74 0c 48 8b 40 10 48 85 c0 74 03 8b 50
kernel: [ 776.930916] RSP: 0018:ffffbeb841a9fcd8 EFLAGS: 00010202
kernel: [ 776.930917] RAX: 0000000000000080 RBX: ffffa0933c829360 RCX: 0000000000000007
kernel: [ 776.930917] RDX: ffffbeb841a9fd20 RSI: 0000000000001000 RDI: ffffa0933c828f00
kernel: [ 776.930918] RBP: ffffbeb841a9fcf0 R08: 0000000000000000 R09: 0000000000000000
kernel: [ 776.930919] R10: 0000000000000000 R11: ffffa093948fd800 R12: ffffa0933c829360
kernel: [ 776.930919] R13: ffffa0933c828f38 R14: 0000000000000001 R15: ffffa0933c829360
kernel: [ 776.930921] FS: 00007fbcd8a82700(0000) GS:ffffa0939e4c0000(0000) knlGS:0000000000000000
kernel: [ 776.930921] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: [ 776.930922] CR2: 0000000000001070 CR3: 000000049172a004 CR4: 00000000003606e0

[Jason A. Donenfeld (zx2c4)]

Most likely this is related to an invocation to `ip rule` that's being made, not WireGuard. Take a look at this mailing list post: https://lists.zx2c4.com/pipermail/wireguard/2019-October/004588.html

[Jason A. Donenfeld (zx2c4)]

Yep, confirmed that Eoan is broken. Here's reproduction steps:

root@scw-competent-dirac:~# uname -a
Linux scw-competent-dirac 5.3.0-13-generic #14-Ubuntu SMP Tue Sep 24 02:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@scw-competent-dirac:~# ip netns add crash
root@scw-competent-dirac:~# ip -n crash link add dummy1 type dummy
root@scw-competent-dirac:~# ip -n crash link set dummy1 up
root@scw-competent-dirac:~# ip -n crash -6 route add default dev dummy1
root@scw-competent-dirac:~# ip -n crash -6 rule add table main suppress_prefixlength 0
root@scw-competent-dirac:~# ip netns exec crash ping -f -c 1000 -W 1 1234::1 || true
PING 1234::1(1234::1) 56 data bytes
..Segmentation fault
root@scw-competent-dirac:~# ip -n crash -6 rule del table main suppress_prefixlength 0
root@scw-competent-dirac:~# ip -n crash link del dummy1

[ 100.388052] general protection fault: 0000 [#1] SMP NOPTI
[ 100.396544] CPU: 1 PID: 1680 Comm: ping Tainted: G W 5.3.0-13-generic #14-Ubuntu
[ 100.398869] Hardware name: Scaleway Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[ 100.401359] RIP: 0010:ip6_dst_hoplimit+0x1b/0x50
[ 100.402157] Code: 85 c9 44 8b 45 d0 74 9b eb 82 0f 1f 44 00 00 0f 1f 44 00 00 48 8b 47 10 55 48 83 e0 fc 8b 40 24 48 89 e5 85 c0 75 15 48 8b 07 <48> 8b 90 10 03 00 00 48 85 d2 74 08 8b 82 1c 01 00 00 5d c3 48 8b
[ 100.405133] RSP: 0018:ffffb7dcc04e3c20 EFLAGS: 00010246
[ 100.405940] RAX: 3b3856482af84913 RBX: ffffa01db31d3cf0 RCX: 0000000000000000
[ 100.407045] RDX: 00000000ffffffff RSI: ffffa01dada4e300 RDI: ffffa01dada4e300
[ 100.408261] RBP: ffffb7dcc04e3c20 R08: 0000000000000006 R09: 0000000000000000
[ 100.409433] R10: ffffb7dcc04e3d00 R11: 0000000000000039 R12: ffffb7dcc04e3e10
[ 100.410611] R13: ffffb7dcc04e3d00 R14: ffffa01db31d3900 R15: 0000000000000000
[ 100.411889] FS: 00007f6c12b8e040(0000) GS:ffffa01dbf700000(0000) knlGS:0000000000000000
[ 100.413180] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 100.414126] CR2: 00007f5c067453e0 CR3: 0000000031900000 CR4: 00000000003406e0
[ 100.415335] Call Trace:
[ 100.415746] rawv6_sendmsg+0x81c/0xad0
[ 100.416474] ? sock_common_recvmsg+0x49/0x70
[ 100.417131] inet_sendmsg+0x6c/0x70
[ 100.417730] ? security_socket_sendmsg+0x3f/0x60
[ 100.418468] ? inet_sendmsg+0x6c/0x70
[ 100.419109] sock_sendmsg+0x5e/0x70
[ 100.419775] __sys_sendto+0x113/0x190
[ 100.420517] ? __sys_recvmsg+0x59/0xa0
[ 100.421307] __x64_sys_sendto+0x29/0x30
[ 100.422036] do_syscall_64+0x5a/0x130
[ 100.422692] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 100.423479] RIP: 0033:0x7f6c12cd58aa
[ 100.424123] Code: 48 c7 c0 ff ff ff ff eb bc 0f 1f 80 00 00 00 00 f3 0f 1e fa 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 15 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 76 c3 0f 1f 44 00 00 55 48 83 ec 30 44 89 4c
[ 100.426939] RSP: 002b:00007ffe8eed1d28 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[ 100.428248] RAX: ffffffffffffffda RBX: 0000000000000040 RCX: 00007f6c12cd58aa
[ 100.429498] RDX: 0000000000000040 RSI: 0000560c046766c0 RDI: 0000000000000004
[ 100.430647] RBP: 0000560c046766c0 R08: 0000560c04674640 R09: 000000000000001c
[ 100.431843] R10: 0000000000000000 R11: ...

[Andrew Rennie (j-an6rew-u)]

Thanks - will wait for kernel fix.

[Jason A. Donenfeld (zx2c4)]

Here's a one liner that *doesn't require root* that you can use to test whether the kernel fix has landed:

    unshare -rUn sh -c 'ip link add dummy1 type dummy && ip link set dummy1 up && ip -6 route add default dev dummy1 && ip -6 rule add table main suppress_prefixlength 0 && ping -f 1234::1'

Note: this will crash your system.

[Andrew Rennie (j-an6rew-u)]

Thanks

[Tyler Hicks (tyhicks)]

Fix submitted: https://lists.ubuntu.com/archives/kernel-team/2019-October/104623.html

Since we're just about one week from the release of Eoan, this fix may not make the Eoan release. If that's the case, it will be included in the initial set of Stable Release Updates (SRU) for the Eoan kernels.

[Tyler Hicks (tyhicks)]

Thanks to Jason for alerting us of this issue and pointing us at the fix!

[Tyler Hicks (tyhicks)]

This is CVE-2019-18198

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-eoan' to 'verification-done-eoan'. If the problem still exists, change the tag 'verification-needed-eoan' to 'verification-failed-eoan'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Teemu Torma (teemu-torma)]

The proposed kernel works fine, tested with wireguard on multiple systems and the one liner mentioned in #4 does not crash the system.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.3.0-19.20

---------------
linux (5.3.0-19.20) eoan; urgency=medium

  * eoan/linux: 5.3.0-19.20 -proposed tracker (LP: #1848648)

  * eoan kernel does not contain "ipv6: do not free rt if FIB_LOOKUP_NOREF is
    set on suppress rule" (LP: #1847478)
    - ipv6: do not free rt if FIB_LOOKUP_NOREF is set on suppress rule

 -- Khalid Elmously <email address hidden> Fri, 18 Oct 2019 04:17:49 -0400

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted linux-gcp-5.3 (5.3.0-1008.9~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-gcp-5.3/unknown (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#linux-gcp-5.3

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!