[8086:24f3] intel Corporation Wireless 8260 regression of 4.4.0-170-generic proposed kernel

[Taihsiang Ho]

When connecting bg-open wifi, the system will freeze.

Hardware: Dell Precision 5720 AIO (CID 201701-25364)
Kernel: 4.4.0-170-generic #199-Ubuntu SMP Thu Nov 14 01:45:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Image: pre-installed image (xenial 4.4 based)

[Steps to Reproduce]
1. install the target image
2. when the system is ready, connect to wifi(bg-open channel) over network-manager applet.

[Expected Result]
Connection established

[Actual Result]
System freezed

[Reproducing Rate]
100% (5 out of 5)

[Other Affected Platforms]
Dell Inspiron 13-7378 (201605-22329)

[Additional Info]
I was sometimes able to catch the following kernel oops

Nov 27 18:40:01 201701-25364 kernel: [ 600.497340] ------------[ cut here ]------------
Nov 27 18:40:01 201701-25364 kernel: [ 600.497345] WARNING: CPU: 0 PID: 30 at /build/linux-AweC5P/linux-4.4.0/block/blk-mq.c:794 __blk_mq_run_hw_queue+0x305/0x3a0()
Nov 27 18:40:01 201701-25364 kernel: [ 600.497346] Modules linked in: nvram msr rfcomm cmac bnep uvcvideo videobuf2_vmalloc videobuf2_memops btusb videobuf2_v4l2 videobuf2_core btrtl v4l2_common btbcm btintel videodev bluetooth m
edia ecdh_generic input_leds joydev hid_multitouch arc4 dell_led i2c_designware_platform i2c_designware_core dell_wmi snd_hda_codec_realtek(OE) sparse_keymap snd_hda_codec_generic(OE) i915_bpo dcdbas dell_smm_hwmon intel_ips drm_k
ms_helper intel_rapl x86_pkg_temp_thermal iwlmvm(OE) intel_powerclamp coretemp drm kvm_intel mac80211(OE) i2c_algo_bit fb_sys_fops syscopyarea kvm sysfillrect sysimgblt snd_hda_intel(OE) snd_hda_codec(OE) irqbypass snd_hda_core(OE
) crct10dif_pclmul crc32_pclmul snd_hwdep ghash_clmulni_intel aesni_intel aes_x86_64 lrw snd_pcm gf128mul iwlwifi(OE) glue_helper ablk_helper cryptd snd_seq_midi snd_seq_midi_event cfg80211(OE) snd_rawmidi snd_seq serio_raw snd_se
q_device snd_timer rtsx_pci_ms compat(OE) snd memstick idma64 soundcore virt_dma mei_me intel_lpss_pci mei shpchp intel_lpss 8250_fintek mac_hid acpi_pad parport_pc ppdev lp parport autofs4 btrfs xor raid6_pq dm_mirror dm_region_h
ash dm_log hid_generic usbhid hid mmc_block uas usb_storage rtsx_pci_sdmmc psmouse e1000e ptp pps_core rtsx_pci ahci nvme libahci wmi video fjes
Nov 27 18:40:01 201701-25364 kernel: [ 600.497418] CPU: 0 PID: 30 Comm: kworker/4:0H Tainted: G OE 4.4.0-170-generic #199-Ubuntu
Nov 27 18:40:01 201701-25364 kernel: [ 600.497419] Hardware name: Dell Inc. Precision 5720 AIO/, BIOS 2.3.6 01/31/2018
Nov 27 18:40:01 201701-25364 kernel: [ 600.497421] Workqueue: kblockd blk_mq_run_work_fn
Nov 27 18:40:01 201701-25364 kernel: [ 600.497422] 0000000000000286 3b6f2186023dc8e9 ffff8808394dbd30 ffffffff8140cde1
Nov 27 18:40:01 201701-25364 kernel: [ 600.497424] 0000000000000000 ffffffff81d07a88 ffff8808394dbd68 ffffffff81086492
Nov 27 18:40:01 201701-25364 kernel: [ 600.497425] ffff880831d52400 ffff8808394dbda0 ffff88085dd15f80 ffff88085dd1b400
Nov 27 18:40:01 201701-25364 kernel: [ 600.497427] Call Trace:
Nov 27 18:40:01 201701-25364 kernel: [ 600.497430] [<ffffffff8140cde1>] dump_stack+0x63/0x82
Nov 27 18:40:01 201701-25364 kernel: [ 600.497433] [<ffffffff81086492>] warn_slowpath_common+0x82/0xc0
Nov 27 18:40:01 201701-25364 kernel: [ 600.497435] [<ffffffff810865da>] warn_slowpath_null+0x1a/0x20
Nov 27 18:40:01 201701-25364 kernel: [ 600.497436] [<ffffffff813e8305>] __blk_mq_run_hw_queue+0x305/0x3a0
Nov 27 18:40:01 201701-25364 kernel: [ 600.497439] [<ffffffff81862441>] ? __schedule+0x301/0x810
Nov 27 18:40:01 201701-25364 kernel: [ 600.497440] [<ffffffff8186244d>] ? __schedule+0x30d/0x810
Nov 27 18:40:01 201701-25364 kernel: [ 600.497441] [<ffffffff81862441>] ? __schedule+0x301/0x810
Nov 27 18:40:01 201701-25364 kernel: [ 600.497443] [<ffffffff813e8742>] blk_mq_run_work_fn+0x12/0x20
Nov 27 18:40:01 201701-25364 kernel: [ 600.497445] [<ffffffff810a0c5b>] process_one_work+0x16b/0x4e0
Nov 27 18:40:01 201701-25364 kernel: [ 600.497446] [<ffffffff810a101e>] worker_thread+0x4e/0x590
Nov 27 18:40:01 201701-25364 kernel: [ 600.497447] [<ffffffff810a0fd0>] ? process_one_work+0x4e0/0x4e0
Nov 27 18:40:01 201701-25364 kernel: [ 600.497449] [<ffffffff810a76f7>] kthread+0xe7/0x100
Nov 27 18:40:01 201701-25364 kernel: [ 600.497450] [<ffffffff81862441>] ? __schedule+0x301/0x810
Nov 27 18:40:01 201701-25364 kernel: [ 600.497452] [<ffffffff810a7610>] ? kthread_create_on_node+0x1e0/0x1e0
Nov 27 18:40:01 201701-25364 kernel: [ 600.497454] [<ffffffff818677a5>] ret_from_fork+0x55/0x80
Nov 27 18:40:01 201701-25364 kernel: [ 600.497456] [<ffffffff810a7610>] ? kthread_create_on_node+0x1e0/0x1e0
Nov 27 18:40:01 201701-25364 kernel: [ 600.497457] ---[ end trace 87c55fce1375aae8 ]---

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-170-generic 4.4.0-170.199
ProcVersionSignature: Ubuntu 4.4.0-170.199-generic 4.4.200
Uname: Linux 4.4.0-170-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.21
Architecture: amd64
Date: Wed Nov 27 20:34:01 2019
DistributionChannelDescriptor:
 # This is a distribution channel descriptor
 # For more information see http://wiki.ubuntu.com/DistributionChannelDescriptor
 canonical-oem-somerville-xenial-amd64-20160624-2
InstallationDate: Installed on 2019-11-27 (0 days ago)
InstallationMedia: Ubuntu 16.04 "Xenial" - Build amd64 LIVE Binary 20160624-10:47
SourcePackage: linux-signed
UpgradeStatus: No upgrade log present (probably fresh install)

[Taihsiang Ho (tai271828)]

Add var log tarball 201701-25364-191127.tar.gz

[Taihsiang Ho (tai271828)]

component information

ubuntu@201701-25364:~$ lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:5918] (rev 05)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
00:14.0 USB controller [0c03]: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller [8086:a12f] (rev 31)
00:15.0 Signal processing controller [1180]: Intel Corporation 100 Series/C230 Series Chipset Family Serial IO I2C Controller #0 [8086:a160] (rev 31)
00:15.2 Signal processing controller [1180]: Intel Corporation 100 Series/C230 Series Chipset Family Serial IO I2C Controller #2 [8086:a162] (rev 31)
00:16.0 Communication controller [0780]: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 [8086:a13a] (rev 31)
00:16.3 Serial controller [0700]: Intel Corporation 100 Series/C230 Series Chipset Family KT Redirection [8086:a13d] (rev 31)
00:17.0 SATA controller [0106]: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] [8086:a102] (rev 31)
00:1b.0 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #17 [8086:a167] (rev f1)
00:1c.0 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #6 [8086:a115] (rev f1)
00:1c.6 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #7 [8086:a116] (rev f1)
00:1d.0 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #13 [8086:a11c] (rev f1)
00:1f.0 ISA bridge [0601]: Intel Corporation C236 Chipset LPC/eSPI Controller [8086:a149] (rev 31)
00:1f.2 Memory controller [0580]: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller [8086:a121] (rev 31)
00:1f.3 Audio device [0403]: Intel Corporation 100 Series/C230 Series Chipset Family HD Audio Controller [8086:a170] (rev 31)
00:1f.4 SMBus [0c05]: Intel Corporation 100 Series/C230 Series Chipset Family SMBus [8086:a123] (rev 31)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-LM [8086:15b7] (rev 31)
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro WX 7100 Mobile] [1002:67c0]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
02:00.0 Non-Volatile memory controller [0108]: Toshiba America Info Systems XG4 NVMe SSD Controller [1179:0115] (rev 01)
03:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS525A PCI Express Card Reader [10ec:525a] (rev 01)
04:00.0 Network controller [0280]: Intel Corporation Wireless 8260 [8086:24f3] (rev 3a)

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1854156

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Rex Tsai (chihchun)]

The dkms was updated at 29 Mar 2019

ubuntu@201701-25364:/tmp$ dkms status
amdgpu-pro, 16.60-377537, 4.4.0-127-generic, x86_64: installed
amdgpu-pro, 16.60-377537, 4.4.0-49-generic, x86_64: installed
oem-audio-hda-daily, 0.201708030416~ubuntu16.04.1, 4.4.0-127-generic, x86_64: installed
oem-audio-hda-daily, 0.201708030416~ubuntu16.04.1, 4.4.0-170-generic, x86_64: installed
oem-wifi-intel-iwlwifi-lp1757035-4.4-dkms, 2.0, 4.4.0-170-generic, x86_64: installed
ubuntu@201701-25364:/tmp$ apt -o Debug::Acquire::http=true download oem-wifi-intel-iwlwifi-lp1757035-4.4-dkms-dkms
0% [Working]GET /updates/pool/public/o/oem-wifi-intel-iwlwifi-lp1757035-4.4-dkms-dkms/oem-wifi-intel-iwlwifi-lp1757035-4.4-dkms-dkms_2.0_all.deb HTTP/1.1
Host: dell.archive.canonical.com
User-Agent: Debian APT-HTTP/1.3 (1.2.32)

0% [Waiting for headers]Answer for: http://dell.archive.canonical.com/updates/pool/public/o/oem-wifi-intel-iwlwifi-lp1757035-4.4-dkms-dkms/oem-wifi-intel-iwlwifi-lp1757035-4.4-dkms-dkms_2.0_all.deb
HTTP/1.1 200 OK
Date: Wed, 27 Nov 2019 14:24:39 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Fri, 29 Mar 2019 10:56:31 GMT
ETag: "6f5c20-5853985faf5c0"
Accept-Ranges: bytes
Content-Length: 7298080
Content-Type: application/x-debian-package

Get:1 http://dell.archive.canonical.com/updates xenial-dell/public amd64 oem-wifi-intel-iwlwifi-lp1757035-4.4-dkms-dkms all 2.0 [7298 kB]
Fetched 7298 kB in 3s (1978 kB/s)

[Taihsiang Ho (tai271828)]

The following tests were perform:

- tested with 4.4.0-169 (Pass)
- re-install the system, dist-upgrade, tested with 4.4.0-170 (Fail, still reproducible)

[You-Sheng Yang (vicamo)]

@Rex, is it possible to get a list of platforms has oem-wifi-intel-iwlwifi-*-dkms installed?

[You-Sheng Yang (vicamo)]

Not reproducible on 4.4.0-169-generic, so it's a kernel regression.

By the way, amdgpu-pro DKMS 16.60-377537 doesn't compile on at least 4.4.0-169-generic and newer, should file a separate bug later.

[You-Sheng Yang (vicamo)]

Also tried removal of that dkms, wifi is still good on both 169 and 170. No hang found on both.

[You-Sheng Yang (vicamo)]

$ git bisect log
git bisect start
# old: [211ade40bea4d9ec98fb2789f6eb5de03fa103e1] UBUNTU: Ubuntu-4.4.0-169.198
git bisect old 211ade40bea4d9ec98fb2789f6eb5de03fa103e1
# new: [6a3433948fdca4055ca7d95741d1582d693cd58d] UBUNTU: Ubuntu-4.4.0-170.199
git bisect new 6a3433948fdca4055ca7d95741d1582d693cd58d
# old: [4d55604ecd8ce0d0cae04121f3ff91fdde5f1c03] dccp: do not leak jiffies on the wire
git bisect old 4d55604ecd8ce0d0cae04121f3ff91fdde5f1c03
# new: [c2cfdf6ee684136452ddec2b75069193087f6ae2] UBUNTU: upstream stable to v4.4.200
git bisect new c2cfdf6ee684136452ddec2b75069193087f6ae2
# new: [5bea6f202ae661455429410d8274c886e46e9bcb] ARM: spectre-v2: harden branch predictor on context switches
git bisect new 5bea6f202ae661455429410d8274c886e46e9bcb
# new: [4f04a79c7b2489ba31611e7f777bd5d7302a4919] ARM: uaccess: remove put_user() code duplication
git bisect new 4f04a79c7b2489ba31611e7f777bd5d7302a4919
# old: [49e4f8fce53e93193895fa472d27ccaac9351b3c] inet: stop leaking jiffies on the wire
git bisect old 49e4f8fce53e93193895fa472d27ccaac9351b3c
# new: [c78d3185cb58118d9d82f3da78dc4ffb79988d63] ARM: 8478/2: arm/arm64: add arm-smccc
git bisect new c78d3185cb58118d9d82f3da78dc4ffb79988d63
# new: [4d7c9d283d6f672b0b46d8735636b7a9d1a0adf3] dmaengine: qcom: bam_dma: Fix resource leak
git bisect new 4d7c9d283d6f672b0b46d8735636b7a9d1a0adf3
# new: [1cd059216d0a021c0086d5d7acbc287de0213b23] net/flow_dissector: switch to siphash
git bisect new 1cd059216d0a021c0086d5d7acbc287de0213b23
# first new commit: [1cd059216d0a021c0086d5d7acbc287de0213b23] net/flow_dissector: switch to siphash

[You-Sheng Yang (vicamo)]

Reproducible when cherry-picked commit 1cd059216d0a ("net/flow_dissector: switch to siphash") to Ubuntu-4.4.0-169.198.

[You-Sheng Yang (vicamo)]

* This oem DKMS package was made from v5.1-rc1, and booting with a v5.1-rc1 kernel is immune,
* The net/flow_dissector patch (commit 55667441c84f) was backported from v5.4-rc6, and booting with v5.4 also just works,
* And backport commit 55667441c84f directly to v5.1-rc1 doesn't hang, either.

[You-Sheng Yang (vicamo)]

This DKMS package carries its own fq.h/fq_impl.h/skbuff.h, and yet it shares net/core/..., so there are interface mismatches and should probably be addressed in the DKMS package instead if we want to keep that stable backport.

[You-Sheng Yang (vicamo)]

Patch verified on both 4.4.0-169.198 and 4.4.0-170-199. Should work for all 4.4 generic kernels with or without net/flow_dissector patch backported.

[You-Sheng Yang (vicamo)]

PPA in https://launchpad.net/~vicamo/+archive/ubuntu/ppa-1854156