iwlwifi driver for Intel 7260 crashing

[Mike Rushton]

Intel NUC's from Axiomtek with Intel Corporation Wireless 7260 (rev bb) chipsets. We have this running on almost 200 of these devices.

Ubuntu 16.04.3 as well as Ubuntu 14.04.5

"loaded firmware version 17.608620.0 op_mode iwlmvm"

linux-firmware trusty: 1.127.24
linux-firmware xenial: 1.157.14

The wireless driver will crash at least daily, if not multiple times in a day depending on the machine. Swapping the chipset out for an Atheros works just fine.

I have followed the many other bug reports related to this same chipset and they are either marked incomplete, refer to hardware we aren't running or have been marked as fix released.

The latest linux-firmware did not resolve or change the behavior of the issue.

Output from dmesg when the crash occurs:

[Wed Dec 6 18:56:38 2017] iwlwifi 0000:02:00.0: Queue 16 stuck for 10000 ms.
[Wed Dec 6 18:56:38 2017] iwlwifi 0000:02:00.0: Current SW read_ptr 236 write_ptr 88
[Wed Dec 6 18:56:38 2017] ------------[ cut here ]------------
[Wed Dec 6 18:56:38 2017] WARNING: CPU: 3 PID: 0 at /build/linux-lts-xenial-CtFtiH/linux-lts-xenial-4.4.0/drivers/net/wireless/iwlwifi/pcie/trans.c:1552 iwl_trans_pcie_grab_nic_access+0xf5/0x100 [iwlwifi]()
[Wed Dec 6 18:56:38 2017] Timeout waiting for hardware access (CSR_GP_CNTRL 0xffffffff)
[Wed Dec 6 18:56:38 2017] Modules linked in: drbg ansi_cprng ctr ccm uvcvideo videobuf2_vmalloc btusb videobuf2_memops btrtl videobuf2_v4l2 btbcm btintel videobuf2_core hid_generic bluetooth v4l2_common videodev usbhid media arc4 snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic iwlmvm mac80211 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass iwlwifi crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cfg80211 aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_soc_rt5640 snd_soc_rl6231 snd_soc_ssm4567 serio_raw lpc_ich i915 shpchp binfmt_misc snd_hda_intel snd_soc_core snd_compress snd_hda_codec ac97_bus snd_seq_midi snd_seq_midi_event snd_rawmidi snd_pcm_dmaengine snd_hda_core snd_hwdep snd_pcm 8250_fintek snd_seq video snd_seq_device snd_timer drm_kms_helper snd elan_i2c 8250_dw drm dw_dmac i2c_hid hid dw_dmac_core mei_me fb_sys_fops syscopyarea snd_soc_sst_acpi i2c_designware_platform mac_hid sysfillrect spi_pxa2xx_platform mei i2c_designware_core soundcore acpi_pad sysimgblt acpi_als kfifo_buf industrialio lp parport igb psmouse ahci e1000e i2c_algo_bit libahci dca ptp pps_core fjes sdhci_acpi sdhci
[Wed Dec 6 18:56:38 2017] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.0-101-generic #124~14.04.1-Ubuntu
[Wed Dec 6 18:56:38 2017] Hardware name: To be filled by O.E.M. To be filled by O.E.M./CRESCENTBAY, BIOS 5.6.5 12/14/2016
[Wed Dec 6 18:56:38 2017] 0000000000000000 ffff88012dd83d00 ffffffff813dfc6c ffff88012dd83d48
[Wed Dec 6 18:56:38 2017] ffffffffc07ca6e0 ffff88012dd83d38 ffffffff8107ea26 ffff8800ca470000
[Wed Dec 6 18:56:38 2017] ffff8800ca4735f0 ffff88012dd83dd8 0000000000000000 ffff880128108bc0
[Wed Dec 6 18:56:38 2017] Call Trace:
[Wed Dec 6 18:56:38 2017] <IRQ> [<ffffffff813dfc6c>] dump_stack+0x63/0x87
[Wed Dec 6 18:56:38 2017] [<ffffffff8107ea26>] warn_slowpath_common+0x86/0xc0
[Wed Dec 6 18:56:38 2017] [<ffffffff8107eaac>] warn_slowpath_fmt+0x4c/0x50
[Wed Dec 6 18:56:38 2017] [<ffffffffc07aa13f>] ? iwl_read32+0x1f/0x90 [iwlwifi]
[Wed Dec 6 18:56:38 2017] [<ffffffffc07b9255>] iwl_trans_pcie_grab_nic_access+0xf5/0x100 [iwlwifi]
[Wed Dec 6 18:56:38 2017] [<ffffffffc07b72bf>] iwl_trans_pcie_read_mem+0x2f/0xa0 [iwlwifi]
[Wed Dec 6 18:56:38 2017] [<ffffffffc07b3f38>] iwl_pcie_txq_stuck_timer+0xd8/0x390 [iwlwifi]
[Wed Dec 6 18:56:38 2017] [<ffffffff816a92ab>] ? intel_pstate_timer_func+0x2db/0x3a0
[Wed Dec 6 18:56:38 2017] [<ffffffffc07b3e60>] ? iwl_pcie_enqueue_hcmd+0xa90/0xa90 [iwlwifi]
[Wed Dec 6 18:56:38 2017] [<ffffffff810e7775>] call_timer_fn+0x35/0x130
[Wed Dec 6 18:56:38 2017] [<ffffffffc07b3e60>] ? iwl_pcie_enqueue_hcmd+0xa90/0xa90 [iwlwifi]
[Wed Dec 6 18:56:38 2017] [<ffffffff810e912e>] run_timer_softirq+0x20e/0x2c0
[Wed Dec 6 18:56:38 2017] [<ffffffff8108319d>] __do_softirq+0xdd/0x290
[Wed Dec 6 18:56:38 2017] [<ffffffff81083575>] irq_exit+0x95/0xa0
[Wed Dec 6 18:56:38 2017] [<ffffffff81811fa2>] smp_apic_timer_interrupt+0x42/0x50
[Wed Dec 6 18:56:38 2017] [<ffffffff81810262>] apic_timer_interrupt+0x82/0x90
[Wed Dec 6 18:56:38 2017] <EOI> [<ffffffff816a99f8>] ? cpuidle_enter_state+0xd8/0x250
[Wed Dec 6 18:56:38 2017] [<ffffffff816a99d4>] ? cpuidle_enter_state+0xb4/0x250
[Wed Dec 6 18:56:38 2017] [<ffffffff816a9ba7>] cpuidle_enter+0x17/0x20
[Wed Dec 6 18:56:38 2017] [<ffffffff810bfad2>] call_cpuidle+0x32/0x60
[Wed Dec 6 18:56:38 2017] [<ffffffff816a9b83>] ? cpuidle_select+0x13/0x20
[Wed Dec 6 18:56:38 2017] [<ffffffff810bfd89>] cpu_startup_entry+0x289/0x350
[Wed Dec 6 18:56:38 2017] [<ffffffff8104f529>] start_secondary+0x149/0x170
[Wed Dec 6 18:56:38 2017] ---[ end trace ad9659e8d87e5230 ]---
[Wed Dec 6 18:56:38 2017] iwl data: 00000000: b0 68 00 00 14 00 00 00 b8 8a 5c 00 00 00 00 00 .h........\.....
[Wed Dec 6 18:56:38 2017] iwlwifi 0000:02:00.0: FH TRBs(0) = 0x5a5a5a5a
[Wed Dec 6 18:56:38 2017] iwlwifi 0000:02:00.0: FH TRBs(1) = 0x5a5a5a5a
[Wed Dec 6 18:56:38 2017] iwlwifi 0000:02:00.0: FH TRBs(2) = 0x5a5a5a5a
[Wed Dec 6 18:56:38 2017] iwlwifi 0000:02:00.0: FH TRBs(3) = 0x5a5a5a5a
[Wed Dec 6 18:56:38 2017] iwlwifi 0000:02:00.0: FH TRBs(4) = 0x5a5a5a5a
[Wed Dec 6 18:56:38 2017] iwlwifi 0000:02:00.0: FH TRBs(5) = 0x5a5a5a5a
[Wed Dec 6 18:56:38 2017] iwlwifi 0000:02:00.0: FH TRBs(6) = 0x5a5a5a5a
[Wed Dec 6 18:56:38 2017] iwlwifi 0000:02:00.0: FH TRBs(7) = 0x5a5a5a5a

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-firmware 1.127.24
ProcVersionSignature: Ubuntu 4.4.0-101.124~14.04.1-generic 4.4.95
Uname: Linux 4.4.0-101-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.27
Architecture: amd64
Date: Wed Dec 6 19:48:34 2017
Dependencies:

InstallationDate: Installed on 2017-01-05 (335 days ago)
InstallationMedia: Ubuntu-Server 14.04.5 LTS "Trusty Tahr" - Release amd64 (20160803)
PackageArchitecture: all
ProcEnviron:
 LANGUAGE=en_US:
 TERM=linux
 PATH=(custom, no user)
 LANG=en_US
 SHELL=/bin/bash
SourcePackage: linux-firmware
UpgradeStatus: No upgrade log present (probably fresh install)

[Simon Ye (yesimon)]

This is a duplicate of https://bugzilla.kernel.org/show_bug.cgi?id=191601, which also contains a script that might be able to reset your card without a reboot.

Please see my longer comments here: https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1673344. In summary, I believe this to be a longstanding hardware-software bug in Linux due to a hardware degraded 7260 NIC. Replacing the 7260 card fixed it for me.