Intel Killer(R) Wi-Fi 6 AX1650i [8086:a0f0] Subsystem [1a56:1651]: iwlwifi firmware crash under iperf stress tests: Microcode SW error detected. Restarting 0x0

[You-Sheng Yang]

[SRU Justification]

BugLink: https://bugs.launchpad.net/bugs/1934217

[Impact]

iwlwifi firmware crash under iperf stress tests: "Microcode SW error
detected. Restarting 0x0."

[Fix]

iwlwifi FWs from core56-54 release may fix this issue.

[Test Case]

This affects plenty of models:

* iwlwifi-9000-pu-b0-jf-b0-*:
  - killer 1550*
  - 9461/9462/9560
* iwlwifi-9260-th-b0-jf-b0-*:
  - 9260
* iwlwifi-Q*-hr-b0-*:
  - AX201
  - AX1650*
* iwlwifi-Q*-jf-b0-*:
  - 9560
* iwlwifi-cc-a0-*:
  - AX200

While -59 fw is only used by 5.11-hwe kernel in Focal, they are tested
only against hwe-5.11. For iwlwifi-9* fw, they are used by kernel since
5.1, they're tested against 5.4/5.6/5.8 and 5.11 kernels.

Test steps:
- Connect both DUTs to the 5G wifi AP
- on DUT 1
   - $ iperf3 -s -B 192.168.50.188
- on DUT 2
   - $ iperf3 -c 192.168.50.188 -B 192.168.50.91 -t 6000
   - After about 30 mins the wifi crashed with firmware related error

[Where problems could occur]

We have little info about each update of firmware blobs. So far we only
know that their revisions match those claimed by kernel driver and we
did not find other issues under the tests.

[Other Info]

Groovy 5.8 kernel supports -58 at max, and Hirsute has this already. So
only Focal is nominated for hwe-5.11 kernel support. For 5.13-oem, -63
fw is being SRU-ed in bug 1933415.

========== original bug description ==========

- Connect both DUTs to the 5G wifi AP
- on DUT 1
   - $ iperf3 -s -B 192.168.50.188
- on DUT 2
   - $ iperf3 -c 192.168.50.188 -B 192.168.50.91 -t 6000
   - After about 30 mins the wifi crashed with firmware related error

[105396.990450] iwlwifi 0000:00:14.3: Microcode SW error detected. Restarting 0x0.
[105396.990539] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
[105396.990541] iwlwifi 0000:00:14.3: Status: 0x00000040, count: 6
[105396.990543] iwlwifi 0000:00:14.3: Loaded firmware version: 55.d9698065.0 QuZ-a0-hr-b0-55.ucode
[105396.990545] iwlwifi 0000:00:14.3: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL
[105396.990547] iwlwifi 0000:00:14.3: 0x000022F0 | trm_hw_status0
[105396.990548] iwlwifi 0000:00:14.3: 0x00000000 | trm_hw_status1
[105396.990549] iwlwifi 0000:00:14.3: 0x004CA3DE | branchlink2
[105396.990551] iwlwifi 0000:00:14.3: 0x000007F6 | interruptlink1
[105396.990552] iwlwifi 0000:00:14.3: 0x000007F6 | interruptlink2
[105396.990553] iwlwifi 0000:00:14.3: 0x00016504 | data1
[105396.990554] iwlwifi 0000:00:14.3: 0x00001000 | data2
[105396.990555] iwlwifi 0000:00:14.3: 0x00000000 | data3
[105396.990557] iwlwifi 0000:00:14.3: 0x900148E4 | beacon time
[105396.990558] iwlwifi 0000:00:14.3: 0x0C6FF71A | tsf low
[105396.990559] iwlwifi 0000:00:14.3: 0x000001B4 | tsf hi
[105396.990560] iwlwifi 0000:00:14.3: 0x00000000 | time gp1
[105396.990562] iwlwifi 0000:00:14.3: 0x58CF3E54 | time gp2
[105396.990563] iwlwifi 0000:00:14.3: 0x00000001 | uCode revision type
[105396.990564] iwlwifi 0000:00:14.3: 0x00000037 | uCode version major
[105396.990565] iwlwifi 0000:00:14.3: 0xD9698065 | uCode version minor
[105396.990567] iwlwifi 0000:00:14.3: 0x00000351 | hw version
[105396.990568] iwlwifi 0000:00:14.3: 0x00C89004 | board version
[105396.990569] iwlwifi 0000:00:14.3: 0x800FFC03 | hcmd
[105396.990570] iwlwifi 0000:00:14.3: 0x00020000 | isr0
[105396.990572] iwlwifi 0000:00:14.3: 0x00400000 | isr1
[105396.990573] iwlwifi 0000:00:14.3: 0x08F00002 | isr2
[105396.990574] iwlwifi 0000:00:14.3: 0x04C338C8 | isr3
[105396.990575] iwlwifi 0000:00:14.3: 0x00000000 | isr4
[105396.990577] iwlwifi 0000:00:14.3: 0x05D6001C | last cmd Id
[105396.990578] iwlwifi 0000:00:14.3: 0x00016504 | wait_event
[105396.990579] iwlwifi 0000:00:14.3: 0x000000C4 | l2p_control
[105396.990580] iwlwifi 0000:00:14.3: 0x00000000 | l2p_duration
[105396.990582] iwlwifi 0000:00:14.3: 0x00000007 | l2p_mhvalid
[105396.990583] iwlwifi 0000:00:14.3: 0x00000000 | l2p_addr_match
[105396.990584] iwlwifi 0000:00:14.3: 0x00000009 | lmpm_pmg_sel
[105396.990585] iwlwifi 0000:00:14.3: 0x00000000 | timestamp
[105396.990586] iwlwifi 0000:00:14.3: 0x0000C0D4 | flow_handler
[105396.990628] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
[105396.990629] iwlwifi 0000:00:14.3: Status: 0x00000040, count: 7
[105396.990631] iwlwifi 0000:00:14.3: 0x2000307C | ADVANCED_SYSASSERT
[105396.990632] iwlwifi 0000:00:14.3: 0x00000000 | umac branchlink1
[105396.990633] iwlwifi 0000:00:14.3: 0x80465D2E | umac branchlink2
[105396.990634] iwlwifi 0000:00:14.3: 0x80473128 | umac interruptlink1
[105396.990635] iwlwifi 0000:00:14.3: 0x00000000 | umac interruptlink2
[105396.990637] iwlwifi 0000:00:14.3: 0x00000000 | umac data1
[105396.990638] iwlwifi 0000:00:14.3: 0xDEADBEEF | umac data2
[105396.990639] iwlwifi 0000:00:14.3: 0xDEADBEEF | umac data3
[105396.990640] iwlwifi 0000:00:14.3: 0x00000037 | umac major
[105396.990641] iwlwifi 0000:00:14.3: 0xD9698065 | umac minor
[105396.990642] iwlwifi 0000:00:14.3: 0x58CF3E47 | frame pointer
[105396.990644] iwlwifi 0000:00:14.3: 0xC0885EB0 | stack pointer
[105396.990645] iwlwifi 0000:00:14.3: 0x0080010D | last host cmd
[105396.990646] iwlwifi 0000:00:14.3: 0x00000000 | isr status reg
[105396.990668] iwlwifi 0000:00:14.3: Fseq Registers:
[105396.990691] iwlwifi 0000:00:14.3: 0x60000000 | FSEQ_ERROR_CODE
[105396.990694] iwlwifi 0000:00:14.3: 0x80290033 | FSEQ_TOP_INIT_VERSION
[105396.990717] iwlwifi 0000:00:14.3: 0x00090006 | FSEQ_CNVIO_INIT_VERSION
[105396.990748] iwlwifi 0000:00:14.3: 0x0000A482 | FSEQ_OTP_VERSION
[105396.990764] iwlwifi 0000:00:14.3: 0x00000003 | FSEQ_TOP_CONTENT_VERSION
[105396.990787] iwlwifi 0000:00:14.3: 0x4552414E | FSEQ_ALIVE_TOKEN
[105396.990808] iwlwifi 0000:00:14.3: 0x20000302 | FSEQ_CNVI_ID
[105396.990811] iwlwifi 0000:00:14.3: 0x01300504 | FSEQ_CNVR_ID
[105396.990834] iwlwifi 0000:00:14.3: 0x20000302 | CNVI_AUX_MISC_CHIP
[105396.990841] iwlwifi 0000:00:14.3: 0x01300504 | CNVR_AUX_MISC_CHIP
[105396.990868] iwlwifi 0000:00:14.3: 0x05B0905B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
[105396.990887] iwlwifi 0000:00:14.3: 0x0000025B | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
[105396.991043] iwlwifi 0000:00:14.3: WRT: Collecting data: ini trigger 5 fired.
[105396.991047] ieee80211 phy0: Hardware restart was requested

[You-Sheng Yang (vicamo)]

Testing PPA: https://launchpad.net/~vicamo/+archive/ubuntu/linux-firmware-staging (focal)

[You-Sheng Yang (vicamo)]

SRU linux-firmware: https://lists.ubuntu.com/archives/kernel-team/2021-July/122248.html (focal)

[Chris Halse Rogers (raof)]

Hello You-Sheng, or anyone else affected,

Accepted linux-firmware into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/1.187.16 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[You-Sheng Yang (vicamo)]

Verified linux-firmware version 1.187.16 from focal-proposed.

[Kai-Chuan Hsieh (kchsieh)]

Verify Dell XPS 9310 and Precision 5750.

The Wi-Fi card in XPS 9310 and Precision 5750 are both AX201, both utilize fw QuZ-a0-hr-b0-59.ucode.

They can perform iperf test for 6000 seconds without firmware crash log.

Test steps:

1. enable proposed
2. install iperf3 and linux-firmware on both machine
3. Connect both DUTs to the 5G wifi AP
- on Dell Precision 5750
   - $ iperf3 -s -i 1
- on Dell XPS 9310
   - $ iperf3 -c <target ip addr> -i 1 -t 6000

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 1.187.16

---------------
linux-firmware (1.187.16) focal; urgency=medium

  * Intel Killer(R) Wi-Fi 6 AX1650i [8086:a0f0] Subsystem [1a56:1651]: iwlwifi
    firmware crash under iperf stress tests: Microcode SW error detected.
    Restarting 0x0 (LP: #1934217)
    - iwlwifi: update and add new FWs from core56-54 release

  * [SRU] Update amdgpu/renoir firmware (LP: #1934014)
    - amdgpu: update renoir firmware from 21.20

  * To support AMD W6600 and W6700 (LP: #1938149)
    - amdgpu: add sienna cichlid firmware for 20.45
    - amdgpu: update sienna cichlid firmware for 20.50
    - amdgpu: update sienna cichlid firmware from 21.10
    - amdgpu: update sienna cichlid firmware from 21.20
    - amdgpu: add initial dimgrey cavefish firmware from 21.20

  * [SRU][F/H/I] Add MT7921 wifi fw (LP: #1936452)
    - linux-firmware: add firmware for MT7921
    - linux-firmware: update firmware for MT7921 WiFi device
    - linux-firmware: update firmware for MT7921 WiFi device to 20210612122753
    - linux-firmware: add frimware for mediatek bluetooth chip (MT7921)
    - linux-firmware: update frimware for mediatek bluetooth chip (MT7921)

  * Intel AX210 iwlwifi firmware crash under stress tests: Microcode SW error
    detected. Restarting 0x0. (LP: #1933415)
    - iwlwifi: add new FWs from core60-51 release

 -- Timo Aaltonen <email address hidden> Mon, 02 Aug 2021 18:41:24 +0300

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for linux-firmware has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.