Intel AX210 iwlwifi firmware crash under stress tests: Microcode SW error detected. Restarting 0x0.

[You-Sheng Yang]

[SRU Justification]

[Impact]

iwlwifi may encounter firmware crash when running 5.13 kernel with
outdated firmware.

[Fix]

Commit 2b13afc1a0ff82 ("iwlwifi: add new FWs from core60-51 release")
added new rev -63 firmware.

[Test Case]

1. disconnect/connect stress

   $ sudo add-apt-repository ppa:checkbox-dev/ppa
   $ sudo apt install plainbox-provider-checkbox
   $ checkbox-cli run com.canonical.certification::stress/wireless_bluetooth_coex_connect_stress

   ignore connect bluetooth part, and set the SSID and password to proceed.

2. suspend/resume stress

   $ for i in $(seq 200); do \
       echo "Round $i..."; \
       nmcli d; \
       sleep 15; \
       rtcwake -m mem -s 15; \
   done

[Where problems could occur]

Rev -63 firmware is only used by 5.13 kernel or newer.

[Other Info]

While 5.13 kernel is only available in Focal and Impish, only Focal is
nominated here.

========== original bug description ==========

[Reproduce steps]

1. disconnect/connect stress
$ sudo add-apt-repository ppa:checkbox-dev/ppa
$ sudo apt install plainbox-provider-checkbox
$ checkbox-cli run com.canonical.certification::stress/wireless_bluetooth_coex_connect_stress
ignore connect bluetooth part, and set the SSID and password to proceed.

2. suspend/resume stress
$ for i in $(seq 200); do \
    echo "Round $i..."; \
    nmcli d; \
    sleep 15; \
    rtcwake -m mem -s 15; \
done

kernel: iwlwifi 0000:03:00.0: Microcode SW error detected. Restarting 0x0. [100/1757]
kernel: iwlwifi 0000:03:00.0: Start IWL Error Log Dump:
kernel: iwlwifi 0000:03:00.0: Status: 0x00000040, count: 6
kernel: iwlwifi 0000:03:00.0: Loaded firmware version: 59.601f3a66.0 ty-a0-gf-a0-59.ucode
kernel: iwlwifi 0000:03:00.0: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL
kernel: iwlwifi 0000:03:00.0: 0x00A08200 | trm_hw_status0
kernel: iwlwifi 0000:03:00.0: 0x00000000 | trm_hw_status1
kernel: iwlwifi 0000:03:00.0: 0x004D9BDC | branchlink2
kernel: iwlwifi 0000:03:00.0: 0x0000922A | interruptlink1
kernel: iwlwifi 0000:03:00.0: 0x0000922A | interruptlink2
kernel: iwlwifi 0000:03:00.0: 0x000161C4 | data1
kernel: iwlwifi 0000:03:00.0: 0x00000010 | data2
kernel: iwlwifi 0000:03:00.0: 0x00000000 | data3
kernel: iwlwifi 0000:03:00.0: 0x1C405003 | beacon time
kernel: iwlwifi 0000:03:00.0: 0x72B40004 | tsf low
kernel: iwlwifi 0000:03:00.0: 0x000003E4 | tsf hi
kernel: iwlwifi 0000:03:00.0: 0x00000000 | time gp1
kernel: iwlwifi 0000:03:00.0: 0x00EF1CDC | time gp2
kernel: iwlwifi 0000:03:00.0: 0x00000001 | uCode revision type
kernel: iwlwifi 0000:03:00.0: 0x0000003B | uCode version major
kernel: iwlwifi 0000:03:00.0: 0x601F3A66 | uCode version minor
kernel: iwlwifi 0000:03:00.0: 0x00000420 | hw version
kernel: iwlwifi 0000:03:00.0: 0x00489002 | board version
kernel: iwlwifi 0000:03:00.0: 0x8011001C | hcmd
kernel: iwlwifi 0000:03:00.0: 0xE6863000 | isr0
kernel: iwlwifi 0000:03:00.0: 0x01040000 | isr1
kernel: iwlwifi 0000:03:00.0: 0x68F0000A | isr2
kernel: iwlwifi 0000:03:00.0: 0x00C44C4F | isr3
kernel: iwlwifi 0000:03:00.0: 0x00200000 | isr4
kernel: iwlwifi 0000:03:00.0: 0x007C019C | last cmd Id
kernel: iwlwifi 0000:03:00.0: 0x000161C4 | wait_event
kernel: iwlwifi 0000:03:00.0: 0x000000D0 | l2p_control
kernel: iwlwifi 0000:03:00.0: 0x00018034 | l2p_duration
kernel: iwlwifi 0000:03:00.0: 0x0000003F | l2p_mhvalid
kernel: iwlwifi 0000:03:00.0: 0x00CF00F8 | l2p_addr_match
kernel: iwlwifi 0000:03:00.0: 0x00000009 | lmpm_pmg_sel
kernel: iwlwifi 0000:03:00.0: 0x00000000 | timestamp
kernel: iwlwifi 0000:03:00.0: 0x00009890 | flow_handler
kernel: iwlwifi 0000:03:00.0: Start IWL Error Log Dump:
kernel: iwlwifi 0000:03:00.0: Status: 0x00000040, count: 7
kernel: iwlwifi 0000:03:00.0: 0x2000345A | ADVANCED_SYSASSERT
kernel: iwlwifi 0000:03:00.0: 0x00000000 | umac branchlink1
kernel: iwlwifi 0000:03:00.0: 0x8045D8DE | umac branchlink2
kernel: iwlwifi 0000:03:00.0: 0xC0085288 | umac interruptlink1
kernel: iwlwifi 0000:03:00.0: 0x00000000 | umac interruptlink2
kernel: iwlwifi 0000:03:00.0: 0x00000007 | umac data1 [55/1757]
kernel: iwlwifi 0000:03:00.0: 0x00000006 | umac data2
kernel: iwlwifi 0000:03:00.0: 0xDEADBEEF | umac data3
kernel: iwlwifi 0000:03:00.0: 0x0000003B | umac major
kernel: iwlwifi 0000:03:00.0: 0x601F3A66 | umac minor
kernel: iwlwifi 0000:03:00.0: 0x00EF1CD5 | frame pointer
kernel: iwlwifi 0000:03:00.0: 0xC0885E24 | stack pointer
kernel: iwlwifi 0000:03:00.0: 0x007C019C | last host cmd
kernel: iwlwifi 0000:03:00.0: 0x00000000 | isr status reg
kernel: iwlwifi 0000:03:00.0: IML/ROM dump:
kernel: iwlwifi 0000:03:00.0: 0x00000B03 | IML/ROM error/state
kernel: iwlwifi 0000:03:00.0: 0x0000881A | IML/ROM data1
kernel: iwlwifi 0000:03:00.0: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0
kernel: iwlwifi 0000:03:00.0: Fseq Registers:
kernel: iwlwifi 0000:03:00.0: 0x60000000 | FSEQ_ERROR_CODE
kernel: iwlwifi 0000:03:00.0: 0x00440000 | FSEQ_TOP_INIT_VERSION
kernel: iwlwifi 0000:03:00.0: 0x00040090 | FSEQ_CNVIO_INIT_VERSION
kernel: iwlwifi 0000:03:00.0: 0x0000A652 | FSEQ_OTP_VERSION
kernel: iwlwifi 0000:03:00.0: 0x00000002 | FSEQ_TOP_CONTENT_VERSION
kernel: iwlwifi 0000:03:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN
kernel: iwlwifi 0000:03:00.0: 0x00400410 | FSEQ_CNVI_ID
kernel: iwlwifi 0000:03:00.0: 0x00400410 | FSEQ_CNVR_ID
kernel: iwlwifi 0000:03:00.0: 0x00400410 | CNVI_AUX_MISC_CHIP
kernel: iwlwifi 0000:03:00.0: 0x00400410 | CNVR_AUX_MISC_CHIP
kernel: iwlwifi 0000:03:00.0: 0x00009061 | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
kernel: iwlwifi 0000:03:00.0: 0x00000061 | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
kernel: iwlwifi 0000:03:00.0: WRT: Collecting data: ini trigger 4 fired (delay=0ms).

[You-Sheng Yang (vicamo)]

SRU: https://lists.ubuntu.com/archives/kernel-team/2021-June/121676.html (linux-firmware: focal)
PPA: https://launchpad.net/~vicamo/+archive/ubuntu/linux-firmware-staging

[You-Sheng Yang (vicamo)]

Committed to git tree. Need a new release.

[Chris Halse Rogers (raof)]

Hello You-Sheng, or anyone else affected,

Accepted linux-firmware into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/1.187.16 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[You-Sheng Yang (vicamo)]

Verified linux-firmware version 1.187.16 from focal-proposed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 1.187.16

---------------
linux-firmware (1.187.16) focal; urgency=medium

  * Intel Killer(R) Wi-Fi 6 AX1650i [8086:a0f0] Subsystem [1a56:1651]: iwlwifi
    firmware crash under iperf stress tests: Microcode SW error detected.
    Restarting 0x0 (LP: #1934217)
    - iwlwifi: update and add new FWs from core56-54 release

  * [SRU] Update amdgpu/renoir firmware (LP: #1934014)
    - amdgpu: update renoir firmware from 21.20

  * To support AMD W6600 and W6700 (LP: #1938149)
    - amdgpu: add sienna cichlid firmware for 20.45
    - amdgpu: update sienna cichlid firmware for 20.50
    - amdgpu: update sienna cichlid firmware from 21.10
    - amdgpu: update sienna cichlid firmware from 21.20
    - amdgpu: add initial dimgrey cavefish firmware from 21.20

  * [SRU][F/H/I] Add MT7921 wifi fw (LP: #1936452)
    - linux-firmware: add firmware for MT7921
    - linux-firmware: update firmware for MT7921 WiFi device
    - linux-firmware: update firmware for MT7921 WiFi device to 20210612122753
    - linux-firmware: add frimware for mediatek bluetooth chip (MT7921)
    - linux-firmware: update frimware for mediatek bluetooth chip (MT7921)

  * Intel AX210 iwlwifi firmware crash under stress tests: Microcode SW error
    detected. Restarting 0x0. (LP: #1933415)
    - iwlwifi: add new FWs from core60-51 release

 -- Timo Aaltonen <email address hidden> Mon, 02 Aug 2021 18:41:24 +0300

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for linux-firmware has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[You-Sheng Yang (vicamo)]

This caused bug 1941044, bug 1943918, and bug 1946222 in Focal. Also bug 1922406 in Hirsute.