Intermittent AX211 iwlwifi RT ucode init failed -110 at boot

[You-Sheng Yang]

[SRU Justification]

[Impact]

AX211 might fail to load -64 firmware on some platforms.

  iwlwifi 0000:00:14.3: Failed to start RT ucode: -110

[Fix]

This can only be fixed in the firmware, and currently v5.14 kernel has
FW API 64, v5.15 has 66 and only v5.16 or above has 67.

[Test Case]

Run checkbox reboot stress tests to try trigger this issue:

  $ checkbox-cli run com.canonical.certification::stress/reboot

[Where problems could occur]

The source commit originally contains 67 fw for several models, and
yet AX211, which taks iwlwifi-so-a0-gf-a0-*.ucode and its
corresponding pnvm file, is the only known problematic model
affected. This pull request imports only the needed files to
eliminate the possibility of regression as possible.

========== original bug report ==========

[ 5.675835] iwlwifi 0000:00:14.3: enabling device (0000 -> 0002)
[ 5.693214] iwlwifi 0000:00:14.3: api flags index 2 larger than supported by driver
[ 5.693277] iwlwifi 0000:00:14.3: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.0.2.25
[ 5.698065] iwlwifi 0000:00:14.3: loaded firmware version 64.97bbee0a.0 so-a0-gf-a0-64.ucode op_mode iwlmvm
[ 5.933114] iwlwifi 0000:00:14.3: Detected Intel(R) Wi-Fi 6E AX211 160MHz, REV=0x370
[ 6.101350] iwlwifi 0000:00:14.3: loaded PNVM version 0x4b50f925
[ 6.113388] iwlwifi 0000:00:14.3: Detected RF GF, rfid=0x2010d000
[ 6.181309] iwlwifi 0000:00:14.3: base HW address: 7c:50:79:c9:29:67
[ 6.201398] iwlwifi 0000:00:14.3 wlp0s20f3: renamed from wlan0
[ 7.396811] iwlwifi 0000:00:14.3: SecBoot CPU1 Status: 0x71f7, CPU2 Status: 0xb03
[ 7.396859] iwlwifi 0000:00:14.3: UMAC PC: 0xc0081906
[ 7.396912] iwlwifi 0000:00:14.3: LMAC PC: 0x1541c
[ 7.396914] iwlwifi 0000:00:14.3: WRT: Collecting data: ini trigger 13 fired (delay=0ms).
[ 7.399116] iwlwifi 0000:00:14.3: Loaded firmware version: 64.97bbee0a.0 so-a0-gf-a0-64.ucode
[ 7.399119] iwlwifi 0000:00:14.3: 0x00000000 | ADVANCED_SYSASSERT
[ 7.399121] iwlwifi 0000:00:14.3: 0x00000000 | trm_hw_status0
[ 7.399123] iwlwifi 0000:00:14.3: 0x00000000 | trm_hw_status1
[ 7.399124] iwlwifi 0000:00:14.3: 0x00000000 | branchlink2
[ 7.399125] iwlwifi 0000:00:14.3: 0x00000000 | interruptlink1
[ 7.399127] iwlwifi 0000:00:14.3: 0x00000000 | interruptlink2
[ 7.399128] iwlwifi 0000:00:14.3: 0x00000000 | data1
[ 7.399129] iwlwifi 0000:00:14.3: 0x00000000 | data2
[ 7.399130] iwlwifi 0000:00:14.3: 0x00000000 | data3
[ 7.399132] iwlwifi 0000:00:14.3: 0x00000000 | beacon time
[ 7.399133] iwlwifi 0000:00:14.3: 0x00000000 | tsf low
[ 7.399134] iwlwifi 0000:00:14.3: 0x00000000 | tsf hi
[ 7.399135] iwlwifi 0000:00:14.3: 0x00000000 | time gp1
[ 7.399136] iwlwifi 0000:00:14.3: 0x00000000 | time gp2
[ 7.399137] iwlwifi 0000:00:14.3: 0x00000000 | uCode revision type
[ 7.399139] iwlwifi 0000:00:14.3: 0x00000000 | uCode version major
[ 7.399140] iwlwifi 0000:00:14.3: 0x00000000 | uCode version minor
[ 7.399141] iwlwifi 0000:00:14.3: 0x00000000 | hw version
[ 7.399143] iwlwifi 0000:00:14.3: 0x00000000 | board version
[ 7.399144] iwlwifi 0000:00:14.3: 0x00000000 | hcmd
[ 7.399145] iwlwifi 0000:00:14.3: 0x00000000 | isr0
[ 7.399146] iwlwifi 0000:00:14.3: 0x00000000 | isr1
[ 7.399147] iwlwifi 0000:00:14.3: 0x00000000 | isr2
[ 7.399148] iwlwifi 0000:00:14.3: 0x00000000 | isr3
[ 7.399149] iwlwifi 0000:00:14.3: 0x00000000 | isr4
[ 7.399150] iwlwifi 0000:00:14.3: 0x00000000 | last cmd Id
[ 7.399152] iwlwifi 0000:00:14.3: 0x00000000 | wait_event
[ 7.399153] iwlwifi 0000:00:14.3: 0x00000000 | l2p_control
[ 7.399154] iwlwifi 0000:00:14.3: 0x00000000 | l2p_duration
[ 7.399155] iwlwifi 0000:00:14.3: 0x00000000 | l2p_mhvalid
[ 7.399157] iwlwifi 0000:00:14.3: 0x00000000 | l2p_addr_match
[ 7.399158] iwlwifi 0000:00:14.3: 0x00000000 | lmpm_pmg_sel
[ 7.399159] iwlwifi 0000:00:14.3: 0x00000000 | timestamp
[ 7.399161] iwlwifi 0000:00:14.3: 0x00000000 | flow_handler
[ 7.399222] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
[ 7.399223] iwlwifi 0000:00:14.3: Transport status: 0x00000042, valid: 7
[ 7.399224] iwlwifi 0000:00:14.3: 0x20000066 | NMI_INTERRUPT_HOST
[ 7.399226] iwlwifi 0000:00:14.3: 0x00000000 | umac branchlink1
[ 7.399227] iwlwifi 0000:00:14.3: 0x8045DE0C | umac branchlink2
[ 7.399227] iwlwifi 0000:00:14.3: 0x80481126 | umac interruptlink1
[ 7.399228] iwlwifi 0000:00:14.3: 0xC00821AC | umac interruptlink2
[ 7.399229] iwlwifi 0000:00:14.3: 0x01000000 | umac data1
[ 7.399230] iwlwifi 0000:00:14.3: 0xC00821AC | umac data2
[ 7.399231] iwlwifi 0000:00:14.3: 0x00000000 | umac data3
[ 7.399231] iwlwifi 0000:00:14.3: 0x00000040 | umac major
[ 7.399232] iwlwifi 0000:00:14.3: 0x97BBEE0A | umac minor
[ 7.399233] iwlwifi 0000:00:14.3: 0x000F6F15 | frame pointer
[ 7.399234] iwlwifi 0000:00:14.3: 0xC0887F00 | stack pointer
[ 7.399234] iwlwifi 0000:00:14.3: 0x00000000 | last host cmd
[ 7.399235] iwlwifi 0000:00:14.3: 0x00000404 | isr status reg
[ 7.399274] iwlwifi 0000:00:14.3: IML/ROM dump:
[ 7.399275] iwlwifi 0000:00:14.3: 0x00000B03 | IML/ROM error/state
[ 7.399286] iwlwifi 0000:00:14.3: 0x000071F7 | IML/ROM data1
[ 7.399296] iwlwifi 0000:00:14.3: 0x00000080 | IML/ROM WFPM_AUTH_KEY_0
[ 7.399303] iwlwifi 0000:00:14.3: Fseq Registers:
[ 7.399306] iwlwifi 0000:00:14.3: 0x60000000 | FSEQ_ERROR_CODE
[ 7.399352] iwlwifi 0000:00:14.3: 0x803C0000 | FSEQ_TOP_INIT_VERSION
[ 7.399358] iwlwifi 0000:00:14.3: 0x00170000 | FSEQ_CNVIO_INIT_VERSION
[ 7.399404] iwlwifi 0000:00:14.3: 0x0000A652 | FSEQ_OTP_VERSION
[ 7.399410] iwlwifi 0000:00:14.3: 0x00000003 | FSEQ_TOP_CONTENT_VERSION
[ 7.399456] iwlwifi 0000:00:14.3: 0x4552414E | FSEQ_ALIVE_TOKEN
[ 7.399461] iwlwifi 0000:00:14.3: 0x00080400 | FSEQ_CNVI_ID
[ 7.399465] iwlwifi 0000:00:14.3: 0x00400410 | FSEQ_CNVR_ID
[ 7.399468] iwlwifi 0000:00:14.3: 0x00080400 | CNVI_AUX_MISC_CHIP
[ 7.399475] iwlwifi 0000:00:14.3: 0x00400410 | CNVR_AUX_MISC_CHIP
[ 7.399518] iwlwifi 0000:00:14.3: 0x00009061 | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
[ 7.399528] iwlwifi 0000:00:14.3: 0x00000061 | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
[ 7.399670] iwlwifi 0000:00:14.3: WRT: Collecting data: ini trigger 4 fired (delay=0ms).
[ 7.399677] iwlwifi 0000:00:14.3: Failed to start RT ucode: -110
[ 7.399678] iwlwifi 0000:00:14.3: Failed to start RT ucode: -110
[ 7.399679] iwlwifi 0000:00:14.3: WRT: Collecting data: ini trigger 13 fired (delay=0ms).
[ 8.560278] iwlwifi 0000:00:14.3: WRT: Failed to dump region: id=4, type=5
[ 8.560287] iwlwifi 0000:00:14.3: WRT: Failed to dump region: id=6, type=5
[ 8.560292] iwlwifi 0000:00:14.3: WRT: Failed to dump region: id=7, type=5
[ 9.820154] iwlwifi 0000:00:14.3: Failed to send recovery cmd blob was invalid 1
[ 9.852238] WARNING: CPU: 13 PID: 925 at drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c:258 iwl_mvm_mac_ctxt_init+0x1da/0x210 [iwlmvm]
[ 9.852317] cryptd intel_ipu6_isys rapl videobuf2_dma_contig snd_seq iwlwifi v4l2_fwnode dell_wmi v4l2_async ledtrig_audio intel_cstate snd_seq_device hid_sensor_custom_intel_hinge hid_sensor_rotation hid_sensor_gyro_3d dell_smbios videobuf2_memops hid_sensor_accel_3d hid_sensor_als snd_timer videobuf2_v4l2 dcdbas hid_sensor_trigger industrialio_triggered_buffer videobuf2_common kfifo_buf hid_sensor_iio_common input_leds videodev industrialio snd mei_me processor_thermal_device_pci mc dell_wmi_sysman(+) efi_pstore serio_raw processor_thermal_device cros_ec_ishtp dell_wmi_descriptor firmware_attributes_class wmi_bmof soundcore 8250_dw cfg80211 hid_multitouch mei processor_thermal_rfim cros_ec processor_thermal_mbox processor_thermal_rapl intel_rapl_common intel_ipu6 ucsi_acpi typec_ucsi igen6_edac typec int3403_thermal int340x_thermal_zone mac_hid int3400_thermal intel_skl_int3472 acpi_thermal_rel intel_hid power_ctrl_logic sparse_keymap acpi_tad acpi_pad sch_fq_codel msr parport_pc

[You-Sheng Yang (vicamo)]

For oem-5.14, the 1st proposed solution is to run backport-iwlwifi-dkms (impish 9340 should be sufficient to load -67 fw). Mark WONTFIX until further updates.

[You-Sheng Yang (vicamo)]

SRU: https://lists.ubuntu.com/archives/kernel-team/2022-March/128437.html (linux-firmware/focal)

[Andy Whitcroft (apw)]

Hello You-Sheng, or anyone else affected,

Accepted linux-firmware into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/1.187.28 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andy Whitcroft (apw)]

Hello You-Sheng, or anyone else affected,

Accepted linux-firmware into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/1.187.29 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Simonov (nsimonovv)]

I'm faced with almost the same issue:

$ uname -r
5.13.0-35-generic

Ubuntu 21.10

$ sudo dmesg | grep iwlwifi
[sudo] password for nsimonov:
[ 2.116670] Loading modules backported from iwlwifi
[ 2.116671] iwlwifi-stack-public:master:9340:55438a77
[ 2.170158] iwlwifi 0000:00:14.3: enabling device (0000 -> 0002)
[ 2.181954] iwlwifi 0000:00:14.3: api flags index 2 larger than supported by driver
[ 2.181976] iwlwifi 0000:00:14.3: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.0.2.34
[ 2.182323] iwlwifi 0000:00:14.3: loaded firmware version 67.8f59b80b.0 so-a0-gf-a0-67.ucode op_mode iwlmvm
[ 2.332128] iwlwifi 0000:00:14.3: Detected Intel(R) Wi-Fi 6E AX211 160MHz, REV=0x370
[ 2.338515] iwlwifi 0000:00:14.3: WRT: Failed to set DRAM buffer for alloc id 1, ret=-1
[ 2.338517] iwlwifi 0000:00:14.3: WRT: Failed to set DRAM buffer for alloc id 2, ret=-1
[ 2.338518] iwlwifi 0000:00:14.3: WRT: Failed to set DRAM buffer for alloc id 3, ret=-1
[ 2.486654] iwlwifi 0000:00:14.3: Detected RF GF, rfid=0x2010d000
[ 2.487991] iwlwifi 0000:00:14.3: Microcode SW error detected. Restarting 0x0.
[ 2.488131] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
[ 2.488133] iwlwifi 0000:00:14.3: Transport status: 0x0000004A, valid: 6
[ 2.488135] iwlwifi 0000:00:14.3: Loaded firmware version: 67.8f59b80b.0 so-a0-gf-a0-67.ucode
[ 2.488137] iwlwifi 0000:00:14.3: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL
[ 2.488139] iwlwifi 0000:00:14.3: 0x000002F0 | trm_hw_status0
[ 2.488140] iwlwifi 0000:00:14.3: 0x00000000 | trm_hw_status1
[ 2.488142] iwlwifi 0000:00:14.3: 0x004D9EF2 | branchlink2
[ 2.488143] iwlwifi 0000:00:14.3: 0x004D01BA | interruptlink1
[ 2.488144] iwlwifi 0000:00:14.3: 0x004D01BA | interruptlink2
[ 2.488145] iwlwifi 0000:00:14.3: 0x004D8BF2 | data1
[ 2.488146] iwlwifi 0000:00:14.3: 0x00000010 | data2
[ 2.488148] iwlwifi 0000:00:14.3: 0x00000000 | data3
[ 2.488149] iwlwifi 0000:00:14.3: 0x00000000 | beacon time
[ 2.488150] iwlwifi 0000:00:14.3: 0x00012764 | tsf low
[ 2.488151] iwlwifi 0000:00:14.3: 0x00000000 | tsf hi
[ 2.488152] iwlwifi 0000:00:14.3: 0x00000000 | time gp1
[ 2.488153] iwlwifi 0000:00:14.3: 0x0002413E | time gp2
[ 2.488154] iwlwifi 0000:00:14.3: 0x00000001 | uCode revision type
[ 2.488155] iwlwifi 0000:00:14.3: 0x00000043 | uCode version major
[ 2.488157] iwlwifi 0000:00:14.3: 0x8F59B80B | uCode version minor
[ 2.488158] iwlwifi 0000:00:14.3: 0x00000370 | hw version
[ 2.488159] iwlwifi 0000:00:14.3: 0x00C89002 | board version
[ 2.488160] iwlwifi 0000:00:14.3: 0x8007FF05 | hcmd
[ 2.488161] iwlwifi 0000:00:14.3: 0x00020000 | isr0
[ 2.488162] iwlwifi 0000:00:14.3: 0x60000000 | isr1
[ 2.488163] iwlwifi 0000:00:14.3: 0x48F00002 | isr2
[ 2.488164] iwlwifi 0000:00:14.3: 0x00C0000C | isr3
[ 2.488166] iwlwifi 0000:00:14.3: 0x00000000 | isr4
[ 2.488167] iwlwifi 0000:00:14.3: 0x00000000 | last cmd Id
[ 2.488168] iwlwifi 0000:00:14.3: 0x004D8BF2 | wait_event
[ 2.488169] iwlwifi 0000:00:14.3: 0x00000000 | l2p_control
[ 2.488170] iwlwifi 0000:00:14.3: 0x00000000 | l2p_duration
[ 2.488171] iwlwifi 0000:00:14.3: 0x00000000 | l2p_mhvalid
[ 2.4881...

[You-Sheng Yang (vicamo)]

verified linux-firmware/focal-proposed version 1.187.29.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 1.187.29

---------------
linux-firmware (1.187.29) focal; urgency=medium

  * mt7921e device disappear after warm boot stress (LP: #1963614)
    - linux-firmware: update firmware for MT7921 WiFi device

linux-firmware (1.187.28) focal; urgency=medium

  * bnx2x: Add FW 7.13.21.0 (LP: #1962155)
    - bnx2x: Add FW 7.13.20.0
    - bnx2x: Add FW 7.13.21.0
  * Add missing firmware for HWE 5.15 kernel (LP: #1962299)
    - linux-firmware: wilc1000: add wilc1000 v15.4 FW
    - linux-firmware: Update AMD SEV firmware
    - linux-firmware: Update AMD SEV firmware
    - linux-firmware: Update AMD SEV firmware
    - Mellanox: Add new mlxsw_spectrum firmware xx.2008.2406
    - amdgpu: add initial support for navy flounder
    - amdgpu: add arcturus firmware
    - rtl_nic: add new firmware for RTL8153 and RTL8156 series
    - amdgpu: update navy flounder firmware from 21.10
    - amdgpu: update arcturus firmware from 21.10
    - firmware: nvidia: Add VIC firmware for Tegra194
    - amdgpu: update navy flounder firmware from 21.20
    - amdgpu: update arcturus firmware from 21.20
    - firmware/i915/guc: Add GuC v62.0.0 for all platforms
    - firmware/i915/guc: Add GuC v62.0.3 for ADL-P
    - firmware/i915/guc: Add HuC v7.9.3 for TGL & DG1
    - iwlwifi: add ty firmware from Core63-43
    - i915: Add v2.12 DMC for TGL
    - i915: Add v2.03 DMC for RKL
    - amdgpu: update arcturus firmware from 21.30
    - amdgpu: update navy flounder firmware from 21.30
    - amdgpu: add initial vangogh support
    - rtl_nic: update firmware of RTL8153C
    - linux-firmware: Update AMD SEV firmware
    - amdgpu: update VCN firmware for vangogh
    - amdgpu: update VCN firmware for navy flounder
    - amdgpu: update arcturus firmware from 21.40
    - amdgpu: update navy flounder firmware from 21.40
    - amdgpu: add cyan skillfish firmware from 21.40
    - amdgpu: update vangogh firmware from 21.40
    - amdgpu: update vangogh DMCUB firmware
    - linux-firmware: wilc1000: update WILC1000 firmware to v15.4.1
    - iwlwifi: add new FWs from core63-136 release
    - amdgpu: Add aldebaran firmware from 21.50
    - amdgpu: Update arcturus firmware from 21.50
    - amdgpu: Update cyan skillfish2 firmware from 21.50
    - amdgpu: Update navy flounder firmware from 21.50
    - amdgpu: Update vangogh firmware from 21.50
    - WHENCE: link to similar config file for rtl8821a support
  * Intermittent AX211 iwlwifi RT ucode init failed -110 at boot (LP: #1962515)
    - iwlwifi: add new FWs from core64-96 release
  * mt7921e device disappear after warm boot stress (LP: #1963614)
    - linux-firmware: update firmware for mediatek bluetooth chip(MT7921)

 -- Juerg Haefliger <email address hidden> Thu, 10 Mar 2022 08:05:39 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for linux-firmware has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[You-Sheng Yang (vicamo)]

RFC: https://lists.ubuntu.com/archives/kernel-team/2022-April/129516.html (lrm/unstable)

[You-Sheng Yang (vicamo)]

We did not backport FW API nor embed backport-iwlwifi-dkms into kernel source tree. Instead, bug 1969434 takes a different method that enables dkms module builds along with the kernel source and a linux-modules-iwlwifi-<ABI> binary package and the corresponding meta package linux-modules-iwlwifi-<FLAVOR> are created to carry prebuilt iwlwifi dkms binary modules.

For people still have stability issues for AX211 on oem-5.14, hwe-5.15, and generic-5.15 kernels, please install linux-modules-iwlwifi-<ABI> or linux-modules-iwlwifi-<FLAVOR>.