occasional wifi firmware loading failures: wiwlwifi: BE200: Failed to start RT ucode: -110

[You-Sheng Yang]

[SRU Justification]

[Impact]

Within stress reboot tests, iwlwifi may fail to load firmware, triggers retries,
and finally makes wifi operative.

  iwlwifi 0000:05:00.0: Loaded firmware version: 83.d24e06ed.0 gl-c0-fm-c0-83.ucode
  iwlwifi 0000:05:00.0: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL
  iwlwifi 0000:05:00.0: 0x000002F0 | trm_hw_status0
  iwlwifi 0000:05:00.0: 0x00000000 | trm_hw_status1
  iwlwifi 0000:05:00.0: 0x002C27F2 | branchlink2
  iwlwifi 0000:05:00.0: 0x002B79AE | interruptlink1
......
  iwlwifi 0000:05:00.0: UMAC CURRENT PC: 0x8026934e
  iwlwifi 0000:05:00.0: LMAC1 CURRENT PC: 0xd0
  iwlwifi 0000:05:00.0: LMAC2 CURRENT PC: 0xd0
  iwlwifi 0000:05:00.0: UMAC CURRENT PC 1: 0x8026934c
  iwlwifi 0000:05:00.0: TCM1 CURRENT PC: 0xd0
  iwlwifi 0000:05:00.0: RCM1 CURRENT PC: 0xd0
  iwlwifi 0000:05:00.0: RCM2 CURRENT PC: 0xd0
  iwlwifi 0000:05:00.0: Function Scratch status:
  iwlwifi 0000:05:00.0: 0x01010101 | Func Scratch
  iwlwifi 0000:05:00.0: Failed to start RT ucode: -110
  iwlwifi 0000:05:00.0: WRT: Collecting data: ini trigger 13 fired (delay=0ms).

[Fix]

Resolved in upstream commit 318c64abfb33 ("iwlwifi: update gl FW for core80-165
release").

While the iwlwifi firmware updates come with a sequential manner, it pulls a few
more:

* 765492b8b ("iwlwifi: add new FWs from core81-65 release"), already in Mantic
* a07fd0b96 ("iwlwifi: add new FWs from core83-55 release")
* f27dec616 ("iwlwifi: fix for the new FWs from core83-55 release")

These contain firmware blob for ucore 84/86, and are not used by kernels < v6.7.

[Test Case]

Use checkbox reboot stress test case to observe iwlwifi dmesg logs:

  $ checkbox-cli run com.canonical.certification::stress/reboot

[Where problems could occur]

Opaque firmware update. No known issue found yet after the update.

[Other Info]

This affects series with kernels that load iwlwifi ucode 83 firmware, namely
Noble, Mantic, and for oem-6.5, Jammy.

========== original bug report ==========

Within stress reboot tests, iwlwifi may fail to load firmware and triggers retry, and finally makes wifi operative.

Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: Loaded firmware version: 83.d24e06ed.0 gl-c0-fm-c0-83.ucode
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: 0x00000071 | NMI_INTERRUPT_UMAC_FATAL
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: 0x000002F0 | trm_hw_status0
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: 0x00000000 | trm_hw_status1
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: 0x002C27F2 | branchlink2
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: 0x002B79AE | interruptlink1
......
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: UMAC CURRENT PC: 0x8026934e
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: LMAC1 CURRENT PC: 0xd0
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: LMAC2 CURRENT PC: 0xd0
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: UMAC CURRENT PC 1: 0x8026934c
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: TCM1 CURRENT PC: 0xd0
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: RCM1 CURRENT PC: 0xd0
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: RCM2 CURRENT PC: 0xd0
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: Function Scratch status:
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: 0x01010101 | Func Scratch
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: Failed to start RT ucode: -110
Oct 12 00:16:21 32090 kernel: iwlwifi 0000:05:00.0: WRT: Collecting data: ini trigger 13 fired (delay=0ms).

Test steps:

$ checkbox-cli run com.canonical.certification::stress/reboot

Proposed fix:

* https://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/linux-firmware.git/commit/?id=318c64abfb33c4211b002f5e5b4ff25cd19f712b

[You-Sheng Yang (vicamo)]

Commit 318c64abf touches PNVM as well:

$ git log --oneline --stat 318c64abf
318c64abf (tag: iwlwifi-fw-2023-12-21, korg-iwlwifi/for-upstream) iwlwifi: update gl FW for core80-165 release
 WHENCE | 2 +-
 iwlwifi-gl-c0-fm-c0-83.ucode | Bin 1799504 -> 1797996 bytes
 iwlwifi-gl-c0-fm-c0.pnvm | Bin 226668 -> 246228 bytes
 3 files changed, 1 insertion(+), 1 deletion(-)

Which then pulls two more commits due the the modification chain of the PNVM file:

$ git log --oneline --stat f27dec616
f27dec616 (tag: iwlwifi-fw-2023-11-16-fixed) iwlwifi: fix for the new FWs from core83-55 release
 iwlwifi-gl-c0-fm-c0-86.ucode | Bin 1840116 -> 1840116 bytes
 iwlwifi-gl-c0-fm-c0.pnvm | Bin 226668 -> 226668 bytes
 iwlwifi-ma-b0-gf-a0-86.ucode | Bin 1695788 -> 1695788 bytes
 iwlwifi-ma-b0-gf-a0.pnvm | Bin 55092 -> 55092 bytes
 iwlwifi-ma-b0-gf4-a0-86.ucode | Bin 1670940 -> 1670940 bytes
 iwlwifi-ma-b0-gf4-a0.pnvm | Bin 27820 -> 27820 bytes
 iwlwifi-ma-b0-hr-b0-86.ucode | Bin 1520636 -> 1520636 bytes
 iwlwifi-so-a0-gf-a0-86.ucode | Bin 1679788 -> 1679788 bytes
 iwlwifi-so-a0-gf-a0.pnvm | Bin 55172 -> 55172 bytes
 iwlwifi-so-a0-gf4-a0-86.ucode | Bin 1653836 -> 1653836 bytes
 iwlwifi-so-a0-gf4-a0.pnvm | Bin 27860 -> 27860 bytes
 iwlwifi-so-a0-hr-b0-86.ucode | Bin 1507388 -> 1507388 bytes
 iwlwifi-ty-a0-gf-a0-86.ucode | Bin 1632464 -> 1632464 bytes
 iwlwifi-ty-a0-gf-a0.pnvm | Bin 55020 -> 55020 bytes
 14 files changed, 0 insertions(+), 0 deletions(-)

$ git log --oneline --stat a07fd0b96
a07fd0b96 (tag: iwlwifi-fw-2023-11-15) iwlwifi: add new FWs from core83-55 release
 WHENCE | 24 ++++++++++++++++++++++++
 iwlwifi-gl-c0-fm-c0-86.ucode | Bin 0 -> 1840116 bytes
 iwlwifi-gl-c0-fm-c0.pnvm | Bin 246228 -> 226668 bytes
 iwlwifi-ma-b0-gf-a0-86.ucode | Bin 0 -> 1695788 bytes
 iwlwifi-ma-b0-gf-a0.pnvm | Bin 55500 -> 55092 bytes
 iwlwifi-ma-b0-gf4-a0-86.ucode | Bin 0 -> 1670940 bytes
 iwlwifi-ma-b0-gf4-a0.pnvm | Bin 28024 -> 27820 bytes
 iwlwifi-ma-b0-hr-b0-86.ucode | Bin 0 -> 1520636 bytes
 iwlwifi-so-a0-gf-a0-86.ucode | Bin 0 -> 1679788 bytes
 iwlwifi-so-a0-gf-a0.pnvm | Bin 55580 -> 55172 bytes
 iwlwifi-so-a0-gf4-a0-86.ucode | Bin 0 -> 1653836 bytes
 iwlwifi-so-a0-gf4-a0.pnvm | Bin 28064 -> 27860 bytes
 iwlwifi-so-a0-hr-b0-86.ucode | Bin 0 -> 1507388 bytes
 iwlwifi-ty-a0-gf-a0-86.ucode | Bin 0 -> 1632464 bytes
 iwlwifi-ty-a0-gf-a0.pnvm | Bin 55484 -> 55020 bytes
 15 files changed, 24 insertions(+)

While the two commits are for ucode 86 releases that is currently not supported by all available kernels in Jammy, there is not hard/effect that may be introduced by pulling both of them.

[You-Sheng Yang (vicamo)]

And for Jammy, it takes also:

$ git show --stat 765492b8b90e3985fe60b3cf8d386538aa38b6ac
commit 765492b8b90e3985fe60b3cf8d386538aa38b6ac (tag: iwlwifi-fw-2023-09-21)
Author: Gregory Greenman <email address hidden>
Date: Thu Sep 21 17:02:12 2023 +0300

    iwlwifi: add new FWs from core81-65 release

    Add the -84 ucode firmwares for the currently supported hardware and
    update the PNVM binaries.

    Build number: Core_build_core81-65
    Revision: 8e5c544c (so, ty)

    Signed-off-by: Gregory Greenman <email address hidden>

 WHENCE | 12 ++++++++++++
 iwlwifi-so-a0-gf-a0-84.ucode | Bin 0 -> 1737644 bytes
 iwlwifi-so-a0-gf-a0.pnvm | Bin 55580 -> 55580 bytes
 iwlwifi-so-a0-gf4-a0-84.ucode | Bin 0 -> 1709900 bytes
 iwlwifi-so-a0-gf4-a0.pnvm | Bin 28064 -> 28064 bytes
 iwlwifi-so-a0-hr-b0-84.ucode | Bin 0 -> 1549088 bytes
 iwlwifi-ty-a0-gf-a0-84.ucode | Bin 0 -> 1676912 bytes
 iwlwifi-ty-a0-gf-a0.pnvm | Bin 55484 -> 55484 bytes
 8 files changed, 12 insertions(+)

[You-Sheng Yang (vicamo)]

This depends on bug 2028065, and pull request for linux-firmware/noble has not yet been merged at this moment.

[You-Sheng Yang (vicamo)]

SRU:
* https://lists.ubuntu.com/archives/kernel-team/2024-January/148030.html (jammy)
* https://lists.ubuntu.com/archives/kernel-team/2024-January/148030.html (mantic)

[Timo Aaltonen (tjaalton)]

Hello You-Sheng, or anyone else affected,

Accepted linux-firmware into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/20230919.git3672ccab-0ubuntu2.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello You-Sheng, or anyone else affected,

Accepted linux-firmware into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/20220329.git681281e4-0ubuntu3.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[You-Sheng Yang (vicamo)]

verified l-f/jammy-proposed version 20220329.git681281e4-0ubuntu3.25, mantic-proposed version 20230919.git3672ccab-0ubuntu2.6.

[Timo Aaltonen (tjaalton)]

Hello You-Sheng, or anyone else affected,

Accepted linux-firmware into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/20220329.git681281e4-0ubuntu3.26 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 20220329.git681281e4-0ubuntu3.26

---------------
linux-firmware (20220329.git681281e4-0ubuntu3.26) jammy; urgency=medium

  * Miscellaneous Ubuntu changes
    - [Workflow] Add initial gitea workflow files
    - SAUCE: [Workflow] check_whence.py: Update list of known files
  * Revert "WCN6856 Wi-FI Unavailable and no function during suspend stress (LP: #2048977)"
    - Revert "ath11k: WCN6855 hw2.0: update to WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.36"
    - Revert "ath11k: WCN6855 hw2.0: update board-2.bin"
    - Revert "ath11k: WCN6855 hw2.0: update board-2.bin"

linux-firmware (20220329.git681281e4-0ubuntu3.25) jammy; urgency=medium

  * Dell dock station WD22TB4 loses ether dev after reboot/power off (LP: #2048747)
    - rtl_nic: update firmware of USB devices
  * occasional wifi firmware loading failures: wiwlwifi: BE200: Failed to start RT ucode: -110 (LP: #2048853)
    - iwlwifi: add new FWs from core81-65 release
    - iwlwifi: add new FWs from core83-55 release
    - iwlwifi: fix for the new FWs from core83-55 release
    - iwlwifi: update gl FW for core80-165 release
  * WCN6856 Wi-FI Unavailable and no function during suspend stress (LP: #2048977)
    - ath11k: WCN6855 hw2.0: update board-2.bin
    - ath11k: WCN6855 hw2.0: update board-2.bin
    - ath11k: WCN6855 hw2.0: update to WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.36
  * Support Mediatek MT7925 WiFi/BT (LP: #2043542)
    - linux-firmware: add firmware for MT7925
    - linux-firmware: add firmware for mediatek bluetooth chip (MT7925)
  * upstream Intel VSC firmware paths moved to intel/vsc/ (LP: #2048991)
    - intel: vsc: Add firmware for Visual Sensing Controller
    - SAUCE: intel: vsc: use upstream binaries

 -- Juerg Haefliger <email address hidden> Tue, 30 Jan 2024 11:04:32 +0100

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for linux-firmware has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Timo Aaltonen (tjaalton)]

Hello You-Sheng, or anyone else affected,

Accepted linux-firmware into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/20230919.git3672ccab-0ubuntu2.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[You-Sheng Yang (vicamo)]

linux-firmware/noble version 20240202.git36777504-0ubuntu1 has all the required commits already.

[You-Sheng Yang (vicamo)]

verified linux-firmware/mantic-proposed version 20230919.git3672ccab-0ubuntu2.8.

[Timo Aaltonen (tjaalton)]

Hello You-Sheng, or anyone else affected,

Accepted linux-firmware into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/20230919.git3672ccab-0ubuntu2.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[You-Sheng Yang (vicamo)]

Changes have been landed under tag 20230919.git3672ccab-0ubuntu2.6, and verified in 20230919.git3672ccab-0ubuntu2.8, which is lately respined as 20230919.git3672ccab-0ubuntu2.9 with no additional change made.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 20230919.git3672ccab-0ubuntu2.9

---------------
linux-firmware (20230919.git3672ccab-0ubuntu2.9) mantic; urgency=medium

  * Update firmware for MT7921 in order to fix Framework 13 AMD 7040 (LP: #2049220)
    - linux-firmware: update firmware for MT7922 WiFi device
    - linux-firmware: update firmware for mediatek bluetooth chip (MT7922)

linux-firmware (20230919.git3672ccab-0ubuntu2.8) mantic; urgency=medium

  * DP connection swap to break eDP behavior on AMD 7735U (LP: #2049758)
    - SAUCE: Update DCN312 DMCUB firmware

linux-firmware (20230919.git3672ccab-0ubuntu2.7) mantic; urgency=medium

  * Miscellaneous Ubuntu changes
    - [Packaging] scripts: Fix shellcheck warnings
    - [Workflow] Add initial gitea workflow file
    - SAUCE: [Workflow] Disable markdownlint pre-commit hook
    - SAUCE: [Workflow] check_whence.py: Update list of known files
    - [Packaging] scripts/generate-changelog: Fix array initialization
    - [Packaging] control: Add XSBC-Original-Maintainer field
    - [Packaging] scripts/install-firmware: Fix installation of license files
  * AMD phoenix/phoenix2 platforms facing amdgpu(PHX) hangs during stress loading (LP: #2051636)
    - amdgpu: update PSP 13.0.4 firmware from 5.7 branch
    - amdgpu: update GC 11.0.1 firmware from 5.7 branch
    - amdgpu: update GC 11.0.4 firmware from 5.7 branch
    - amdgpu: update PSP 13.0.11 firmware from 5.7 branch
    - amdgpu: update GC 11.0.1 firmware
    - amdgpu: update PSP 13.0.4 firmware
    - amdgpu: update VCN 4.0.2 firmware
    - amdgpu: update GC 11.0.4 firmware
    - amdgpu: update PSP 13.0.11 firmware
  * Update firmware for MT7921 in order to fix Framework 13 AMD 7040 (LP: #2049220)
    - linux-firmware: update firmware for MT7921 WiFi device
    - linux-firmware: update firmware for mediatek bluetooth chip (MT7921)
  * WCN6856 Wi-FI Unavailable and no function during suspend stress (LP: #2048977)
    - ath11k: WCN6855 hw2.0: update to WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.37

linux-firmware (20230919.git3672ccab-0ubuntu2.6) mantic; urgency=medium

  * occasional wifi firmware loading failures: wiwlwifi: BE200: Failed to start RT ucode: -110 (LP: #2048853)
    - iwlwifi: add new FWs from core83-55 release
    - iwlwifi: fix for the new FWs from core83-55 release
    - iwlwifi: update gl FW for core80-165 release
  * WCN6856 Wi-FI Unavailable and no function during suspend stress (LP: #2048977)
    - ath11k: WCN6855 hw2.0: update board-2.bin
    - ath11k: WCN6855 hw2.0: update to WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.36

 -- Juerg Haefliger <email address hidden> Wed, 21 Feb 2024 10:41:18 +0100