Intel AX210 Wi-Fi card does not work since updating to 1.196

[Alberto Abrao]

Running 21.04, updated daily.

Since updating the linux-firmware package yesterday, the Wireless card does not work with current kernel (5.11.0-13).

The error message is as follows:

Apr 02 21:15:18 Lenovo-V15-IIL kernel: iwlwifi 0000:01:00.0: Timeout waiting for PNVM load!
Apr 02 21:15:18 Lenovo-V15-IIL kernel: iwlwifi 0000:01:00.0: Failed to start RT ucode: -110
Apr 02 21:15:18 Lenovo-V15-IIL kernel: iwlwifi 0000:01:00.0: iwl_trans_send_cmd bad state = 1

It **does** work with 5.11.0-11 (previous version)

Distro: Ubuntu 21.04 (Hirsute Hippo)
Machine: Type: Laptop System: LENOVO product: 82C5 v: Lenovo V15-IIL Mobo: LENOVO model: LNVNB161216 v: SDK0J40700 WIN UEFI: LENOVO v: DKCN51WW date: 12/23/2020
CPU: Info: Dual Core model: Intel Core i3-1005G1 bits: 64 type: MT MCP L2 cache: 4 MiB

[daboross (daboross-linux-kernel-bugs)]

Created attachment 295967
BROKEN dmesg log for 5.11.7

kernel 5.11.7-200.fc33.x86_64+debug

Likely related to my running an Intel AX210 chip.

iwlwifi detects the AX210 chip, but it errors out while loading, and NetworkManager does not find the device.

This is broken on at least 5.11.3-300.fc34.x86_64, 5.11.7-200.fc33.x86_64+debug and 5.12.0-0.rc3.20210319git8b12a62a4e3e.172.fc35.x86_64.

This works (the bug does NOT occur) on 5.11.2-300.fc34.x86_64.

Related arch linux bug at https://bugs.archlinux.org/task/70071. I'm submitting this here because I'm on Fedora, and thus I believe this is a cross-system kernel issue.

[ 11.466042] iwlwifi 0000:04:00.0: enabling device (0000 -> 0002)
[ 11.513767] iwlwifi 0000:04:00.0: api flags index 2 larger than supported by driver
[ 11.513897] iwlwifi 0000:04:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[ 11.515808] iwlwifi 0000:04:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[ 11.724017] iwlwifi 0000:04:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[ 11.912643] iwlwifi 0000:04:00.0: loaded PNVM version 0x324cd670
[ 12.014922] iwlwifi 0000:04:00.0: Timeout waiting for PNVM load!
[ 12.014931] iwlwifi 0000:04:00.0: Failed to start RT ucode: -110
[ 12.014941] iwlwifi 0000:04:00.0: iwl_trans_send_cmd bad state = 1
[ 12.222898] iwlwifi 0000:04:00.0: firmware didn't ACK the reset - continue anyway
[ 12.236033] iwlwifi 0000:04:00.0: Failed to run INIT ucode: -110

Full dmesg logs for 5.12.0, 5.11.3 (BROKEN) and 5.11.2 (WORKING) attached.

[daboross (daboross-linux-kernel-bugs)]

Created attachment 295969
BROKEN dmesg log for 5.12.0-0.rc3

[daboross (daboross-linux-kernel-bugs)]

Created attachment 295971
WORKING dmesg log for 5.11.2

[daboross (daboross-linux-kernel-bugs)]

Created attachment 295973
BROKEN dmesg log for 5.11.3

[daboross (daboross-linux-kernel-bugs)]

Created attachment 295975
lspci -vvnn

[daboross (daboross-linux-kernel-bugs)]

May be related to https://bugzilla.kernel.org/show_bug.cgi?id=211165. Same error message, but that bug has only 2/100 reproducibility, while this bug is 100% reproducible on my system.

[lordheavym (lordheavym-linux-kernel-bugs)]

Archlinux bug report: https://bugs.archlinux.org/task/70071

[pbrobinson (pbrobinson-linux-kernel-bugs)]

Fedora bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1940263

[cuebix (cuebix-linux-kernel-bugs)]

I have the same issue on a brand new system that came with the AX210 and haven't been able to get it to work at all. I get pretty much the same error messages:

➜ ~ sudo dmesg | grep iwlwifi
[ 5.300620] iwlwifi 0000:03:00.0: enabling device (0000 -> 0002)
[ 5.315109] iwlwifi 0000:03:00.0: api flags index 2 larger than supported by driver
[ 5.315125] iwlwifi 0000:03:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[ 5.315357] iwlwifi 0000:03:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[ 5.649887] iwlwifi 0000:03:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[ 5.819781] iwlwifi 0000:03:00.0: loaded PNVM version 0x324cd670
[ 5.922953] iwlwifi 0000:03:00.0: Timeout waiting for PNVM load!
[ 5.922956] iwlwifi 0000:03:00.0: Failed to start RT ucode: -110
[ 5.922959] iwlwifi 0000:03:00.0: iwl_trans_send_cmd bad state = 1
[ 6.126286] iwlwifi 0000:03:00.0: firmware didn't ACK the reset - continue anyway
[ 6.138669] iwlwifi 0000:03:00.0: Failed to run INIT ucode: -110

[cuebix (cuebix-linux-kernel-bugs)]

Oh, I almost forgot, here's the output of my uname -a:

➜ ~ uname -a
Linux gladiator 5.11.7-arch1-1 #1 SMP PREEMPT Wed, 17 Mar 2021 16:59:58 +0000 x86_64 GNU/Linux

[daboross (daboross-linux-kernel-bugs)]

Dannel Albert, if you're able to find a way to downgrade your kernel version to something that's at least 5.11.0 (the version which added support for the AX210), but older than 5.11.3 (the version which broke it), then I would recommend trying that.

Since it seems like you're on arch, maybe you could download linux-5.11.2 from https://archive.archlinux.org/packages/l/linux/ and follow https://wiki.archlinux.org/index.php/Arch_Linux_Archive#How_to_downgrade_one_package to downgrade?

Or maybe it'd be better to follow what the person making the arch report (linked above) said, and downgrade linux-firmware to 20210208.b79d239-1 from https://archive.archlinux.org/packages/l/linux-firmware/.

I'm not an arch linux user myself, but downgrading the kernel in fedora by manually downloading & installing older kernel packages was a good temporary fix for me.

I'm not sure what the etiquette for providing help/workarounds on kernel bug reports is, so I apologize if this post is out of line and/or spamming everyone :)

[richrocksmyworld (richrocksmyworld-linux-kernel-bugs)]

Same issue with AX210 and 5.11.6 for me. Workaround by downgrading firmware to previous also worked.

[cuebix (cuebix-linux-kernel-bugs)]

Created attachment 296011
attachment-4622-0.html

Thanks for the suggestion. Those are good ideas, but I actually anticipated having issues when I bought this laptop and got an AX200 just in case, so when it didn't work, I just swapped that in.

I'll probably keep that in until I hear reports that this is working and/or there is a fix that needs to be tested and I can help contribute.
On Mar 23, 2021, 1:38 AM -0400, <email address hidden>, wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=212371
>
> --- Comment #10 from David Ross (<email address hidden>) ---
> Dannel Albert, if you're able to find a way to downgrade your kernel version
> to
> something that's at least 5.11.0 (the version which added support for the
> AX210), but older than 5.11.3 (the version which broke it), then I would
> recommend trying that.
>
> Since it seems like you're on arch, maybe you could download linux-5.11.2
> from
> https://archive.archlinux.org/packages/l/linux/ and follow
>
> https://wiki.archlinux.org/index.php/Arch_Linux_Archive#How_to_downgrade_one_package
> to downgrade?
>
> Or maybe it'd be better to follow what the person making the arch report
> (linked above) said, and downgrade linux-firmware to 20210208.b79d239-1 from
> https://archive.archlinux.org/packages/l/linux-firmware/.
>
> I'm not an arch linux user myself, but downgrading the kernel in fedora by
> manually downloading & installing older kernel packages was a good temporary
> fix for me.
>
> I'm not sure what the etiquette for providing help/workarounds on kernel bug
> reports is, so I apologize if this post is out of line and/or spamming
> everyone
> :)
>
> --
> You may reply to this email to add a comment.
>
> You are receiving this mail because:
> You are on the CC list for the bug.

[paulius.zaleckas (paulius.zaleckas-linux-kernel-bugs)]

Another simple workaround is to remove/rename /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm file.
Apparently currently used firmware is not compatible with this PNVM:
https://bugs.gentoo.org/777324#c6

[daboross (daboross-linux-kernel-bugs)]

(In reply to Paulius Zaleckas from comment #13)
> Another simple workaround is to remove/rename
> /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm file.
> Apparently currently used firmware is not compatible with this PNVM:
> https://bugs.gentoo.org/777324#c6

Thank you! This workaround works for me.

I've confirmed that renaming that file and rebooting with 5.11.8 gets me working wifi. Additionally, renaming it back to the original name and rebooting breaks wifi again (and with the same kernel error messages, if I'm reading them correctly).

[cuebix (cuebix-linux-kernel-bugs)]

I swapped my AX210 back into my system and removing the pnvm file mentioned in the workaround works for me too!

Thanks!

[luca (luca-linux-kernel-bugs)]

Unfortunately we can't just backport the FW version bump. There are
actual API changes that need to be supported, so that would require an
unfeasible amount of changes for a stable tree.

We have two options:

1. Try to take a fix to the -59.ucode firmware (I'll take this
internally)

2. Prevent the driver from loading the PNVM file when older firmwares
are used (I'll check if there is a proper way to do this without
relying on the actual version number).

I'll take this forward.

[cuebix (cuebix-linux-kernel-bugs)]

Created attachment 296073
attachment-13323-0.html

Thank you for handling this Luca!
On Mar 26, 2021, 2:49 AM -0400, <email address hidden>, wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=212371
>
> Luca Coelho (<email address hidden>) changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> Status|NEW |ASSIGNED
> CC| |<email address hidden>
>
> --- Comment #16 from Luca Coelho (<email address hidden>) ---
> Unfortunately we can't just backport the FW version bump. There are
> actual API changes that need to be supported, so that would require an
> unfeasible amount of changes for a stable tree.
>
> We have two options:
>
> 1. Try to take a fix to the -59.ucode firmware (I'll take this
> internally)
>
> 2. Prevent the driver from loading the PNVM file when older firmwares
> are used (I'll check if there is a proper way to do this without
> relying on the actual version number).
>
> I'll take this forward.
>
> --
> You may reply to this email to add a comment.
>
> You are receiving this mail because:
> You are on the CC list for the bug.

[Alberto Abrao (albertoabrao)]

Renaming file iwlwifi-ty-a0-gf-a0.pnvm to iwlwifi-ty-a0-gf-a0.pnvm.old located on /lib/firmware/ is enough to make it work as-is on current kernel.

If there are any tests needed to help troubleshoot this for good, please let me know.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1922406

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Seth Forshee (sforshee)]

Looks like 5.11 isn't able to use the new firmware versions added alongside this file anyway, so I'll get a new version uploaded which removes all the version 62 API files along with iwlwifi-ty-a0-gf-a0.pnvm.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 1.197

---------------
linux-firmware (1.197) hirsute; urgency=medium

  * Intel AX210 Wi-Fi card does not work since updating to 1.196
    (LP: #1922406)
    - SAUCE: Revert "iwlwifi: add new FWs from core59-66 release"

 -- Seth Forshee <email address hidden> Wed, 07 Apr 2021 08:50:11 -0500

[Alberto Abrao (albertoabrao)]

I can confirm it is working fine after the update:

- Restored original name for the file previously renamed.
- Installed linux-firmware update.
- Rebooted.
- Wi-Fi test: OK.

[tsangares (tsangares-linux-kernel-bugs)]

*** Bug 213683 has been marked as a duplicate of this bug. ***

[tsangares (tsangares-linux-kernel-bugs)]

Hey is there any updates?

This is still an issue for me, I cannot update my linux kernel.

[doug.coleman (doug.coleman-linux-kernel-bugs)]

A solution to the AX210 pnvm error that worked for me on Ubuntu 20.04.3 kernel 5.11.0-27-generic with an AX210 is to delete /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm

I found this at https://www.reddit.com/r/Fedora/comments/m9p2h2/ax210_drivers/

[kernel.org (kernel.org-linux-kernel-bugs)]

I also had to delete the /lib/firmware/iwlwifi-so-a0-gf-a0.pnvm file even though I have a `ty` aka AX210 otherwise it was causing the card to loop and not actually connect on 5.10.

[niclas (niclas-linux-kernel-bugs)]

GUys,

I have a Samsung laptop with the AX210, so also the ty.

I've been running Debian on the laptop on stable, but decided to do a complete new fresh install with the latest testing variant instead.

Everything works except iwlwifi has this .pvnm file and if..

.. I remove it, wifi stops working and bluetooth starts fine
.. I put it back, bluetooth stops working and wifi starts fine

Well. That means I cant have both working at the same time. Any suggestions as to where to begin? Can I somehow make either bluetooth accept pvnm or have wifi ignore it?

Regards,

Niclas

[tsangares (tsangares-linux-kernel-bugs)]

I have been remaining in a 5.11.10 this whole time to make sure my wifi works. Now I need some features in later kernal versions although this still seems to be a persistent issue.

Is there any progress being made on this iwlwifi module loading issue?

[niclas (niclas-linux-kernel-bugs)]

For me, I'm on Debian testing (bookworm) with the latest debian-released kernel 5.15.0, I have no problems anymore. I did'nt have problem with 5.13 either actually.

And the mentioned problem from me above, about not having both bluetooth and the wifi side of the chip working at the same time, has also gone away.

So which kernel are you considering? The fix of removing the pnvm-file in /lib/firmware works without problem if you really need to move up to a higher revision of the kernel I guess.

//Niclas

[janpieter.sollie (janpieter.sollie-linux-kernel-bugs)]

I have the same issue on a mips24kc machine with AX210 card. I'm using openwrt snapshots, which currently uses 5.15-rc6 backport drivers. I generated an additonal patch to update drivers/.../iwlwifi to 5.15.5, but it didn't help.
Tried with all currently released firmwares in linux-firmware tree, 59-66, and a few pnvm files visible though the commit log, none of them works.

[raymond (raymond-linux-kernel-bugs)]

I had similar problem on a gaming PC, I'm on Ubuntu 5.11.0-46-generic kernel. I renamed pnvm file, rebooted and it worked for me. Bluetooth also seems to be working for me. Any idea when will it be fixed?

[doug.coleman (doug.coleman-linux-kernel-bugs)]

I updated Ubuntu and got a broken wifi driver again. I got fed up with old kernels (5.11, 5.13) and installed 5.16 and it fixed the wifi and a screen-tearing issue that was driving me mad!

sudo add-apt-repository ppa:tuxinvader/lts-mainline -y
sudo apt update
sudo apt-get install linux-generic-5.16 -y

[raymond (raymond-linux-kernel-bugs)]

(In reply to Doug Coleman from comment #27)
> I updated Ubuntu and got a broken wifi driver again. I got fed up with old
> kernels (5.11, 5.13) and installed 5.16 and it fixed the wifi and a
> screen-tearing issue that was driving me mad!
>
> sudo add-apt-repository ppa:tuxinvader/lts-mainline -y
> sudo apt update
> sudo apt-get install linux-generic-5.16 -y

Yes, it fixed for me too when I upgraded to 5.16 but but..
5.16 isn't official kernel?

[kernel-org.myq (kernel-org.myq-linux-kernel-bugs)]

I'm getting the same error but with the recent kernel version 5.16.9, and the workaround of renaming the .pnvm file does not work. I can see from the dmesg log that it tries loading the -67.ucode firmware (which gives the same timeout error reported by others; see below), and I've tried renaming every version after -59 as well as the .pnvm file to force it to use the -59 firmware (which is the version Intel provides on their website), but I still get the timeout error in dmesg:

> Timeout waiting for PNVM load!
> Failed to start RT ucode: -110

Note: I *did* get this working by downgrading the kernel to 5.11 and renaming the .pnvm file, but I don't want to be stuck on such an old kernel. I was expecting everything to "just work" after I upgraded to the 5.16 kernel.

[paulius.zaleckas (paulius.zaleckas-linux-kernel-bugs)]

(In reply to pepito from comment #29)
> Note: I *did* get this working by downgrading the kernel to 5.11 and
> renaming the .pnvm file, but I don't want to be stuck on such an old kernel.
> I was expecting everything to "just work" after I upgraded to the 5.16
> kernel.

Currently I am using 5.14.18 and it does not have this problem. 5.11 had it.
Later if I upgrade to 5.16, I will let you know if it happens again.

[raymond (raymond-linux-kernel-bugs)]

Renaming/deleting .pvnm file solution stopped working for me after updating to 20.04.

Does anyone know how to fix it now?

[raymond (raymond-linux-kernel-bugs)]

(In reply to Raymond from comment #31)
> Renaming/deleting .pvnm file solution stopped working for me after updating
> to 20.04.
>
> Does anyone know how to fix it now?

Kernel 5.13.0-39-generic

[daboross (daboross-linux-kernel-bugs)]

This no longer occurs for me on Fedora 35 with Kernel version 5.16.5-200.fc35.x86_64. I'm going to unsubscribe but leave this open, as I don't know where the fix was, and don't know the proper procedure for figuring out if this is both fixed & if the kernel has a test.

[ccdrecc (ccdrecc-linux-kernel-bugs)]

I had this problem, even after upgrading to 5.19.0-23-generic.

The answer turned out to be that I had to remove "pci=nomsi,noaer" which was in my /etc/default/grub file.

[samwinchester19999 (samwinchester19999-linux-kernel-bugs)]

Created attachment 303914
app

[eddik98 (eddik98-linux-kernel-bugs)]

Hello, I have the same problem, with a lot of linux-firmware versions.
I have found this issue also with linux-firmware-20220329.git681281e4-0ubuntu3.12.

After upgrading linux-firmware lib, I always have to use this command `sudo mv /usr/lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm /usr/lib/firmware/iwlwifi-ty-a0-gf-a0.bak` (https://askubuntu.com/questions/1360175/intel-wifi-6-ax210-wifi-not-working-after-update), and after this command, Wi-Fi module works.

I have the same logs as other users have above.

- linux-firmware-20220329.git681281e4-0ubuntu3.12
- Linux version 5.10.110-rockchip-rk3588 (aarch64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1 SMP Wed Mar 29 08:28:12 BST 2023

00:34:55 kernel: iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
00:34:55 kernel: iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
00:34:55 kernel: iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
00:34:54 kernel: iwlwifi 0004:41:00.0: loaded PNVM version 0x881c99e1

[eddik98 (eddik98-linux-kernel-bugs)]

Additional logs:

- Before executing above mv command:
[ 18.502463] Intel(R) Wireless WiFi driver for Linux
[ 18.502891] iwlwifi 0004:41:00.0: enabling device (0000 -> 0002)
[ 18.543590] iwlwifi 0004:41:00.0: api flags index 2 larger than supported by driver
[ 18.543628] iwlwifi 0004:41:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[ 18.544154] iwlwifi 0004:41:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[ 18.552252] iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[ 18.631147] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 18.728168] iwlwifi 0004:41:00.0: loaded PNVM version 0x881c99e1
[ 19.001708] iwlwifi 0004:41:00.0: Timeout waiting for PNVM load!
[ 19.001730] iwlwifi 0004:41:00.0: Failed to start RT ucode: -110
[ 19.001740] iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
[ 19.014020] iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
[ 19.026420] iwlwifi 0004:41:00.0: retry init count 0
[ 19.028901] iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[ 19.081044] Process accounting resumed
[ 19.479068] iwlwifi 0004:41:00.0: Timeout waiting for PNVM load!
[ 19.479114] iwlwifi 0004:41:00.0: Failed to start RT ucode: -110
[ 19.479137] iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
[ 19.491411] iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
[ 19.504714] iwlwifi 0004:41:00.0: retry init count 1
[ 19.958375] iwlwifi 0004:41:00.0: Timeout waiting for PNVM load!
[ 19.958420] iwlwifi 0004:41:00.0: Failed to start RT ucode: -110
[ 19.958445] iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
[ 19.971024] iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
[ 19.983452] iwlwifi 0004:41:00.0: retry init count 2

- After executing above mv command:
[ 18.750383] iwlwifi 0004:41:00.0: enabling device (0000 -> 0002)
[ 18.765745] iwlwifi 0004:41:00.0: api flags index 2 larger than supported by driver
[ 18.765774] iwlwifi 0004:41:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[ 18.766197] iwlwifi 0004:41:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[ 18.777427] iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[ 18.896842] sd 0:0:0:0: Attached scsi generic sg0 type 0
[ 19.033116] iwlwifi 0004:41:00.0: base HW address: 04:cf:4b:21:24:e2
[ 19.047555] thermal thermal_zone7: power_allocator: sustainable_power will be estimated
[ 19.047621] thermal thermal_zone7: failed to read out thermal zone (-61)
[ 19.055209] iwlwifi 0004:41:00.0 wlP4p65s0: renamed from wlan0