iwlwifi Microcode SW error detected. Restarting 0x2000000.

[UrkoM]

This seems to be a return of an existing bug, in newer hardware and microcode versions.

Wireless card: Intel Corporation Wireless 8265 / 8275 [8086:24fd] (rev 78)
Firmware version: 34.0.1
Description: Ubuntu 18.04.1 LTS

Frequent errors as soon as I connect to wifi:
Microcode SW error detected. Restarting 0x2000000.

Expected to happen: Great and stable wifi like a couple of weeks ago
What happens: very slow, intermittent internet access, lots of ping loss

Found a suggestion of a workaround: adding these lines to /etc/NetworkManager/NetworkManager.conf

[device]
wifi.scan-rand-mac-address=no

These lines were already there, not added by me.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-firmware 1.173.2 [modified: lib/firmware/iwlwifi-8265-36.ucode]
ProcVersionSignature: Ubuntu 4.15.0-39.42-generic 4.15.18
Uname: Linux 4.15.0-39-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
Date: Fri Nov 23 15:23:54 2018
Dependencies:

InstallationDate: Installed on 2018-09-03 (81 days ago)
InstallationMedia: Kubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
PackageArchitecture: all
SourcePackage: linux-firmware
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Radoslav (radoslavius)]

Same problem on Ubuntu 18.10

$ uname -a
Linux uran 4.18.0-11-generic #12-Ubuntu SMP Tue Oct 23 19:22:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Network controller [0280]: Intel Corporation Wireless 8265 / 8275 [8086:24fd] (rev 78)

driverversion=4.18.0-11-generic
firmware=36.e91976c0.0

[Emmanuel Grumbach (egrumbach)]

Dmesg output is needed.

[UrkoM (urko-masse)]

dmesg output.

Note: lately the perceived effect of this issue seems to have gotten better. I still see the error in the logs, but wifi performance is acceptable.

[Emmanuel Grumbach (egrumbach)]

Ok, this firmware is fairly old.

[UrkoM (urko-masse)]

It's what Ubuntu 18.04.1 fully updated has... ¯\_(ツ)_/¯

[Christian Ehrhardt  (paelzer)]

Hi I'm affected as well.
I had hoped that the HWE kernel would somehow magically make things better but it doesn't.

System is a Lenovo T580 with Dual Band Wireless AC 8265

Attaching dmesg section where the wifi crashes

Interesting lines:
 529 [293336.199635] iwlwifi 0000:04:00.0: loaded firmware version 36.e91976c0.0 op_mode iwlmvm
 530 [293336.199661] iwlwifi 0000:04:00.0: Detected Intel(R) Dual Band Wireless AC 8265, REV=0x230
 531 [293336.261557] iwlwifi 0000:04:00.0: base HW address: 18:1d:ea:2d:5e:2e

 540 [293340.092557] iwlwifi 0000:04:00.0: Microcode SW error detected. Restarting 0x2000000.

 649 [293340.832903] iwlwifi 0000:04:00.0: Firmware error during reconfiguration - reprobe!
 650 [293340.832931] iwlwifi 0000:04:00.0: iwlwifi transaction failed, dumping registers

733 [293340.836537] WARNING: CPU: 1 PID: 22378 at /build/linux-hwe-jDoxDp/linux-hwe-4.18.0/net/mac80211/driver-ops.h:19 drv_remove_interface+0xfe/0x110 [mac80211]
(Repeats)

[Christian Ehrhardt  (paelzer)]

/lib/firmware/iwlwifi-8265-36.ucode is the newest even in the linux-firmware package of Ubuntu Disco

 linux-firmware | 1.173.2 | bionic-updates | source, all
 linux-firmware | 1.175.1 | cosmic-updates | source, all
 linux-firmware | 1.176 | disco | source, all

$ dpkg -L linux-firmware | grep iwlwifi | grep 8265
/lib/firmware/iwlwifi-8265-21.ucode
/lib/firmware/iwlwifi-8265-22.ucode
/lib/firmware/iwlwifi-8265-27.ucode
/lib/firmware/iwlwifi-8265-31.ucode
/lib/firmware/iwlwifi-8265-34.ucode
/lib/firmware/iwlwifi-8265-36.ucode

Version 1.176 is on commit 1baa3486 of Date: Fri Oct 26 08:13:19 2018 -0400
That is not too old IMHO and still has only version 36

But it seems not only "is it 36 or newer", there are changes like this:
https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/commit/?id=8707ef3313ee0f6737683c93aa2cb1dff6483643

Which bumps it further.
Overall there are three updates to the upstream repo that affect iwlwifi-8265* blobs and the Ubuntu releases in between matching the sha they picked:

8707ef3 Sun Dec 16 10:46:40 2018 +0200 iwlwifi: update firmwares for 8000 series
  Disco 1baa3486
  Cosmic fea76a04
58265e0 Sun May 6 14:50:51 2018 +0300 iwlwifi: update firmwares for 7000, 8000 and 9000 series
c711ea5 Fri Mar 2 12:29:35 2018 +0200 iwlwifi: add some new FW versions and update older ones
  Bionic 7344ec9e
5a05332 Fri Oct 13 14:22:26 2017 +0300 iwlwifi: add new firmware version 34 for 8000C and 8265
796c912 Fri Oct 13 14:22:26 2017 +0300 iwlwifi: update firmwares for 3160, 3168, 7265D, 8000C and 8265
d392df8 Fri Jun 16 14:20:27 2017 +0200 iwlwifi: add new firmware for 3168, 7265D, 8000C and 8265
8613e56 Fri Jun 16 11:48:51 2017 +0300 iwlwifi: update -27 firmware for 3168, 7265D, 8000C and 8265
b689306 Thu Jan 12 21:28:05 2017 +0200 iwlwifi: add -27 firmware for 3168, 7265D, 8000C and 8265
c1844f3 Sat Jan 7 22:29:08 2017 +0200 iwlwifi: update -22 firmware for 7265D and up
b224bd0 Sun Jul 10 09:32:17 2016 +0300 iwlwifi: add new -22 firmware for 7265D and up
a2c354e Tue Mar 8 09:29:48 2016 +0200 iwlwifi: add new -21.ucode for 7265D, 8260, 3168 and 8265 devices

So even thou all Versions are up to "36" there are maybe differences (you never know too much with binary blob updates in git).

[Christian Ehrhardt  (paelzer)]

I created a PPA for Bionic with the latest firmware (git of today) merged into the Package version we have in Disco. That can be found at [1].
But even that didn't help - which means even the very latest upstream will not fix the issue for us auto-magically.

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1804841-iwlwifi-linux-firmware/+packages

[Christian Ehrhardt  (paelzer)]

I thought it might be worth to try the other versions of the FW.
So I renamed the files like:
sudo mv /lib/firmware/iwlwifi-8265-36.ucode /lib/firmware/DONOTLOAD-iwlwifi-8265-36.ucode

Eventually I ended up with:
-rw-r--r-- 1 root root 1811984 Apr 24 2018 /lib/firmware/DONOTLOAD-iwlwifi-8265-22.ucode
-rw-r--r-- 1 root root 2234528 Dez 5 2017 /lib/firmware/DONOTLOAD-iwlwifi-8265-27.ucode
-rw-r--r-- 1 root root 2307104 Dez 6 2017 /lib/firmware/DONOTLOAD-iwlwifi-8265-31.ucode
-rw-r--r-- 1 root root 2440780 Apr 25 2018 /lib/firmware/DONOTLOAD-iwlwifi-8265-34.ucode
-rw-r--r-- 1 root root 2498044 Jan 11 10:03 /lib/firmware/DONOTLOAD-iwlwifi-8265-36.ucode
-rw-r--r-- 1 root root 2389968 Nov 17 2017 /lib/firmware/iwlwifi-8265-21.ucode

All versions >=22 Failed, no change in behavior at all.
When running with only 21 left it failed to load at all, see the following dmesg.

[ 5.651536] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-36.ucode failed with error -2
[ 5.651651] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-35.ucode failed with error -2
[ 5.652037] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-34.ucode failed with error -2
[ 5.652050] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-33.ucode failed with error -2
[ 5.652057] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-32.ucode failed with error -2
[ 5.652065] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-31.ucode failed with error -2
[ 5.652072] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-30.ucode failed with error -2
[ 5.652080] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-29.ucode failed with error -2
[ 5.652087] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-28.ucode failed with error -2
[ 5.652095] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-27.ucode failed with error -2
[ 5.652103] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-26.ucode failed with error -2
[ 5.652111] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-25.ucode failed with error -2
[ 5.652119] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-24.ucode failed with error -2
[ 5.652127] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-23.ucode failed with error -2
[ 5.652135] iwlwifi 0000:04:00.0: Direct firmware load for iwlwifi-8265-22.ucode failed with error -2
[ 5.652137] iwlwifi 0000:04:00.0: no suitable firmware found!
[ 5.652140] iwlwifi 0000:04:00.0: minimum version required: iwlwifi-8265-22
[ 5.652142] iwlwifi 0000:04:00.0: maximum version supported: iwlwifi-8265-36
[ 5.652144] iwlwifi 0000:04:00.0: check git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git

I thought, well then I run it without recent FW load, but then the wireless adapter doesn't initialize at all.

No chance to configure it in cmdline or UI tools.

Therefore it seems I know no way to get wireless to work at all on my T580 :-/
is there a Wireless dongle that works well with Linux?

If there is anything I can try to resolve this please let me know.

[Christian Ehrhardt  (paelzer)]

Note: the workaround of https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1710390 is in place by default in Bionic - this needs some other fix.

[Seth Forshee (sforshee)]

@Emmanuel: We generally only update firmware in stable releases in response to specific issues, as new firmware sometimes introduces regressions. Besides, ucode version 34 is the newest supported in the 4.15-based kernel in 18.04.

@UrkoM: When you say wifi used to be stable, was that prior to upgrading to bionic? Because the firmware version you're using looks to be the same one that bionic released with, so there were no updates which could have caused regressions.

[Christian Ehrhardt  (paelzer)]

Since some similar issues in the past were reported to be related to kernel updates lets add a task

[Christian Ehrhardt  (paelzer)]

Hi Seth,
to answer at least for me - it never worked on my system.
No "degradation on upgrade" at least for me, just never worked.

I'm considering filing a bug with Intel and or Lenovo (for a remote chance to get their FW developers to look at it) - if only I knew where.
If someone did that in the past let me know where the best place to start would be.

[UrkoM (urko-masse)]

@sforshee: No, wifi had been stable on Bionic before that. Somehow it seems quite stable again now. I got this laptop in September, so I installed Bionic directly.
I have checked my syslog, and I don't see the microcode loading errors anymore.

[Christian Ehrhardt  (paelzer)]

After all my tries with different Firmware versions I went back and since then the error didn't show up anymore. If anything it might be only (or more likely) showing up when the laptop is docked - but I don#t see how that would happen.

Anyway, for now it seems I'm good.
I'll check my logs in a few weeks and see if it came back at some point.

Since this seemed to resolve for UrkoM as well as myself - even thou we don't know the reason (yet) lets mark it incomplete for now.

[UrkoM (urko-masse)]

@paelzer: agreed. I also tried getting later firmwares from Intel's git repository and made no difference. I'll keep an eye out as well.

[Christian Ehrhardt  (paelzer)]

At least for me the error stopped showing up since 11th January up to now.
So whatever I did when experimenting/reporting that bug has resolved it.
Unfortunately I can't clearly tell anyone else affected what exactly that would be :-/

[Christian Ehrhardt  (paelzer)]

Oh I can tell what it was, I still use the most up-to-date-from-git version of the firmware from [1]. The testing back then wasn't successful, but I'm reluctant to go back to the base version as I'm afraid the error strikes me at some odd point when I really need wireless.

If anyone else is affected and could try if the PPA fixes it for him as well that might be helpful.

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1804841-iwlwifi-linux-firmware

[Jalon Funk (francescohickle15)]

It may be the same as this issue reported upstream: https://bugzilla.kernel.org/show_bug.cgi?id=202513

[Seth Forshee (sforshee)]

We have a fix applied for bug 1808389, and if that's the same issue then it should help when running the 4.18 hwe kernel in bionic. I plan to upload new packages today.

[Christian Ehrhardt  (paelzer)]

Yep, the changes accepted to B/C-unapproved as part of an SRU should be what we need.
Marking as a Duplicate so that we can test that SRU as well to hopefully resolve it for all of us.

[Aleksandr Mikhalitsyn (mihalicyn)]

@paelzer,
>If anything it might be only (or more likely) showing up when the laptop is docked - but I don#t see >how that would happen.
you are completely right!
After you comment I've made detailed experiments with that. And on my ASUS VivoBook Pro with Intel 8265 problem will appearing quickly after I've put something under my laptop (cooling dock, any another metal thing). As I think problem is in signal power settings in driver or something similar. Device is affected by interference of electromagnetic waves that radiated by themself...

Very interesting fact that in Windows 10 with standard drivers all works fine in the same conditions...

[Aleksandr Mikhalitsyn (mihalicyn)]

https://bugzilla.kernel.org/show_bug.cgi?id=203277#c5

[Aleksandr Mikhalitsyn (mihalicyn)]

May be it will be helpful for someone...
echo "options cfg80211 cfg80211_disable_40mhz_24ghz=Y" > /etc/modprobe.d/cfg80211.conf
worked for me on my ASUS VivoBook Pro N580GD (platform noise issues)

[Illi (illimaier)]

Ok, this is a complex matter. I just experienced this unpleasant feature on my laptop using the above described driver and hardware for the last two days. =>It seems to be a radio transmission issue.<= (and this is damned complex!)

The Linux driver as well as the microcode of the WiFi device must be able to cope with the small frequency jumps of the access point. Yes, frequencies are =>NOT<= very stable. This is shown in the logs by the following messages: wpa_supplicant[971]: wlp1s0: CTRL-EVENT-SIGNAL-CHANGE. When this event occurs, your hardware must resync with the access point. The driver as well as the microcode of your WiFi device may or may not support such a frequency jump. In other term they may or may not be able to cope with the noise. The following log message shows that my system was not able to cope with it: iwlwifi 0000:01:00.0: Microcode SW error detected.

What actually solved the issue is re-configuring the access point. I limited the channel width on the 2.4 Ghz frequency to 20 Mhz to avoid interference with other access points in the area. In addition I defined a precise channel for the access. For the channel selection, prefer channel 1, 6 or 11, since they do not overlap. Now my connection is stable and my logs are clean.

For the 5 Ghz network, these problem might not occur as frequently as on the 2.4 Ghz one, since it is less used. In case similar issues are faced, proceed similarly.

Hope it helps, Illi

[Russell Neches (ubuntu-vort)]

I was hoping a firmware update would solve the problem, but now it is significantly worse. The crashes happen so frequently that the machine locks up for a minute or two. Even the capslock light on the keyboard becomes unresponsive. This usually happens just after boot or coming out of suspend, but not always.

$ uname -a
Linux evenedric 5.3.0-19-generic #20-Ubuntu SMP Fri Oct 18 09:04:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ fwupdmgr get-devices
20FBCTO1WW System Firmware
  DeviceId: [redacted]
  Guid: [redacted]
  Plugin: uefi
  Flags: internal|updatable|require-ac|supported|registered|needs-reboot
  Version: 0.1.42
  VersionLowest: 0.1.31
  VersionFormat: triplet
  Icon: computer
  Created: 2019-11-12

UEFI Device Firmware
  DeviceId: [redacted]
  Guid: [redacted]
  Plugin: uefi
  Flags: internal|updatable|require-ac|supported|registered|needs-reboot
  Version: 184.65.3590
  VersionLowest: 0.0.1
  VersionFormat: triplet
  Icon: audio-card
  Created: 2019-11-12

UEFI Device Firmware
  DeviceId: [redacted]
  Guid: [redacted]
  Plugin: uefi
  Flags: internal|updatable|require-ac|supported|registered|needs-reboot
  Version: 0.1.12
  VersionLowest: 0.0.1
  VersionFormat: triplet
  Icon: audio-card
  Created: 2019-11-12
  UpdateState: success

$ iw reg get
global
country US: DFS-FCC
 (2402 - 2472 @ 40), (N/A, 30), (N/A)
 (5170 - 5250 @ 80), (N/A, 23), (N/A), AUTO-BW
 (5250 - 5330 @ 80), (N/A, 23), (0 ms), DFS, AUTO-BW
 (5490 - 5730 @ 160), (N/A, 23), (0 ms), DFS
 (5735 - 5835 @ 80), (N/A, 30), (N/A)
 (57240 - 63720 @ 2160), (N/A, 40), (N/A)

phy#0 (self-managed)
country 00: DFS-UNSET
 (2402 - 2437 @ 40), (6, 22), (N/A), AUTO-BW, NO-HT40MINUS, NO-80MHZ, NO-160MHZ
 (2422 - 2462 @ 40), (6, 22), (N/A), AUTO-BW, NO-80MHZ, NO-160MHZ
 (2447 - 2482 @ 40), (6, 22), (N/A), AUTO-BW, NO-HT40PLUS, NO-80MHZ, NO-160MHZ
 (5170 - 5190 @ 80), (6, 22), (N/A), NO-OUTDOOR, AUTO-BW, IR-CONCURRENT, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
 (5190 - 5210 @ 80), (6, 22), (N/A), NO-OUTDOOR, AUTO-BW, IR-CONCURRENT, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
 (5210 - 5230 @ 80), (6, 22), (N/A), NO-OUTDOOR, AUTO-BW, IR-CONCURRENT, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
 (5230 - 5250 @ 80), (6, 22), (N/A), NO-OUTDOOR, AUTO-BW, IR-CONCURRENT, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
 (5250 - 5270 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
 (5270 - 5290 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
 (5290 - 5310 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
 (5310 - 5330 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
 (5490 - 5510 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
 (5510 - 5530 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
 (5530 - 5550 @ 80), (6, 22), (0 m...