firmware broken after upgrade to Ubuntu 23.10

[joepadmiraal]

[ Impact ]

 * Focal preinstalled systems with OEM kernel, if force upgraded to interim releases, before transitional packages have been made available resulted in systems having been stuck on focal oem kernel and not getting any kernel updates.

 * Reintroduce all focal oem kernel transitionals to trigger upgrade to hwe kernel in all stable releases for the oem-20.04* kernels

[ Test Plan ]

 * Starting with focal, install linux-image-oem-20.04* linux-headers-oem-20.04* metapackages
 * Force upgrade to mantic
 * Observe that all oem-20.04 meta packages got upgraded and rolled to generic hwe kernel (of appropriate version for jammy & mantic)

[ Where problems could occur ]

 * This may trigger dkms package recompilation, and nvidia driver missmatches
 * Likely users will have to rerun `ubuntu-drivers` or use additional drivers GUI to reinstall nvidia drivers
 * Users may experience jump from v5.10 kernel to v6.5 kernel

[ Other Info ]

 * Original bug report

After upgrading from Ubuntu 23.04 to 23.10 on a Dell Precision 5560, wifi and bluetooth did not work anymore.
Looking at the dmesg output I saw this:
Direct firmware load for iwlwifi-QuZ-a0-hr-b0-59.ucode failed with error -2
..
Direct firmware load for iwlwifi-QuZ-a0-hr-b0-39.ucode failed with error -2
no suitable firmware found!
minimum version required: iwlwifi-QuZ-a0-hr-b0-39
minimum version required: iwlwifi-QuZ-a0-hr-b0-59

These files did seem to exist in /lib/firmware.
I did sudo apt install linux-firmware/mantic --reinstall
Which did not fix it.

I manually overwrote the iwlwifi... files with the corresponding files from linux-firmware-20230804.tar.gz and this did fix the issue.
I did the same for ibt-19-0-4.sfi and that fixed the bluetooth issue.

So my best guess is that something is wrong in the latest firmware package.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 23.10
Release: 23.10
Codename: mantic

apt-cache policy linux-firmware
linux-firmware:
  Installed: 20230919.git3672ccab-0ubuntu2.1
  Candidate: 20230919.git3672ccab-0ubuntu2.1
  Version table:
 *** 20230919.git3672ccab-0ubuntu2.1 500
        500 http://archive.ubuntu.com/ubuntu mantic/main amd64 Packages
        100 /var/lib/dpkg/status

ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: linux-firmware 20230919.git3672ccab-0ubuntu2.1
ProcVersionSignature: Ubuntu 5.10.0-1053.55-oem 5.10.83
Uname: Linux 5.10.0-1053-oem x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: joep 2545 F.... wireplumber
 /dev/snd/controlC0: joep 2545 F.... wireplumber
 /dev/snd/seq: joep 2537 F.... pipewire
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Sun Oct 22 15:19:18 2023
Dependencies: firmware-sof-signed 2.2.6-1ubuntu1
DistributionChannelDescriptor:
 # This is the distribution channel descriptor for the OEM CDs
 # For more information see http://wiki.ubuntu.com/DistributionChannelDescriptor
 canonical-oem-somerville-focal-amd64-20200502-85
InstallationDate: Installed on 2021-10-16 (736 days ago)
InstallationMedia: Ubuntu 20.04 "Focal" - Build amd64 LIVE Binary 20200502-05:58
MachineType: {report['dmi.sys.vendor']} {report['dmi.product.name']}
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.10.0-1053-oem root=UUID=482b1305-d8fa-4fb1-be20-ae911e6a4def ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-5.10.0-1053-oem N/A
 linux-backports-modules-5.10.0-1053-oem N/A
 linux-firmware 20230919.git3672ccab-0ubuntu2.1
SourcePackage: linux-firmware
UpgradeStatus: Upgraded to mantic on 2023-10-22 (0 days ago)
dmi.bios.date: 07/13/2023
dmi.bios.release: 1.22
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.22.0
dmi.board.name: 0NG7N9
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 10
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.22.0:bd07/13/2023:br1.22:svnDellInc.:pnPrecision5560:pvr:sku0A62:rvnDellInc.:rn0NG7N9:rvrA00:cvnDellInc.:ct10:cvr:
dmi.product.family: Precision
dmi.product.name: Precision 5560
dmi.product.sku: 0A62
dmi.sys.vendor: Dell Inc.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andrzej Stencel (andrzejste)]

I had the same problem and I was able to fix this on my machine using guidelines from @joepadmiraal - thank you!

Here are the steps I followed:

1. I went to to https://cdn.kernel.org/pub/linux/kernel/firmware/ and downloaded https://cdn.kernel.org/pub/linux/kernel/firmware/linux-firmware-20230804.tar.gz file into /home/andrzej/Downloads.

2. I unpacked the downloaded file into /home/andrzej/Downloads/linux-firmware-20230804

    tar zxvf ./linux-firmware-20230804.tar.gz

3. I created a backup directory /home/andrzej/lib-firmware-backup.

    mkdir /home/andrzej/lib-firmware-backup

4. I backed up Wi-Fi firmware files.

    sudo mv /lib/firmware/iwlwifi-* /home/andrzej/lib-firmware-backup

5. I copied Wi-Fi firmware files from downloaded archive into /lib/firmware.

    sudo cp /home/andrzej/Downloads/linux-firmware-20230804/iwlwifi-* /lib/firmware

    Note that the files in /lib/firmware are compressed (they have *.zst extension), and the files the downloaded archive are not compressed. This is not a problem; I just put uncompressed files in /lib/firmware and things worked fine.

6. I backed up Bluetooth firmware files.

    sudo mv /lib/firmware/intel/ibt-* /home/andrzej/lib-firmware-backup/intel

7. I copied Bluetooth firmware files from downloaded archive into /lib/firmware.

    sudo cp /home/andrzej/Downloads/linux-firmware-20230804/intel/ibt-* /lib/firmware/intel

After restarting the machine, Wi-Fi and Bluetooth are back.

[claudia (c-hundertmark)]

I have the same issue. Thanks for providing instructions to fix it. I was able to resolve the Bluetooth problem with your fix, but WiFI still doesn't work.

[Dan (dandandandan4)]

Can confirm it affected me as well for what regards the wireless. I don't use Bluetooth so I can't confirm on that side.
The solution with linux-firmware-20230804.tar.xz worked for me.

[Felicià Barceló (feliciabarcelo)]

I have same problem with wifi and bluetooth. After apply this patch, all is working again. Thanks joepadmiraal and Andrzej Stencel.

[Juerg Haefliger (juergh)]

Can you post the full log from a failure?

$ sudo journalctl -k -b 0 (use -1, -2, ... for earlier boots).

'failed with error -2' means the firmware file is not found, so I'm puzzled...

[joepadmiraal (joep-groovytunes)]

Hi,

I had another look, and it seems like the Ubuntu package is not including the iwlwifi-QuZ....ucode files.
Instead it contains the files in a compressed format: iwlwifi-QuZ....ucode.zst
So I assume the system searches for the uncompressed files, which are not present on my system after doing a clean install.

I checked the Ubuntu 23.04 version of the linux-firmware package, and that one does contain the .ucode files: https://packages.ubuntu.com/lunar/linux-firmware

[Juerg Haefliger (juergh)]

Starting with Lunar, we compress firmware blobs and the kernel can load them. But I just realized that you're running an old kernel that went out of support a long time ago: 5.10.0-1053-oem. Why?

This old kernel can't handle compressed firmwares. You really need to upgrade your kernel.

[joepadmiraal (joep-groovytunes)]

I have no idea why I'm at that kernel version.
I'll try to update it.
I have a Dell precision that came with Ubuntu preinstalled and just did Ubuntu updates on that.
So maybe something in the Dell configuration makes it stick at this kernel version.
And it seems more people are having this issue.
At least people with this issue can now find the cause via google.

[joepadmiraal (joep-groovytunes)]

I installed the latest oem kernel with: `sudo apt install linux-oem-22.04`.
It installed the 6.5 kernel and I can confirm it solved the issue.

[Kai-Chuan Hsieh (kchsieh)]

The platform is shipped with focal preinstalled. May I know how did you upgrade to the current codename you are using?

Could you attach $ sudo -E oem-getlogs to let me know your current configuration?

Thanks,

[Andrzej Stencel (andrzejste)]

> I have no idea why I'm at that kernel version.
> I have a Dell precision that came with Ubuntu preinstalled and just did Ubuntu updates on that.

Same story here. Got the Dell Precision 5560 with Ubuntu 20.04, then I think I updated to:
- 22.04
- 22.10 (or perhaps skipped this one)
- 23.04
- 23.10

$ uname -a
Linux astencel-dell 5.14.0-1054-oem #61-Ubuntu SMP Fri Oct 14 13:05:50 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[Andrzej Stencel (andrzejste)]

Trying to attach the output of `sudo -E oem-getlogs`, but getting an error when trying to post the comment.

[Timo Aaltonen (tjaalton)]

please run 'apt-cache policy linux-image-generic'

[Andrzej Stencel (andrzejste)]

$ apt-cache policy linux-image-generic
linux-image-generic:
  Installed: (none)
  Candidate: 6.5.0.10.12
  Version table:
     6.5.0.10.12 500
        500 http://archive.ubuntu.com/ubuntu mantic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu mantic-security/main amd64 Packages
     6.5.0.9.11 500
        500 http://archive.ubuntu.com/ubuntu mantic/main amd64 Packages

[Timo Aaltonen (tjaalton)]

ack, so what happenened was likely this:

- you had 20.04 preinstalled with an oem image and -oem-20.04b kernel (5.10)
- upgraded to an interim release or 22.04 *before* 5.10 was EOL'd in focal (and moved to hwe-5.15) which happened around March/April '22

-> stuck with oem-5.10 since 22.04 didn't (and still doesn't) come with transitionals for b/c(5.13)/d(5.14) flavors

So to fix that we need to add the b/c/d transitionals to jammy linux-meta, and also lunar/mantic/noble

[Timo Aaltonen (tjaalton)]

the quick fix for this is to run 'sudo apt install linux-generic', which would pull the latest generic (= distro) kernel

Your hw is old enough that it shouldn't require the 6.5 oem kernel :)

[joepadmiraal (joep-groovytunes)]

Great, I assume we don't need to do the `sudo -E oem-getlogs` that Andrzej requested anymore?

[Timo Aaltonen (tjaalton)]

nope, this is a clear process mistake which we'll make sure doesn't happen again..

[Andrzej Stencel (andrzejste)]

Thanks Timo, I updated the kernel and things are still working OK.

$ uname -a
Linux astencel-dell 6.5.0-10-generic #10-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 13 13:49:38 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ apt-cache policy linux-image-generic
linux-image-generic:
  Installed: 6.5.0.10.12
  Candidate: 6.5.0.10.12
  Version table:
 *** 6.5.0.10.12 500
        500 http://archive.ubuntu.com/ubuntu mantic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu mantic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     6.5.0.9.11 500
        500 http://archive.ubuntu.com/ubuntu mantic/main amd64 Packages

[Dimitri John Ledkov (xnox)]

https://lists.ubuntu.com/archives/kernel-team/2023-November/146940.html

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dan (dandandandan4)]

Hello everyone, just chiming in to say that my Ubuntu was installed by me, no OEM at all, and I am on 6.5.0-10-generic, and still I was affected by this issue.
Manually copying iwlwifi* and intel/ibt-* from linux-firmware-20230804.tar.xz solved the issue for me.

[Juerg Haefliger (juergh)]

@Dan, please describe your issue and post kernel logs of the failure. It's can't be the same.

[Dan (dandandandan4)]

@Juerg, here's to you the logs that I grabbed from /var/log/syslog.
I pasted what I believe it's the useful part in https://pastebin.com/raw/KRLAkS4J - anyway, it keeps going and going.

[Dimitri John Ledkov (xnox)]

On Fri, 17 Nov 2023 at 12:01, Dan <email address hidden> wrote:
>
> @Juerg, here's to you the logs that I grabbed from /var/log/syslog.
> I pasted what I believe it's the useful part in https://pastebin.com/raw/KRLAkS4J - anyway, it keeps going and going.

Unfortunately that is not useful. Please open a new bug report using
`ubuntu-bug linux` and please ensure that full boot dmsg is applied
(when issue is occurring), as in `journalctl -b -k` as otherwise we
cannot confirm which kernel is booted, which device initialized, and
when things start to go south; as the last error in syslog is usually
just a symptom and not the cause.

--
okurrr,

Dimitri

[claudia (c-hundertmark)]

On my Dell XPS 13 I installed the new kernel 6.5 because I still was on 5.14. With that new kernel I don't have wifi and even no wired internet. This didn't help at all.

Edit: now I have one, that works for me :-)

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-meta - 6.6.0-14.14

---------------
linux-meta (6.6.0-14.14) noble; urgency=medium

  * Master version: 6.6.0-14.14

 -- Paolo Pisati <email address hidden> Thu, 30 Nov 2023 10:20:25 +0100

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted linux-meta (6.5.0.16.18) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

diffoscope/249 (armhf)
initramfs-tools/0.142ubuntu15.1 (arm64, armhf)
zfs-linux/2.2.0-0ubuntu1~23.10 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#linux-meta

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted linux-meta (6.5.0.17.19) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.142ubuntu15.1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#linux-meta

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Petter Sundlöf (petter-sundlof)]

The fix in https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/2040106/comments/3 used to work for me, but no longer does. Upgraded to latest packages as of 2024-04-29. Bluetooth still broken