Regression on Linux 6.8.0-38: Bluetooth adapter (Intel AX200) stops working after a few suspend/resume cycles

[Eugene San]

SRU Justification:
==============

[Impact]
AX200 Bluetooth stops working after a few suspend/resume cycles.

[Fix]
Update the Bluetooth firmware and align with upstream.

[Test]
Bug reporter tested on hardware, Bluetooth works fine.

[Where problems could occur]
It may break Bluetooth of AX200.
===========================

After upgrading to Linux 6.8.0-38, Bluetooth adapter (USB part of the Intel AX200) stops working after a few suspend/resume cycles. WiFi (PCIe part) continues to work as expected.

Kernel log shows:
Bluetooth: hci0: Reading Intel version command failed (-110)
instead of the usual:
Bluetooth: hci0: Bootloader revision 0.3 build 0 week 24 2017
Bluetooth: hci0: Device revision is 1
Bluetooth: hci0: Secure boot is enabled
Bluetooth: hci0: OTP lock is enabled
Bluetooth: hci0: API lock is enabled
Bluetooth: hci0: Debug lock is disabled
Bluetooth: hci0: Minimum firmware build 1 week 10 2014
Bluetooth: hci0: Found device firmware: intel/ibt-20-1-3.sfi
Bluetooth: hci0: Boot Address: 0x24800
Bluetooth: hci0: Firmware Version: 255-255.255
Bluetooth: hci0: Waiting for firmware download to complete
Bluetooth: hci0: Firmware loaded in 2111457 usecs
Bluetooth: hci0: Waiting for device to boot
Bluetooth: hci0: Device booted in 14560 usecs
Bluetooth: hci0: Malformed MSFT vendor event: 0x02
Bluetooth: hci0: Found Intel DDC parameters: intel/ibt-20-1-3.ddc
Bluetooth: hci0: Applying Intel DDC parameters completed
Bluetooth: hci0: Firmware revision 0.3 build 249 week 27 2023
Bluetooth: hci0: HCI LE Coded PHY feature bit is set, but its usage is not supported.
Bluetooth: MGMT ver 1.22

First appeared on 6.8.0-38, 6.8.0-36 and earlier worked without issues.
I suspect linux-firmware might be missing updates corresponding to changes in the kernel driver or a regression was introduced in the update.

Machine: Dell XPS13-9350
Device: Intel AX200 Rev 1a (WiFi/Bluetooth combo)

[Eugene San (eugenesan)]

Reloading the driver:
```
modprobe -r btusb
modprobe -r btintel
modprobe btintel
modprobe btusb
```
fixes the issue (temporally):
```
Bluetooth: hci0: Reading Intel version command failed (-110)
usbcore: deregistering interface driver btusb
usbcore: registered new interface driver btusb
Bluetooth: hci0: Bootloader revision 0.3 build 0 week 24 2017
Bluetooth: hci0: Device revision is 1
Bluetooth: hci0: Secure boot is enabled
Bluetooth: hci0: OTP lock is enabled
Bluetooth: hci0: API lock is enabled
Bluetooth: hci0: Debug lock is disabled
Bluetooth: hci0: Minimum firmware build 1 week 10 2014
Bluetooth: hci0: Found device firmware: intel/ibt-20-1-3.sfi
Bluetooth: hci0: Boot Address: 0x24800
Bluetooth: hci0: Firmware Version: 255-255.255
Bluetooth: hci0: Waiting for firmware download to complete
Bluetooth: hci0: Firmware loaded in 2291806 usecs
Bluetooth: hci0: Waiting for device to boot
Bluetooth: hci0: Device booted in 14520 usecs
Bluetooth: hci0: Malformed MSFT vendor event: 0x02
Bluetooth: hci0: Found Intel DDC parameters: intel/ibt-20-1-3.ddc
Bluetooth: hci0: Applying Intel DDC parameters completed
Bluetooth: hci0: Firmware revision 0.3 build 249 week 27 2023
Bluetooth: hci0: HCI LE Coded PHY feature bit is set, but its usage is not supported.
Bluetooth: MGMT ver 1.22
```

[AaronMa (mapengyu)]

@eugenesan

Could you upload dmesg after the issue is reproduced?

From the log there is no suspect error.

[AaronMa (mapengyu)]

Please use bug sos-report if it's possible, linux-firmware version matters too.

[Eugene San (eugenesan)]

@mapengyu

linux-firmware 20240318.git3b128b60-0ubuntu2.1
linux 6.8.0-38.38

Attaching kernel log since last boot until btintel stops working.

[AaronMa (mapengyu)]

Hi eugenesan

Could you try to reproduce this issue on 6.8.0-36 to double confirm -38 is the first bad version.

Since I didn't find any related changes between -36 and -38.

[Eugene San (eugenesan)]

@mapengyu

I was running -36 for 16 days without any issues.

I tried to use -38 in June (before -36 was published) and experienced the same issue with Bluetooth right away. I ignored it at the time assuming it was a one time glitch.

But now when I tried -38 again (after it was officially published) I saw the same behavior 2 times inside 3 days. I am 99% confident the issue is somewhere between -36 and -38.

Regarding kernel changes, there was at least 1 suspect change to `intelbt`:
```
8902291975c6 [Luiz Augusto von Dentz] - Bluetooth: btintel: Fixe build regression
```
and a couple firmware related changes in `iwlwifi` that might be related.

[AaronMa (mapengyu)]

Test kernel that revert 2 commits:

commit 454482593427d04f5600b47bdb6b118882522c59 (HEAD -> noble-e1000eulp-lp2073358-btintel-lp2073047)
Author: Aaron Ma <email address hidden>
Date: Wed Jul 17 22:34:52 2024 -0400

    Revert "Bluetooth: btintel: Fix null ptr deref in btintel_read_version"

    This reverts commit 842b2829368f9b434ecdb2ba1e5d644f9aab9cd0.

commit 9b78f711e594a803f9c686b8ac5909da9293dd94
Author: Aaron Ma <email address hidden>
Date: Wed Jul 17 22:34:38 2024 -0400

    Revert "Bluetooth: btintel: Fixe build regression"

    This reverts commit 8902291975c6ca8a3c64a4a831d99d9122566c15.

[Eugene San (eugenesan)]

@mapengyu
Installed the test kernel.
dmesg looks identical to -38.
Will report back asap.

Attaching dmesg for -36, -38 and test kernels.

[Eugene San (eugenesan)]

I've been testing kernel from https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2073047/comments/7 for 6 days now.
I'd continue testing for longer but without kernel headers for DKMS I am missing functionality I need.

So far zero issues, Bluetooth works across many suspend/resume cycles.
Aside from missing bluetooth error msgs on resume, log is pretty much identical to 6.8.0-3x.

The only weird thing I've noticed is severe degradation in AVX performance when compared to previous kernels:
```
[raid6: avx2x4 gen() 29422 MB/s] vs [raid6: avx2x4 gen() 18152 MB/s]
```

P.S.
I've removed bluetooth init errors from previous comment as it was a mistake on my part while comparing log files.

[AaronMa (mapengyu)]

Thanks, eugenesan

linux-headers is uploaded above.

[Eugene San (eugenesan)]

@mapengyu
Thanks!
I'll continue my test and report back in a few days.

[Eugene San (eugenesan)]

@mapengyu

Unfortunately the fix didn't work.
Just now, after 11 days of uptime and ~25 suspend-resume cycles, bluetooth (btintel) didn't wake up after suspend.

I am attaching kernel log of failed suspend-resume with the error msg:
```
Jul 29 14:11:44 hostanme kernel: Bluetooth: hci0: Reading Intel version command failed (-110)
Jul 29 14:11:44 hostanme kernel: Bluetooth: hci0: command 0xfc05 tx timeout
```

It took a couple of attempts to get it working again with:
```
modprobe -r btusb
modprobe -r btintel
modprobe btintel
modprobe btusb
systemctl restart bluetooth
```

```
Jul 29 14:31:38 hostanme kernel: usbcore: deregistering interface driver btusb
Jul 29 14:31:51 hostanme kernel: usbcore: registered new interface driver btusb
Jul 29 14:31:53 hostanme kernel: Bluetooth: hci0: Reading Intel version command failed (-110)
Jul 29 14:31:53 hostanme kernel: Bluetooth: hci0: command 0xfc05 tx timeout
Jul 29 14:33:05 hostanme kernel: usbcore: deregistering interface driver btusb
Jul 29 14:33:24 hostanme kernel: usbcore: registered new interface driver btusb
Jul 29 14:33:24 hostanme kernel: Bluetooth: hci0: Bootloader revision 0.3 build 0 week 24 2017
Jul 29 14:33:24 hostanme kernel: Bluetooth: hci0: Device revision is 1
Jul 29 14:33:24 hostanme kernel: Bluetooth: hci0: Secure boot is enabled
Jul 29 14:33:24 hostanme kernel: Bluetooth: hci0: OTP lock is enabled
Jul 29 14:33:24 hostanme kernel: Bluetooth: hci0: API lock is enabled
Jul 29 14:33:24 hostanme kernel: Bluetooth: hci0: Debug lock is disabled
Jul 29 14:33:24 hostanme kernel: Bluetooth: hci0: Minimum firmware build 1 week 10 2014
Jul 29 14:33:24 hostanme kernel: Bluetooth: hci0: Found device firmware: intel/ibt-20-1-3.sfi
Jul 29 14:33:24 hostanme kernel: Bluetooth: hci0: Boot Address: 0x24800
Jul 29 14:33:24 hostanme kernel: Bluetooth: hci0: Firmware Version: 255-255.255
Jul 29 14:33:27 hostanme kernel: Bluetooth: hci0: Waiting for firmware download to complete
Jul 29 14:33:27 hostanme kernel: Bluetooth: hci0: Firmware loaded in 2188594 usecs
Jul 29 14:33:27 hostanme kernel: Bluetooth: hci0: Waiting for device to boot
Jul 29 14:33:27 hostanme kernel: Bluetooth: hci0: Device booted in 14594 usecs
Jul 29 14:33:27 hostanme kernel: Bluetooth: hci0: Malformed MSFT vendor event: 0x02
Jul 29 14:33:27 hostanme kernel: Bluetooth: hci0: Found Intel DDC parameters: intel/ibt-20-1-3.ddc
Jul 29 14:33:27 hostanme kernel: Bluetooth: hci0: Applying Intel DDC parameters completed
Jul 29 14:33:27 hostanme kernel: Bluetooth: hci0: Firmware revision 0.3 build 249 week 27 2023
Jul 29 14:33:27 hostanme kernel: Bluetooth: hci0: HCI LE Coded PHY feature bit is set, but its usage is not supported.
Jul 29 14:33:27 hostanme kernel: Bluetooth: MGMT ver 1.22
```

[AaronMa (mapengyu)]

All kernel changes from 6.8.0-35 to 6.8.0-38:

8902291975c6 Bluetooth: btintel: Fixe build regression
5a51a3be7e02 Bluetooth: Add new quirk for broken read key length on ATS2851
01a5bff079f2 Bluetooth: btmtk: Add MODULE_FIRMWARE() for MT7922
842b2829368f Bluetooth: btintel: Fix null ptr deref in btintel_read_version
50023458b1c0 Bluetooth: qca: fix device-address endianness
d914e8c40003 Revert "Bluetooth: hci_qca: Set BDA quirk bit if fwnode exists in DT"

I didn't see any related changes.

Could you check with the latest linux-firmware for bluetooth [1]?

Please cp all the files in [1] to /lib/firmware/intel,
$ update-initramfs -u -k all

IMPORTANT: Power cycle and retry.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/intel

[AaronMa (mapengyu)]

Another question is your firmware version is not correct:
Bluetooth: hci0: Firmware Version: 255-255.255

The right value shoulde be like:
Bluetooth: hci0: Firmware Version: 132-3.24

I doubt it's not a driver issue.

[Eugene San (eugenesan)]

@mapengyu

Thanks you for the advice.
I "installed" the latest firmware/ddc images from the linux-firware git and they are very different from the ones in linux-firmware-20240318.git3b128b60-0ubuntu2.1. Uncompressed binary diff is ~1KBytes for .sfi and 1 Byte for ddc.

The firmware works and device init looks completely different:
```
[ 63.709269] kernel: Bluetooth: Core ver 2.22
[ 63.709297] kernel: NET: Registered PF_BLUETOOTH protocol family
[ 63.709299] kernel: Bluetooth: HCI device and connection manager initialized
[ 63.709305] kernel: Bluetooth: HCI socket layer initialized
[ 63.709308] kernel: Bluetooth: L2CAP socket layer initialized
[ 63.709313] kernel: Bluetooth: SCO socket layer initialized
[ 63.881562] kernel: Bluetooth: hci0: Bootloader revision 0.3 build 0 week 24 2017
[ 63.882542] kernel: Bluetooth: hci0: Device revision is 1
[ 63.882549] kernel: Bluetooth: hci0: Secure boot is enabled
[ 63.882551] kernel: Bluetooth: hci0: OTP lock is enabled
[ 63.882553] kernel: Bluetooth: hci0: API lock is enabled
[ 63.882555] kernel: Bluetooth: hci0: Debug lock is disabled
[ 63.882557] kernel: Bluetooth: hci0: Minimum firmware build 1 week 10 2014
[ 63.894660] kernel: Bluetooth: hci0: Found device firmware: intel/ibt-20-1-3.sfi
[ 63.894829] kernel: Bluetooth: hci0: Boot Address: 0x24800
[ 63.894832] kernel: Bluetooth: hci0: Firmware Version: 132-3.24
[ 65.721208] kernel: Bluetooth: hci0: Waiting for firmware download to complete
[ 65.721489] kernel: Bluetooth: hci0: Firmware loaded in 1784005 usecs
[ 65.721632] kernel: Bluetooth: hci0: Waiting for device to boot
[ 65.736516] kernel: Bluetooth: hci0: Device booted in 14623 usecs
[ 65.736519] kernel: Bluetooth: hci0: Malformed MSFT vendor event: 0x02
[ 65.736703] kernel: Bluetooth: hci0: Found Intel DDC parameters: intel/ibt-20-1-3.ddc
[ 65.738526] kernel: Bluetooth: hci0: Applying Intel DDC parameters completed
[ 65.739514] kernel: Bluetooth: hci0: Firmware revision 0.3 build 132 week 3 2024
[ 65.741526] kernel: Bluetooth: hci0: HCI LE Coded PHY feature bit is set, but its usage is not supported.
[ 69.226391] kernel: Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 69.226398] kernel: Bluetooth: BNEP filters: protocol multicast
[ 69.226403] kernel: Bluetooth: BNEP socket layer initialized
[ 69.237051] kernel: Bluetooth: MGMT ver 1.22
```
Comparing kernel logs of the original and updated firmwares, it seems like the original firmware always failed on first try (errro -19) and worked only on the second try even on boot. No wonder it randomly fails after resume. Updated firmware loaded the first try.

In addition, I've noticed that installed firmware tree is a mess of sym-links.
Maybe the problem lies in how linux-firmware package is optimized?
```
ibt-20-0-3.ddc.zst -> ibt-0040-0041.ddc.zst
ibt-20-0-3.sfi.zst
ibt-20-1-3.ddc.zst -> ibt-0040-0041.ddc.zst
ibt-20-1-3.sfi.zst -> ibt-20-0-3.sfi.zst
ibt-20-1-4.ddc.zst -> ibt-0040-0041.ddc.zst
ibt-20-1-4.sfi.zst
```

It's important to note that older kernel that did work for me used the same "defective" firmware version.
Possibly, since the issue is so infrequent, I just was very lucky for a few m...

[Eugene San (eugenesan)]

After 10 days of zero issues, I am convinced that the problem is with linux-firmware and not kernel.
I've changed affected package accordingly.

[AaronMa (mapengyu)]

@Eugene

SRU is sent out to both Jammy and Noble, please verify it when it's merged that status will be changed here.

[Timo Aaltonen (tjaalton)]

Hello Eugene, or anyone else affected,

Accepted linux-firmware into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/20240318.git3b128b60-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Eugene, or anyone else affected,

Accepted linux-firmware into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/20220329.git681281e4-0ubuntu3.32 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Eugene San (eugenesan)]

After installing linux-firmware 20240318.git3b128b60-0ubuntu2.2 i can confirm that now the driver loads the correct versions of firmware/ddc and everything works as expected.

Since identical (manually installed) firmware binaries worked perfectly for weeks, I am confident the new package will have no issues. I'll post an update if anything changes.

Thanks to everyone involved!

[Timo Aaltonen (tjaalton)]

Hello Eugene, or anyone else affected,

Accepted linux-firmware into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/20220329.git681281e4-0ubuntu3.33 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 20240318.git3b128b60-0ubuntu2.2

---------------
linux-firmware (20240318.git3b128b60-0ubuntu2.2) noble; urgency=medium

  * CS42L43 and CS35L56 for soundwire on Intel LNL/ARL (LP: #2069412)
    - linux-firmware: Add firmware for Cirrus CS35L56 for Dell laptops
  * Missing firmware for Intel NPU driver on Intel Lunar Lake platform: Failed to request firmware: -2 (LP: #2071771)
    - SAUCE: Add firmware for Intel NPU on Lunar Lake platforms
  * Add support for DCN 3.5.1 (LP: #2068001)
    - amdgpu: Add support for DCN 3.5.1
    - amdgpu: add support for GC 11.5.1
    - amdgpu: add support for SDMA 6.1.1
    - amdgpu: add support for VCN 4.0.6
    - amdgpu: add support for VPE 6.1.1
    - amdgpu: add support for PSP 14.0.1
  * Add support for AMD ISP 4.1.1 (LP: #2071812)
    - amdgpu: add new ISP 4.1.1 firmware
    - amdgpu: Update ISP FW for isp v4.1.1
  * Regression on Linux 6.8.0-38: Bluetooth adapter (Intel AX200) stops working after a few suspend/resume cycles (LP: #2073047)
    - Intel Bluetooth: Update firmware file for Intel Bluetooth AX200

 -- Juerg Haefliger <email address hidden> Tue, 13 Aug 2024 09:29:26 +0200

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for linux-firmware has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted linux-firmware (20220329.git681281e4-0ubuntu3.33) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

linux-firmware-raspi/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#linux-firmware

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Juerg Haefliger (juergh)]

User is running and was testing Noble. We also backport this to Jammy for the 6.8 HWE kernel. Unable to test this on Jammy so assuming success.

[Timo Aaltonen (tjaalton)]

Hello Eugene, or anyone else affected,

Accepted linux-firmware into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/20220329.git681281e4-0ubuntu3.34 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.