BCM43602 [14e4:43ba] Subsystem [1028:0020]: Upgraded to 20.04 and wifi broke

[Claire Newman]

[Impact]

 * A driver supports FT (IEEE 802.11r-2008 or fast BSS transition (FT),
   also called fast roaming) if it either supports SME or the
   NL80211_CMD_UPDATE_FT_IES command. When selecting AKM suites,
   wpa_supplicant currently doesn't take into account whether or not either
   of those conditions are met. This can cause association failures, e.g.,
   when an AP supports both WPA-EAP and FT-EAP but the driver doesn't
   support FT (wpa_supplicant will decide to do FT-EAP since it is unaware
   the driver doesn't support it).

 * This is known to affect users at least users with popular chip-sets,
   such as BCM20703A1 or BCM4350 (notably present in the popular XPS 13/15
   laptops).

 * The upload fixes the bug by allowing an FT suite to be selected only
   when the driver also supports FT.

[Test Case]

 * requires affected hardware and wpa_supplicant built with CONFIG_IEEE80211R
   and connecting to an AP supporting WPA-EAP and FT-EAP

[Regression Potential]

 * a driver not advertising correctly its support for SME or the
   NL80211_CMD_UPDATE_FT_IES command via the drv_flags would be prevented from
   using FT but would still be able to connect to APs

[Other Info]

 * Upstream fix: https://w1.fi/cgit/hostap/commit/?id=23dc196fde951b3d508f367a603cddffbd053490
 * Initial report: https://bugs.archlinux.org/task/63397
-----------------------

Upgraded to 20.04 and the wifi broke. It was working fine on 18.04. Tried a variety of things with apw but nada. /var/log attached
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.2
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: ubuntu 1438 F.... pulseaudio
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 20.04
InstallationDate: Installed on 2019-07-09 (329 days ago)
InstallationMedia: Ubuntu 18.04.2 LTS "Bionic Beaver" - Release amd64 (20190210)
MachineType: Dell Inc. XPS 15 9550
Package: linux (not installed)
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-33-generic root=UUID=f73bcd37-31c3-4314-81cf-102bfd94a03e ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 5.4.0-33.37-generic 5.4.34
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-33-generic N/A
 linux-backports-modules-5.4.0-33-generic N/A
 linux-firmware 1.187
Tags: focal
Uname: Linux 5.4.0-33-generic x86_64
UpgradeStatus: Upgraded to focal on 2020-05-21 (12 days ago)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 12/12/2019
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.13.1
dmi.board.name: 0N7TVV
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 9
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.13.1:bd12/12/2019:svnDellInc.:pnXPS159550:pvr:rvnDellInc.:rn0N7TVV:rvrA00:cvnDellInc.:ct9:cvr:
dmi.product.family: XPS
dmi.product.name: XPS 15 9550
dmi.product.sku: 06E4
dmi.sys.vendor: Dell Inc.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1881549

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Andy Whitcroft (apw)]

It looks that we are trying to associate having seen the Ubuntu beacon, and after 3/4s we report a status_code=16 response:

    wlp2s0: Trying to associate with SSID 'Ubuntu'
    <info> [1591017293.4711] device (wlp2s0): supplicant interface state: scanning -> associating
    <info> [1591017293.4711] device (p2p-dev-wlp2s0): supplicant management interface state: scanning -> associating
    wlp2s0: CTRL-EVENT-ASSOC-REJECT bssid=00:00:00:00:00:00 status_code=16

Which I believe is an authentication timeout, as if the station did not hear us or us it:

    #define WLAN_STATUS_AUTH_TIMEOUT 16

Of course if the authentication was incorrect it would ignore us.

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[Claire Newman (claire-newman)]

apport information

[You-Sheng Yang (vicamo)]

Had some tests on a XPS 15 9550 as well. With identical kernel and firmware, cannot reproduce this issue so far. All the seem related error messages also appear in the healthy one, the firmware has stayed as it is currently since 2015, and verified bionic/focal installed the same blob. Also tried upgrading it from Bionic to Focal, yet still cannot reproduce it even under a 100 run stress test.

Someone[1] reported that it could be a bug in NetworkManager that it tries to reuse a misconfigured connection, and simply delete that and reconnect will do:

  NetworkManager[767]: <info> [1591110830.1613] device (wlp2s0): state change: need-auth -> failed (reason 'no-secrets', sys-iface-state: 'managed')

So maybe you would like to try find the connection in WiFi settings, click "Forget Connection" in its own settings page. Maybe even remove all the remembered connections as well.

[1]: https://unix.stackexchange.com/questions/420640/unable-to-connect-to-any-wifi-with-networkmanager-due-to-error-secrets-were-req

[Anthony Wong (anthonywong)]

Not conclusive this's a kernel driver issue yet, also affects to "wpa" as wpasupplicant could be related.

[François Guerraz (kubrick)]

Prior to this commit https://w1.fi/cgit/hostap/commit/?id=23dc196fde951b3d508f367a603cddffbd053490 hostapd doesn't check if the driver supports 802.11R aka. FT.

[François Guerraz (kubrick)]

I built a fixed version here if you're willing to trust a stranger on the internet:
https://drive.google.com/file/d/1BrXOZH-E_cw93HBmk7id1LzUJtnYVsQn/view?usp=sharing

[Kai-Heng Feng (kaihengfeng)]

Maybe push it to PPA?

[François Guerraz (kubrick)]

First I would need some feedback that this is indeed fixing the problem.
PPA is for maintaining software outside of the repos, I don't intend to maintain it, the ubuntu team needs to fix this, which they should be able to do easily if there is a patch.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Kai-Heng Feng (kaihengfeng)]

Claire, please test the following PPA, which includes the fix mentioned by François.

https://launchpad.net/~kaihengfeng/+archive/ubuntu/lp1881549

[Anthony Wong (anthonywong)]

I worked with Claire and the wpasupplicant in the above PPA (#31) can fix the issue.

[François Guerraz (kubrick)]

🎉 glad that I could help.

[Sebastien Bacher (seb128)]

Kai, Anthony, François, is anyone wanting to get that uploaded? If so I can help with the that part, note that there is already a SRU in the focal queue so it would need merging if we want to include this change in the next upload

[Kai-Heng Feng (kaihengfeng)]

debdiff for Groovy

[Kai-Heng Feng (kaihengfeng)]

debdiff for Focal

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "wpa_2.9-1ubuntu8.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Kai-Heng Feng (kaihengfeng)]

I missed comment #34.

Seb, what else should I do to merge them?

[Sebastien Bacher (seb128)]

@Kai, it's fine, I uploaded the G-serie one and I'm going to merge to focal and reupload there

[Sebastien Bacher (seb128)]

@Anthony, Kai-Heng, could you help with the SRU description and provide a summary of the testcase to use and the regression potential?

[François Guerraz (kubrick)]

@seb128, as I dealt with it first hand a few month ago with upstream packages

[Sebastien Bacher (seb128)]

@François, it's uploaded now, if you have any idea on how to improve the description for the SRU that would help getting the update accepted (it's likely the SRU team isn't going to approve it until there is a good testcase description and I don't understand enough the problem and which devices it applies to to write one)

[François Guerraz (kubrick)]

(sorry, previous comment is a misclick)

@seb128, as I dealt with it first hand a few month ago with upstream packages (see https://bugs.archlinux.org/task/63397 ) maybe I can shed some light on it and help you write the SRU. This would be my suggestion:

[Impact]

 * A driver supports FT (IEEE 802.11r-2008 or fast BSS transition (FT),
   also called fast roaming) if it either supports SME or the
   NL80211_CMD_UPDATE_FT_IES command. When selecting AKM suites,
   wpa_supplicant currently doesn't take into account whether or not either
   of those conditions are met. This can cause association failures, e.g.,
   when an AP supports both WPA-EAP and FT-EAP but the driver doesn't
   support FT (wpa_supplicant will decide to do FT-EAP since it is unaware
   the driver doesn't support it).

 * This is known to affect users at least users with popular chip-sets,
   such as BCM20703A1 or BCM4350 (notably present in the popular XPS 13/15
   laptops).

 * The upload fixes the bug by allowing an FT suite to be selected only
   when the driver also supports FT.

[Test Case]

 * requires affected hardware and wpa_supplicant built with CONFIG_IEEE80211R
   and connecting to an AP supporting WPA-EAP and FT-EAP

[Regression Potential]

 * a driver not advertising correctly its support for SME or the
   NL80211_CMD_UPDATE_FT_IES command via the drv_flags would be prevented from
   using FT but would still be able to connect to APs

[Other Info]

 * Upstream fix: https://w1.fi/cgit/hostap/commit/?id=23dc196fde951b3d508f367a603cddffbd053490
 * Initial report: https://bugs.archlinux.org/task/63397

[Sebastien Bacher (seb128)]

@François, thanks a lot! I've copied your description at the top of the report now

[Launchpad Janitor (janitor)]

This bug was fixed in the package wpa - 2:2.9-1ubuntu8

---------------
wpa (2:2.9-1ubuntu8) groovy; urgency=medium

  * debian/patches/Check-for-FT-support-when-selecting-FT-suites.patch:
    - Backport upstream fix "Check for FT support when selecting FT suites" to
      fix drivers without FT support (lp: #1881549).

 -- Kai-Heng Feng <email address hidden> Wed, 10 Jun 2020 11:48:39 +0800

[Brian Murray (brian-murray)]

Hello Claire, or anyone else affected,

Accepted wpa into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/wpa/2:2.9-1ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Anthony Hook (anthonyhook)]

I've enabled proposed repository, rebooted, and can confirm my issue is resolved. I can connect to WPA2 networks! Also marked my bug from yesterday as a dup.

[Anthony Wong (anthonywong)]

Also verified on Claire's laptop and the propose package works fine.

[Claire Newman (claire-newman)]

Thanks so much to everyone who helped fix the bug, and to those who
tutored me through the process :-)

On 6/16/20 19:45, Brian Murray wrote:
> Hello Claire, or anyone else affected,
>
> Accepted wpa into focal-proposed. The package will build now and be
> available at https://launchpad.net/ubuntu/+source/wpa/2:2.9-1ubuntu4.1
> in a few hours, and then in the -proposed repository.
>
> Please help us by testing this new package. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
> to enable and use -proposed. Your feedback will aid us getting this
> update out to other Ubuntu users.
>
> If this package fixes the bug for you, please add a comment to this bug,
> mentioning the version of the package you tested, what testing has been
> performed on the package and change the tag from verification-needed-
> focal to verification-done-focal. If it does not fix the bug for you,
> please add a comment stating that, and change the tag to verification-
> failed-focal. In either case, without details of your testing we will
> not be able to proceed.
>
> Further information regarding the verification process can be found at
> https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
> advance for helping!
>
> N.B. The updated package will be released to -updates after the bug(s)
> fixed by this package have been verified and the package has been in
> -proposed for a minimum of 7 days.
>
> ** Changed in: wpa (Ubuntu Focal)
> Status: Confirmed => Fix Committed
>
> ** Tags added: verification-needed verification-needed-focal
>

[Launchpad Janitor (janitor)]

This bug was fixed in the package wpa - 2:2.9-1ubuntu4.1

---------------
wpa (2:2.9-1ubuntu4.1) focal; urgency=medium

  * debian/patches/git_roaming_interface.patch:
    - backport upstream fix 'dbus: Move roam metrics to the correct
      interface', should reduces the number of events (lp: #1879087)
  * debian/NetworkManager/no-mac-addr-change.conf:
    - remove configuration options that aren't needed and were copied to
      the wrong section, fixes warnings in the log (lp: #1826639)
  * debian/patches/Check-for-FT-support-when-selecting-FT-suites.patch:
    - Backport upstream fix "Check for FT support when selecting FT suites"
      to fix drivers without FT support, thanks Kai-Heng Feng (lp: #1881549)

 -- Sebastien Bacher <email address hidden> Wed, 20 May 2020 13:01:53 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for wpa has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.