Unreliable 802.11ac connection on our raspi images

[Łukasz Zemczak]

We were not able to completely pin-point the issue yet, so this is more of a blanket bug for the current issues we are seeing. We don't know where exactly the issue can be located.

While preparing and testing 18.04.4, certification reported issues with our uc18 images on the pi4 with wifi ac tests. One of our engineers (Brian) confirmed it locally with a wifi access point restricted only to 802.11ac. After some additional testing, he noticed that he had the same unreliable symptoms for both the classic and core images, as well as the 19.10.1 classic images. The symptoms seem to vary, from inability to detect 802.11ac APs, inability to connect and/or receive an IP address and no traffic getting through.

We will provide all the information that we have. Raspbian works better in this regard. We have pulled in the wifi firmware bits from Raspbian and using those, and that seems to resolve all flakiness.

[Brian Murray (brian-murray)]

I'm actually getting good results using the updated brcmfmac43455 firmware files which Raspbian is using. To use them I first removed '/lib/firmware/brcm/brcmfmac43455-sdio.raspberrypi*' then put the brcmfmac43455-sdio* files in the same directory and rebooted.

ubuntu@ubuntu:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
ubuntu@ubuntu:~$ ip address show wlan0
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether dc:a6:32:37:37:08 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.111/24 brd 192.168.1.255 scope global dynamic wlan0
       valid_lft 85747sec preferred_lft 85747sec
    inet6 fe80::dea6:32ff:fe37:3708/64 scope link
       valid_lft forever preferred_lft forever
ubuntu@ubuntu:~$ md5sum /lib/firmware/brcm/brcmfmac43455-sdio.*
963eb0d4903040974ee88b4f85cb1f4f /lib/firmware/brcm/brcmfmac43455-sdio.bin
c5aeca0e33de4ae870986c517963fef7 /lib/firmware/brcm/brcmfmac43455-sdio.clm_blob
7b983812a8aee7bf701460436c686226 /lib/firmware/brcm/brcmfmac43455-sdio.txt
ubuntu@ubuntu:~$ dmesg | grep brcmfmac
[ 12.265990] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 12.271828] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 12.284536] usbcore: registered new interface driver brcmfmac
[ 12.355127] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt failed with error -2
[ 12.548549] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 12.579186] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[ 13.578117] brcmfmac: power management disabled
ubuntu@ubuntu:~$ iw dev
phy#0
        Unnamed/non-netdev interface
                wdev 0x2
                addr 52:4f:c2:5b:63:7a
                type P2P-device
                txpower 31.00 dBm
        Interface wlan0
                ifindex 3
                wdev 0x1
                addr dc:a6:32:37:37:08
                ssid Linksys01146_5GHz
                type managed
                channel 44 (5220 MHz), width: 80 MHz, center1: 5210 MHz
                txpower 31.00 dBm

For the record I'm connecting to a Linksys EA7450 wireless access point and I've restricted the 5GHz frequency to 802.11ac only.

[Juerg Haefliger (juergh)]

Is this a problem with the Pi 4 only or across the whole fleet?

[Dave Jones (waveform)]

@juergh given the Pi 3A+, 3B+, and 4B share the same wifi chipset (but not the 3B) I'd expect similar behaviour across those three models, if indeed the firmware is the issue.

[Brian Murray (brian-murray)]

The only other Pi that I currently have is a 3B which does not have 802.11ac capabilities.

https://www.raspberrypi.org/documentation/faqs/#hardware

[Dave Jones (waveform)]

A package with the latest firmware is available from the following PPA:

https://launchpad.net/~waveform/+archive/ubuntu/firmware/+packages

I've now tested this booting and operating wifi on a 3B+ and a 4B against "classic" 2.4GHz wifi, and 5GHz wifi but only 802.11n (as that's all I've got locally). Would be grateful if others could try installing this and see if 802.11ac is any better.

The proposed package also updates the boot firmware to the 2020-02-12 version (as our boot firmware was several months, and versions out of date too).

[Brian Murray (brian-murray)]

I flashed the focal preinstalled image for arm64 and put in an RPi4. I then did a dist-upgrade, setup a netplan config, rebooted (an undocumented step?) and I am not able to connect to the access point. I confirmed that the right firmware is being used:

[ 9.005150] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 9.014444] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 9.025103] usbcore: registered new interface driver brcmfmac
[ 9.280853] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[ 9.301187] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[ 10.310028] brcmfmac: power management disabled

I found this in my journal:

Feb 19 19:28:14 ubuntu wpa_supplicant[1620]: wlan0: Failed to initiate sched scan
Feb 19 19:28:22 ubuntu wpa_supplicant[1620]: wlan0: Failed to initiate sched scan
Feb 19 19:28:29 ubuntu wpa_supplicant[1620]: wlan0: Failed to initiate sched scan
Feb 19 19:28:37 ubuntu wpa_supplicant[1620]: wlan0: Failed to initiate sched scan
Feb 19 19:28:44 ubuntu wpa_supplicant[1620]: wlan0: Failed to initiate sched scan
Feb 19 19:28:52 ubuntu wpa_supplicant[1620]: wlan0: Failed to initiate sched scan
Feb 19 19:29:00 ubuntu wpa_supplicant[1620]: wlan0: Failed to initiate sched scan
Feb 19 19:29:07 ubuntu wpa_supplicant[1620]: wlan0: Failed to initiate sched scan

Perhaps we should test this on 19.10.1 so we aren't adding the additional variable of running Focal.

[Dave Jones (waveform)]

> Perhaps we should test this on 19.10.1 so we aren't adding the additional variable of running Focal.

I've added an Eoan version of the package to the same PPA (https://launchpad.net/~waveform/+archive/ubuntu/firmware/+packages).

[Brian Murray (brian-murray)]

I've tested this today on both 20.04 (with linux-firmware-raspi2 1.20200212-0ubuntu1~ppa1) and 19.10.1 (with linux-firmware-raspi2 1.20200212-0ubuntu1~19.10.1~ppa1) and neither of them were able to connect to the 802.11ac access point.

I see the same journal entries mentioned previously:

Mar 10 20:57:10 ubuntu wpa_supplicant[1090]: wlan0: Failed to initiate sched scan
Mar 10 20:57:18 ubuntu wpa_supplicant[1090]: wlan0: Failed to initiate sched scan

Additionally, I notice that the output of 'iw dev' shows different channels and widths than on Raspbian:

        Interface wlan0
                ifindex 3
                wdev 0x1
                addr dc:a6:32:37:37:08
                type managed
                channel 128 (5640 MHz), width: 20 MHz, center1: 5640 MHz
                txpower 31.00 dBm

[Brian Murray (brian-murray)]

I decided to test this with the armhf version of 20.04 given that Raspbian is armhf only and the results were the same.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware-raspi2 - 1.20200212-0ubuntu1

---------------
linux-firmware-raspi2 (1.20200212-0ubuntu1) focal; urgency=medium

  * New upstream release, 1.20200212
  * Updated wifi firmware to support 802.11ac wifi (LP: #1862760)
  * Added diversions to override linux-firmware's versions of the wifi
    firmware (LP: #1862146)

 -- Dave Jones <email address hidden> Tue, 10 Mar 2020 11:46:49 +0000

[Brian Murray (brian-murray)]

I managed to get my RPi4 to connect to the 802.11ac access point by setting the regulatory domain to US - "iw reg set US". Here's a before and after output of "iw reg get".

ubuntu@ubuntu:~$ iw reg get
global
country 00: DFS-UNSET
        (2402 - 2472 @ 40), (N/A, 20), (N/A)
        (2457 - 2482 @ 20), (N/A, 20), (N/A), AUTO-BW, PASSIVE-SCAN
        (2474 - 2494 @ 20), (N/A, 20), (N/A), NO-OFDM, PASSIVE-SCAN
        (5170 - 5250 @ 80), (N/A, 20), (N/A), AUTO-BW, PASSIVE-SCAN
        (5250 - 5330 @ 80), (N/A, 20), (0 ms), DFS, AUTO-BW, PASSIVE-SCAN
        (5490 - 5730 @ 160), (N/A, 20), (0 ms), DFS, PASSIVE-SCAN
        (5735 - 5835 @ 80), (N/A, 20), (N/A), PASSIVE-SCAN
        (57240 - 63720 @ 2160), (N/A, 0), (N/A)

ubuntu@ubuntu:~$ iw reg get
global
country US: DFS-FCC
        (2402 - 2472 @ 40), (N/A, 30), (N/A)
        (5170 - 5250 @ 80), (N/A, 23), (N/A), AUTO-BW
        (5250 - 5330 @ 80), (N/A, 23), (0 ms), DFS, AUTO-BW
        (5490 - 5730 @ 160), (N/A, 23), (0 ms), DFS
        (5735 - 5835 @ 80), (N/A, 30), (N/A)
        (57240 - 63720 @ 2160), (N/A, 40), (N/A)

However, this is much different than my 20.04 laptop running Ubuntu.

[Brian Murray (brian-murray)]

Here's the "iw reg get" output from a 20.04 laptop.

[ 10:58AM 10014 ] [ bdmurray@speedy:~ ]
 $ iw reg get
global
country 00: DFS-UNSET
        (2402 - 2472 @ 40), (N/A, 20), (N/A)
        (2457 - 2482 @ 20), (N/A, 20), (N/A), AUTO-BW, PASSIVE-SCAN
        (2474 - 2494 @ 20), (N/A, 20), (N/A), NO-OFDM, PASSIVE-SCAN
        (5170 - 5250 @ 80), (N/A, 20), (N/A), AUTO-BW, PASSIVE-SCAN
        (5250 - 5330 @ 80), (N/A, 20), (0 ms), DFS, AUTO-BW, PASSIVE-SCAN
        (5490 - 5730 @ 160), (N/A, 20), (0 ms), DFS, PASSIVE-SCAN
        (5735 - 5835 @ 80), (N/A, 20), (N/A), PASSIVE-SCAN
        (57240 - 63720 @ 2160), (N/A, 0), (N/A)

phy#0 (self-managed)
country US: DFS-UNSET
        (2402 - 2437 @ 40), (6, 22), (N/A), AUTO-BW, NO-HT40MINUS, NO-80MHZ, NO-160MHZ
        (2422 - 2462 @ 40), (6, 22), (N/A), AUTO-BW, NO-80MHZ, NO-160MHZ
        (2447 - 2482 @ 40), (6, 22), (N/A), AUTO-BW, NO-HT40PLUS, NO-80MHZ, NO-160MHZ
        (5170 - 5190 @ 80), (6, 22), (N/A), NO-OUTDOOR, AUTO-BW, IR-CONCURRENT, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5190 - 5210 @ 80), (6, 22), (N/A), NO-OUTDOOR, AUTO-BW, IR-CONCURRENT, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5210 - 5230 @ 80), (6, 22), (N/A), NO-OUTDOOR, AUTO-BW, IR-CONCURRENT, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5230 - 5250 @ 80), (6, 22), (N/A), NO-OUTDOOR, AUTO-BW, IR-CONCURRENT, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5250 - 5270 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5270 - 5290 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5290 - 5310 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5310 - 5330 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5490 - 5510 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5510 - 5530 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5530 - 5550 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5550 - 5570 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5570 - 5590 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5590 - 5610 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5610 - 5630 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5630 - 5650 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5650 - 5670 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5670 - 5690 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5690 - 5710 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5710 - 5730 @ 80), (6, 22), (0 ms), DFS, AUTO-BW, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5735 - 5755 @ 80), (6, 22), (N/A), AUTO-BW, IR-CONCURRENT, NO-HT40MINUS, NO-160MHZ, PASSIVE-SCAN
        (5755 - 5775 @ 80), (6, 22), (N/A), AUTO-BW, IR-CONCURRENT, NO-HT40PLUS, NO-160MHZ, PASSIVE-SCAN
        (5775 - 5795 @ 80), (6, 22), (N/A), AUTO-BW,...

[Brian Murray (brian-murray)]

On Raspbian the output of "iw reg get" matches what I saw after setting "iw reg set US".

[Seth Forshee (sforshee)]

Generally speaking we don't know what country a device is operating in, so we default to the world domain which should be safe throughout the world. The AP may send a hint to the client as to what regulatory domain to use, but many APs do not do this.

Also note that on many desktop systems now the regulatory domain is managed by the wireless card firmware, which is why you may see the domain set there even if the AP does not send a hint.

[Brian Murray (brian-murray)]

Just to be clear this is what iw reg get shows on Rasbian:

pi@raspberrypi:~$ iw reg get
global
country US: DFS-FCC
        (2402 - 2472 @ 40), (N/A, 30), (N/A)
        (5170 - 5250 @ 80), (N/A, 23), (N/A), AUTO-BW
        (5250 - 5330 @ 80), (N/A, 23), (0 ms), DFS, AUTO-BW
        (5490 - 5730 @ 160), (N/A, 23), (0 ms), DFS
        (5735 - 5835 @ 80), (N/A, 30), (N/A)
        (57240 - 63720 @ 2160), (N/A, 40), (N/A)

[Seth Forshee (sforshee)]

Based on some quick googling, I think there's some localization set up on raspbian that must also cause it to set the regulatory domain.

https://www.raspberrypi.org/documentation/configuration/wireless/desktop.md

[Brian Murray (brian-murray)]

I went ahead and ran through the Raspbian setup again and this time I said I was in the UK. After that and rebooting I ran 'iw reg get' and the results showed "country GB: DFS-ETSI".

[Jerry Vonau (jvonau)]

Think this might relate to how netplan constructs its /run/netplan/wpa-<IFACE>.conf file(1). In raspbian(2) users set the country= line in its wpa_supplicant.conf file before starting the service. Perhaps netplan's 'wifis' section could grow the ability to add the country= and (2)ssid_scan= lines based on what is a user setting in network-config(4) that is present on RPi's?

1. https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/1874494

2. https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md

3. https://bugs.launchpad.net/ubuntu/+source/nplan/+bug/1787482

4. https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#3-wifi-or-ethernet

[Jerry Vonau (jvonau)]

Looks like the ssid_scan= is being addressed upstream
https://github.com/CanonicalLtd/netplan/pull/132

[Juerg Haefliger (juergh)]

I'm not able to reproduce this issue but I can't restrict my router to use 802.11ac only, it also uses 802.11n in the 5GHz spectrum. Is there a way to check which standard is in use for a connection or is it possible to restrict which standard to use on the client side?

Brian, are you still seeing this issue with a current focal image and kernel 5.4.0-1009?

FWIW, my log:
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd[1]: Starting Network Service...
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd[1]: Started WPA supplicant for netplan wlan0.
Apr 29 06:17:46 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: Successfully initialized wpa_supplicant
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-networkd[1593]: eth0: Gained IPv6LL
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-networkd[1593]: Enumeration completed
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-timesyncd[1092]: Network configuration changed, trying to establish connection.
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-timesyncd[1092]: Network configuration changed, trying to establish connection.
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd[1]: Started Network Service.
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-networkd[1593]: wlan0: IPv6 successfully enabled
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-networkd[1593]: eth0: IPv6 successfully enabled
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-timesyncd[1092]: Network configuration changed, trying to establish connection.
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-networkd[1593]: eth0: DHCPv4 address 192.168.99.34/24 via 192.168.99.1
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-timesyncd[1092]: Network configuration changed, trying to establish connection.
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-timesyncd[1092]: Network configuration changed, trying to establish connection.
Apr 29 06:17:46 rpi-4b-rev1d1-17cf systemd-timesyncd[1092]: Initial synchronization to time server 192.168.99.1:123 (192.168.99.1).
Apr 29 06:17:46 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-16 retry=1
Apr 29 06:17:47 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-16 retry=1
Apr 29 06:17:48 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-16 retry=1
Apr 29 06:17:49 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-16 retry=1
Apr 29 06:17:50 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-16 retry=1
Apr 29 06:17:51 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-16 retry=1
Apr 29 06:17:52 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-16 retry=1
Apr 29 06:17:53 rpi-4b-rev1d1-17cf kernel: ieee80211 phy0: brcmf_escan_timeout: timer expired
Apr 29 06:17:55 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: wlan0: Trying to associate with SSID 'pi-test'
Apr 29 06:17:58 rpi-4b-rev1d1-17cf iwd[1176]: Unexpected connection related event -- is another supplicant running?
Apr 29 06:17:58 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: wlan0: Associated with e0:28:6d:9e:b9:25
Apr 29 06:17:58 rpi-4b-rev1d1-17cf wpa_supplicant[1598]: wlan0: CTRL-EVENT-CONNECTED - Connection to e0:28:6d:9e:b9:25 completed [id=0 id_str=]
Apr 29 06:...

[Jerry Vonau (jvonau)]

Think the regulatory domain is not being set anywhere as /etc/default/crda has REGDOMAIN= and sudo cat /run/netplan/wpa-wlan0.conf has

ctrl_interface=/run/wpa_supplicant

network={
  ssid="SHEW-DB9990"
  key_mgmt=WPA-PSK
  psk="nope"
}

Might explain 'ieee80211 phy0: brcmf_escan_timeout: timer expired' above.

https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/crda.git/tree/README#n8
For manual changing of regulatory domains use iw (iw reg set) or wpa_supplicant.

[Juerg Haefliger (juergh)]

Maybe related? https://github.com/raspberrypi/linux/issues/3378

[Brian Murray (brian-murray)]

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release

[pcgeek86 (pcgeek86)]

I just tried to set up Ubuntu 20.04.1 LTS Focal Fossa on a Raspberry Pi 3 B+ and ran into this bug. I used the ARM 64-bit image from this page: https://cdimage.ubuntu.com/releases/20.04/release/

I was able to use my 2.4Ghz SSID just fine. Tried switching back to the 5Ghz and failed. Went back to 2.4Ghz for now.

[Juerg Haefliger (juergh)]

What kernel and firmware versions? 20.04.1 is quite old. Have you tried updating?

[True-night (hlpimfalling)]

Hi all, I found that adding the correct region to the /etc/default/crda file resolved my 5GHZ woes on the raspi4b ubuntu 20.10. After a reboot it finally connected to my 5GHZ network. Just putting this out there if it helps anyone.

[Juerg Haefliger (juergh)]

Somewhat related: LP: #1908951