wireless does not work on boot on RPi 3s

[Brian Murray]

I've setup a wireless config in /etc/netplan/config.yaml and it does not activate on boot. I have to run 'sudo netplan apply' to get my wireless connected. This works fine on an RPi4 but not an RPi3B.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: systemd 245.2-1ubuntu2
ProcVersionSignature: User Name 5.4.0-1006.6-raspi2 5.4.24
Uname: Linux 5.4.0-1006-raspi2 armv7l
ApportVersion: 2.20.11-0ubuntu21
Architecture: armhf
Date: Thu Apr 2 19:02:27 2020
Lspci:

Lsusb:
 Bus 001 Device 007: ID 0424:ec00 Microchip Technology, Inc. (formerly SMSC) SMSC9512/9514 Fast Ethernet Adapter
 Bus 001 Device 006: ID 0424:9514 Microchip Technology, Inc. (formerly SMSC) SMC9514 Hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
 /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
     |__ Port 1: Dev 6, If 0, Class=Hub, Driver=hub/5p, 480M
         |__ Port 1: Dev 7, If 0, Class=Vendor Specific Class, Driver=smsc95xx, 480M
ProcKernelCmdLine: coherent_pool=1M 8250.nr_uarts=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbswap=1 smsc95xx.macaddr=B8:27:EB:FF:6B:0F vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 net.ifnames=0 dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)

[Brian Murray (brian-murray)]

From /var/log/syslog:

ubuntu@ubuntu:~$ grep wlan0 /var/log/syslog
Apr 2 14:18:19 ubuntu systemd[1]: Found device /sys/subsystem/net/devices/wlan0.
Apr 2 14:18:19 ubuntu systemd[1]: Started WPA supplicant for netplan wlan0.
Apr 2 14:18:19 ubuntu kernel: [ 22.355445] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Apr 2 14:18:19 ubuntu wpa_supplicant[1028]: wlan0: Trying to associate with SSID 'javelin7'
Apr 2 14:18:19 ubuntu wpa_supplicant[1028]: wlan0: Associated with 50:c7:bf:15:c0:93
Apr 2 14:18:19 ubuntu wpa_supplicant[1028]: wlan0: CTRL-EVENT-CONNECTED - Connection to 50:c7:bf:15:c0:93 completed [id=0 id_str=]
Apr 2 14:18:19 ubuntu wpa_supplicant[1028]: wlan0: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: Gained IPv6LL
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: Link DOWN
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: Lost carrier
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: Link UP
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: Gained carrier
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: Connected WiFi access point: javelin7 (50:c7:bf:15:c0:93)
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: Re-configuring with /run/systemd/network/10-netplan-wlan0.network
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: IPv6 successfully enabled
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: IPv6 successfully enabled
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: DHCP4 CLIENT: Failed to attach event: Device or resource busy
Apr 2 14:18:19 ubuntu systemd-networkd[1109]: wlan0: Failed
Apr 2 14:18:19 ubuntu systemd-networkd-wait-online[1110]: managing: wlan0
Apr 2 14:18:19 ubuntu systemd-networkd-wait-online[1110]: managing: wlan0
Apr 2 14:18:19 ubuntu systemd-networkd-wait-online[1110]: managing: wlan0
Apr 2 14:18:19 ubuntu systemd-networkd-wait-online[1110]: managing: wlan0

[Dave Jones (waveform)]

Also tested under armhf and arm64 on Raspberry Pi models 3B, 3A+, and 3B+. Results are the same as for bdmurray's tests with armhf on the 3B above.

My initial suspicion was that this was related to the wifi chipset used on the 3B (the BCM43430) to the 4B (the BCM43455). However, given the 3A+ and 3B+ also use the BCM43455 (same as the 4B) it's definitely not that.

Given that the only error ("DHCP4 CLIENT: Failed to attach event...") occurred in systemd-networkd I attempted to downgrade from version 245.2 to 244.3 (the prior release on Ubuntu). This fixes the issue on the 3B and 3A+ under armhf and arm64 (I skipped checking the 3B+ for time at this point).

At this point I'm reasonably confident it's an issue in systemd introduced sometime between 244.3 and 245.2 - and that it may be timing sensitive given that the 4B (which is significantly faster than the 3, 3A+ or 3B+) doesn't show the same symptoms but shares a wifi chipset with the 3A+ and 3B+.

[Dan Streetman (ddstreet)]

commits 08c588d18b1 and 1b43e246020

[Balint Reczey (rbalint)]

@ddstreet thanks, I'll pick those for the next upload (merging upcoming 245.4-2)

[Brian Murray (brian-murray)]

Well I'm seeing a different failure with the systemd from the PPA.

Apr 3 19:47:53 ubuntu systemd-networkd[1112]: wlan0: Link UP
Apr 3 19:47:53 ubuntu systemd-networkd[1112]: wlan0: Gained carrier
Apr 3 19:47:53 ubuntu systemd-networkd[1112]: wlan0: Connected WiFi access point: javelin7 (50:c7:bf:15:c0:93)
Apr 3 19:47:53 ubuntu systemd-networkd[1112]: wlan0: Re-configuring with /run/systemd/network/10-netplan-wlan0.network
Apr 3 19:47:53 ubuntu systemd-networkd[1112]: wlan0: IPv6 successfully enabled
Apr 3 19:47:53 ubuntu systemd-networkd[1112]: wlan0: IPv6 successfully enabled
Apr 3 19:47:53 ubuntu systemd-networkd[1112]: wlan0: DHCP4 CLIENT: Failed to set request flag for MTU: File exists
Apr 3 19:47:53 ubuntu systemd-networkd[1112]: wlan0: Failed

ubuntu@ubuntu:~$ apt-cache policy systemd
systemd:
  Installed: 245.4-2~ubuntu1~rbalint1

[Dave Jones (waveform)]

I can confirm the behaviour Brian's seeing on the Pi 3B on armhf and arm64 with systemd 245.4-2 from the PPA Balint mentioned (and the same error message which appears to be from a chunk of code slightly beyond where we were last time - so we're moving forward at least :).

As before, wifi is working fine on the Pi 4B under armhf and arm64. Interestingly however, the Pi 3B+ (which shares the wifi chip with the 4B but is only slighter faster than the 3B) now has working wifi under armhf, but not arm64 (same error as the 3B on arm64). I was slightly surprised at this, so I double checked it with a couple of reboots - same results each time.

[Balint Reczey (rbalint)]

For the record the packages were from this repo:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3801/+packages

I'm preparing a new systemd upload with other fixes and the two commits suggested by @ddstreet.
I'll try to reproduce the remaining issues seen in #6 and #7 locally after that and target fixing them in another upload.

[Balint Reczey (rbalint)]

@brian-murray @waveform Thanks for the testing!

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 245.4-2ubuntu1

---------------
systemd (245.4-2ubuntu1) focal; urgency=medium

  [ Balint Reczey ]
  * Merge to Ubuntu from Debian unstable
  * Refresh patches
    - Dropped changes:
      * units: Don't mount tracefs in LXC.
    - Remaining changes:
      * Specify Ubuntu's Vcs-Git
        File: debian/control
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=fd832930ef280c9a4a9dda2440d5a46a6fdb6232
      * Enable EFI/bootctl on armhf.
        File: debian/control
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=043122f7d8a1487bfd357e815a6ece1ceea6e7d1
      * Recommend networkd-dispatcher
        File: debian/control
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d1e3b2c7e4757119da0d550b0b3c0a6626a176dc
      * Add conflicts with upstart and systemd-shim
        File: debian/control
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=83ed7496afc7c27be026014d109855f7d0ad1176
      * debian/control: strengthen dependencies.
        File: debian/control
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d1ecf0c372f5212129c85ae60fddf26b2271a1fe
      * debian/rules: Set -Duser-path instead of patching the source for the same result
        File: debian/rules
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=67d2ac5d806b55b0d178fca7e0af08cd9d12580c
      * config: tpm-pcrindex=12, on Ubuntu only.
        File: debian/rules
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=2f8c22d44a619ce8fa17eb0cd7c7e2c601cdbbeb
      * Ubuntu/extra: ship dhclient-enter hook.
        Files:
        - debian/extra/dhclient-enter-resolved-hook
        - debian/rules
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f3398a213f80b02bf3db0c1ce9e22d69f6d56764
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=258893bae8cbb12670e4807636fe8f7e9fb5407a
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0725c1169ddde4f41cacba7af3e546704e2206be
      * udev-udeb: ship modprobe.d snippet to force scsi_mod.scan=sync in d-i.
        Files:
        - debian/extra/modprobe.d-udeb/scsi-mod-scan-sync.conf
        - debian/udev-udeb.install
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=eb6d8a2b9504917abb7aa2c4035fdbb7b98227f7
      * debian/extra/start-udev: Set scsi_mod scan=sync even if it's builtin to the kernel (we previously only set it in modprobe.d)
        Files:
        - debian/extra/start-udev
        https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6b72628f8de991e2c67ac4289fc74daf3abe7d14
      * debian/extra/units/systemd-resolved.service.d/resolvconf.conf:
        drop resolvconf.conf drop-in, resolved integration moved to resolvconf package.
      * debian/extra/wrap_cl.py: add changelog formatter
        Files:
        - debian/extra/wrap_cl.py
        - debian/gbp.conf
        https://git.launchpad.net/~ubuntu-c...

[Balint Reczey (rbalint)]

Only some of the affected configurations are fixed.

[Balint Reczey (rbalint)]

I was able to reproduce the issue on RPI 2 B 1.1 with an USB wifi dongle, too.
Looking into the root cause.

[Balint Reczey (rbalint)]

A test build of systemd with the fix is building in:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3801

[Dave Jones (waveform)]

Confirmed test build in the PPA in comment 13 (245.4-4ubuntu3) fixes the issue on both archs (armhf and arm64) on the Pi 3B, 3A+, and 3B+ (re-tested 4B as well to ensure nothing broken, and it's still working).

[Dan Watkins (oddbloke)]

How does this relate to bug 1870346?

[Balint Reczey (rbalint)]

@daniel-thewatkins I used a config very similar to https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1870346/comments/4 , so LP: #1870346 may be fixed as well with systemd 245.4-4ubuntu3.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 245.4-4ubuntu3

---------------
systemd (245.4-4ubuntu3) focal; urgency=medium

  * dhcp: Allow setting request options again
    link_configure() may be called multiple times on a link causing request options
    set multiple times. (LP: #1870410)
    File: debian/patches/dhcp-Allow-setting-request-options-again.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4c48893eb04b01a2ec62d2d2823a79a9f5cb2b80
  * Don't try calling NULL callback in bus_wait_for_units_clear.
    This fixes a crash most frequently occurring in the Unity session
    (LP: #1870930)
    File: debian/patches/shared-Don-t-try-calling-NULL-callback-in-bus_wait_for_un.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b2ec1a3367cef401f8de745fecc9c385f020be78

 -- Balint Reczey <email address hidden> Wed, 22 Apr 2020 11:04:26 +0200