Netplan does not connect to Wireless after `sudo netplan apply` until reboot

[Pierre Equoy]

Summary
=======

Issue observed using 20200422 images focal-preinstalled-server-arm64+raspi.img.xz and focal-preinstalled-server-armhf+raspi.img.xz on:
- Rapsberry Pi 4 4GB
- Rapsberry Pi 3B

When setting up Wireless network via netplan config on a Raspberry Pi 4 4GB with Ubuntu Server 20.04, the settings are not taken into account until the device is rebooted, because the service created (netplan-wpa-wlan0.service) is in a "inactive (dead)" state.

A workaround is to manually restart the service after applying netplan configuration

$ sudo netplan apply
$ sudo systemctl restart netplan-wpa-wlan0.service

This should be done automatically when a new netplan configuration is applied.

SRU acceptance criteria
=======================

The netplan package is generally covered by an SRU exception, so general testing should suffice. That being said, the fix can be verified explicitly by making sure that WiFi configuration still works: using the -proposed packages on a WiFi-enabled device, craft a netplan yaml config enabling wifi and apply the config with `netplan apply`. Wireless should now be working correctly without any other manual steps.

Steps to reproduce
==================

1. Install Ubuntu Server 20.04 image (focal-preinstalled-server-arm64+raspi.img.xz) on a Raspberry Pi 4 4GB (I haven't tested with Ubuntu Server on other devices).
2. Connect a screen and a USB keyboard, or connect to the device via serial console (UART).
3. By default, the device is not connected to the network yet:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether dc:a6:32:57:b9:ba brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether dc:a6:32:57:b9:bb brd ff:ff:ff:ff:ff:ff

$ networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether no-carrier configuring
  3 wlan0 wlan off unmanaged

3 links listed.

4. Create a netplan config file to match your WiFi router config and copy it to /etc/netplan/:

$ cat lab.yaml
network:
  version: 2
  wifis:
    wlan0:
      access-points:
        ubuntu-cert-wpa:
          password: myp455w0rd
      dhcp4: yes

$ sudo cp lab.yaml /etc/netplan/

5. Apply the new configuration:

$ sudo netplan apply

In journalctl:

Apr 23 03:57:38 ubuntu systemd[1]: systemd-networkd-wait-online.service: Succeeded.
Apr 23 03:57:38 ubuntu systemd[1]: Stopped Wait for Network to be Configured.
Apr 23 03:57:38 ubuntu systemd[1]: Stopping Network Service...
Apr 23 03:57:38 ubuntu systemd[1]: systemd-networkd.service: Succeeded.
Apr 23 03:57:38 ubuntu systemd[1]: Stopped Network Service.
Apr 23 03:57:39 ubuntu systemd[1]: Starting Network Service...
Apr 23 03:57:39 ubuntu sudo[1674]: pam_unix(sudo:session): session closed for user root
Apr 23 03:57:39 ubuntu systemd-networkd[1687]: Enumeration completed
Apr 23 03:57:39 ubuntu systemd[1]: Star[ 223.067294] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
ted Network Service.
Apr 23 03:57:39 ubuntu systemd-networkd[1687]: wlan0: IPv6 successfully enabled
Apr 23 03:57:39 ubuntu systemd-networkd[1687]: eth0: IPv6 successfully enabled
Apr 23 03:57:39 ubuntu kernel: brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
Apr 23 03:57:39 ubuntu systemd-networkd[1687]: wlan0: Link UP

$ networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether no-carrier configuring
  3 wlan0 wlan no-carrier configuring

3 links listed.
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether dc:a6:32:57:b9:ba brd ff:ff:ff:ff:ff:ff
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether dc:a6:32:57:b9:bb brd ff:ff:ff:ff:ff:ff

Even if I wait 5 minutes, nothing moves. networkctl tells me wlan0 is "configuring", but nothing happens in the journal.

Trying to restart systemd-networkd doesn't help:

$ sudo systemctl status systemd-networkd
● systemd-networkd.service - Network Service
     Loaded: loaded (/lib/systemd/system/systemd-networkd.service; enabled; ven>
     Active: active (running) since Thu 2020-04-23 03:57:39 UTC; 1min 18s ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
   Main PID: 1687 (systemd-network)
     Status: "Processing requests..."
      Tasks: 1 (limit: 4434)
     CGroup: /system.slice/systemd-networkd.service
             └─1687 /lib/systemd/systemd-networkd

Apr 23 03:57:39 ubuntu systemd[1]: Starting Network Service...
Apr 23 03:57:39 ubuntu systemd-networkd[1687]: Enumeration completed
Apr 23 03:57:39 ubuntu systemd[1]: Started Network Service.
Apr 23 03:57:39 ubuntu systemd-networkd[1687]: wlan0: IPv6 successfully enabled
Apr 23 03:57:39 ubuntu systemd-networkd[1687]: eth0: IPv6 successfully enabled
Apr 23 03:57:39 ubuntu systemd-networkd[1687]: wlan0: Link UP

$ sudo systemctl restart systemd-networkd

Apr 23 03:59:40 ubuntu systemd[1]: Stopping Network Service...
Apr 23 03:59:40 ubuntu systemd[1]: systemd-networkd.service: Succeeded.
Apr 23 03:59:40 ubuntu systemd[1]: Stopped Network Service.
Apr 23 03:59:40 ubuntu systemd[1]: Starting Network Service...
Apr 23 03:59:40 ubuntu systemd-networkd[1722]: Enumeration completed
Apr 23 03:59:40 ubuntu systemd[1]: Started Network Service.
Apr 23 03:59:40 ubuntu sudo[1719]: pam_unix(sudo:session): session closed for user root
Apr 23 03:59:40 ubuntu systemd-networkd[1722]: wlan0: IPv6 successfully enabled
Apr 23 03:59:40 ubuntu systemd-networkd[1722]: eth0: IPv6 successfully enabled

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether dc:a6:32:57:b9:ba brd ff:ff:ff:ff:ff:ff
3: wlan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether dc:a6:32:57:b9:bb brd ff:ff:ff:ff:ff:ff
$ networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether no-carrier configuring
  3 wlan0 wlan no-carrier configuring

3 links listed.

I then found there is a `netplan-wpa-wlan0` systemd service that is "inactive". Restarting it, the connection process happens, the device gets an IP from the DHCP server (my WiFi router), and I can ping the gateway and the outside:

$ sudo systemctl status netplan-wpa-wlan0.service
● netplan-wpa-wlan0.service - WPA supplicant for netplan wlan0
     Loaded: loaded (/run/systemd/system/netplan-wpa-wlan0.service; enabled-run>
     Active: inactive (dead)

$ sudo systemctl restart netplan-wpa-wlan0.service

Apr 23 04:01:25 ubuntu wpa_supplicant[1752]: Successfully initialized wpa_supplicant
Apr 23 04:01:28 ubuntu wpa_supplicant[1752]: wlan0: Trying to associate with SSID 'ubuntu-cert-wpa'
Apr 23 04:01:30 ubuntu wpa_supplicant[1752]: wlan0: CTRL-EVENT-ASSOC-REJECT bssid=00:00:00:00:00:00 status_code=16
Apr 23 04:01:33 ubuntu wpa_supplicant[1752]: wlan0: Trying to associate with SSID 'ubuntu-cert-wpa'
Apr 23 04:01:35 ubuntu systemd-udevd[760]: Network interface NamePolicy= disabled on kernel command line, ignoring.
Apr 23 04:01:35 ubuntu wpa_supplicant[1752]: wlan0: Associated with 04:d9:f5:b4:f8:b8
Apr 23 04:01:35 ubuntu wpa_supplicant[1752]: wlan0: CTRL-EVENT-CONNECTED - Connection to 04:d9:f5:b4:f8:b8 completed [id=0 id_str=]
Apr 23 04:01:35 ubuntu wpa_supplicant[1752]: wlan0: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
Apr 23 04:01:35 ubuntu kernel: IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
Apr 23 04:01:35 ubuntu systemd-networkd[1722]: wlan0: Gained carrier
Apr 23 04:01:35 ubuntu systemd-networkd[1722]: wlan0: Connected WiFi access point: ubuntu-cert-wpa (04:d9:f5:b4:f8:b8)
Apr 23 04:01:36 ubuntu systemd-networkd[1722]: wlan0: DHCPv4 address 192.168.1.228/24 via 192.168.1.1
Apr 23 04:01:36 ubuntu dbus-daemon[1297]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.18' (uid=100 pid=1722 comm="/lib/systemd/systemd-networkd " label="unconfined")
Apr 23 04:01:36 ubuntu systemd-timesyncd[1209]: Network configuration changed, trying to establish connection.
Apr 23 04:01:36 ubuntu systemd[1]: Starting Hostname Service...
Apr 23 04:02:02 ubuntu systemd-timesyncd[1209]: Initial synchronization to time server 91.189.94.4:123 (ntp.ubuntu.com).
Apr 23 04:02:02 ubuntu dbus-daemon[1297]: [system] Successfully activated service 'org.freedesktop.hostname1'
Apr 23 04:02:02 ubuntu systemd[1]: Started Hostname Service.
Apr 23 04:02:03 ubuntu systemd-networkd[1722]: wlan0: Gained IPv6LL

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether dc:a6:32:57:b9:ba brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether dc:a6:32:57:b9:bb brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.228/24 brd 192.168.1.255 scope global dynamic wlan0
       valid_lft 86344sec preferred_lft 86344sec
    inet6 fe80::dea6:32ff:fe57:b9bb/64 scope link
       valid_lft forever preferred_lft forever

$ networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether no-carrier configuring
  3 wlan0 wlan routable configured

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: netplan.io 0.99-0ubuntu1
ProcVersionSignature: User Name 5.4.0-1008.8-raspi 5.4.29
Uname: Linux 5.4.0-1008-raspi aarch64
ApportVersion: 2.20.11-0ubuntu27
Architecture: arm64
CasperMD5CheckResult: skip
Date: Thu Apr 23 04:14:27 2020
SourcePackage: netplan.io
UpgradeStatus: No upgrade log present (probably fresh install)

[Lukas Märdian (slyon)]

Indeed there seems to be a problem with 'netplan apply'. It tries to activate/start a systemd service unit called "<email address hidden>", while the actual name of the service unit is "netplan-wpa-wlan0.service".

Could you try to verify if the following patch fixes your problem?
https://paste.ubuntu.com/p/qgc6FPP4Gf/

[Lukas Märdian (slyon)]

We're working on a fix upstream: https://github.com/CanonicalLtd/netplan/pull/133

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1874377

[Łukasz Zemczak (sil2100)]

Fixed in 0.99-0ubuntu2. There will be a follow-up fix for other edge cases and upgrades, but for now it should all work.

[Brian Murray (brian-murray)]

Hello Pierre, or anyone else affected,

Accepted netplan.io into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.99-0ubuntu3~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Pierre, or anyone else affected,

Accepted netplan.io into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.99-0ubuntu3~19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted netplan.io (0.99-0ubuntu3~19.10.1) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

netplan.io/0.99-0ubuntu3~19.10.1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#netplan.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted netplan.io (0.99-0ubuntu3~20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.136ubuntu6 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#netplan.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Lukas Märdian (slyon)]

Hey Brian,

for SRU verification I did the testing on Eoan/RPi3.

I booted up a freshly installed Ubuntu Server 19.10 system on a Raspberry Pi 3 (ubuntu-19.10.1-preinstalled-server-arm64+raspi3.img.xz).

Enabled the -proposed pocket. Updated the sources and installed/updated netplan.io:
ii libnetplan0:arm64 0.99-0ubuntu3~19.10.1 arm64 YAML network configuration abstraction runtime library
ii netplan.io 0.99-0ubuntu3~19.10.1 arm64 YAML network configuration abstraction for various backends

Afterwards, I verified wlan0 is down, copied my wifi-enabled "lab.yaml" to /etc/netplan, run `sudo netplan apply`, waited approx. 10 seconds, verified wlan0 is up and connected via DHCPv4.

All working as expected.

Attached is my journalctl log (tail), which also shows this process.

[Lukas Märdian (slyon)]

Hey Brian,

for SRU verification I did the testing on Focal/RPi3.

I booted up a freshly installed Ubuntu Server 20.04 system on a Raspberry Pi 3 (ubuntu-20.04-preinstalled-server-arm64+raspi.img.xz).

Enabled the -proposed pocket. Updated the sources and installed/updated netplan.io:
ii libnetplan0:arm64 0.99-0ubuntu3~20.04.1 arm64 YAML network configuration abstraction runtime library
ii netplan.io 0.99-0ubuntu3~20.04.1 arm64 YAML network configuration abstraction for various backends

Afterwards, I verified wlan0 is down, copied my wifi-enabled "lab.yaml" to /etc/netplan, run `sudo netplan apply`, waited approx. 10 seconds, verified wlan0 is up and connected via DHCPv4.

All working as expected.

Attached is my journalctl log (tail), which also shows this process.

[Launchpad Janitor (janitor)]

This bug was fixed in the package netplan.io - 0.99-0ubuntu3~19.10.1

---------------
netplan.io (0.99-0ubuntu3~19.10.1) eoan; urgency=medium

  * Backport netplan.io 0.99 to 19.10. (LP: #1871825)
  * Include proper upstream fix for "Not connect to WiFi after 'netplan apply'
    (LP: #1874377)

 -- Lukas Märdian <email address hidden> Mon, 04 May 2020 11:48:04 +0200

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for netplan.io has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package netplan.io - 0.99-0ubuntu3~20.04.1

---------------
netplan.io (0.99-0ubuntu3~20.04.1) focal; urgency=medium

  * Backport upstream bug fix for 0.99 to 20.04. (LP: #1871825)
    - Upstream provides an imporved fix, which handles additional edge cases
      for "Not connect to WiFi after 'netplan apply'" (LP: #1874377)

 -- Lukas Märdian <email address hidden> Mon, 04 May 2020 11:37:05 +0200

[DanyChen (danychen)]

I'm still having this issue, I think it might be that when systemd-networkd.service starts the /run/systemd/system/netplan-wpa-xxx.service not generated yet! so it only works after a reboot or manually start it.

[mdyn (tamerlaha-gmail)]

20.04.02 it's still the same

[prionico (prionico)]

I can confirm that this is still an issue in 20.04.05