netplan reparses wifi when bridge member is listed before definition

Bug #1809994 reported by Ryan Harper
44
This bug affects 9 people
Affects Status Importance Assigned to Milestone
netplan.io (Ubuntu)
Status tracked in Noble
Mantic
Fix Released
Undecided
Unassigned
Noble
Fix Released
High
Unassigned

Bug Description

[ Impact ]

Netplan will fail to generate network configuration if it finds the same access-point SSID
more than once in the configuration. This situation will happen if the user defined the
access-point more than once for the same interface in the same or different files and, worse,
if the parser required a second pass through the configuration.

The second scenario is particularly bad as it's hard to identify the problem without understanding
how the parser works.

Multiple definition of resources is common in Netplan and shouldn't lead to failures.

It's important to have this fixes backported to Mantic due to the Netplan + NetworkManager integration.
If the user ends up in this situation, they will not be able to use Network Manager as it will
always fail parse the Netplan configuration.
See https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/1412

This patch fixes the issue by gracefully handling access-point duplication. If the parser ever finds
the same access-point for the same interface, the old one will be discarded and replaced by the new
definition.

[ Test Plan ]

How to reproduce the issue:

1) Launch a LXD VM with Mantic

$ lxc launch ubuntu:mantic mantic --vm
$ lxc shell mantic

Problem one: multipass parsing will process interfaces twice

Create the following configuration

cat > /etc/netplan/10-config.yaml <<EOF
network:
  bridges:
    mybridge:
      interfaces: [enp5s0]
  ethernets:
    enp5s0: {}
  wifis:
    wlp6s0:
      access-points:
        "mywifi":
          password: "mypassword"
EOF

Run "netplan generate". You will get the error below:

Error in network definition: wlp6s0: Duplicate access point SSID 'mywifi'

This scenario is particularly bad because the access-point is clearly not
defined twice. Although, because the parser will walk through the configuration
twice, it will process it again.

Problem two: similar to problem one but with real duplication

Delete the previous file and create the configuration below:

rm /etc/netplan/10-config.yaml

cat > /etc/netplan/10-config.yaml <<EOF
network:
  wifis:
    wlp6s0:
      access-points:
        "mywifi":
          password: "mypassword"
EOF

cat > /etc/netplan/20-second-config.yaml <<EOF
network:
  wifis:
    wlp6s0:
      access-points:
        "mywifi":
          password: "mypassword"
EOF

Run "netplan generate". You will get the same error:

Error in network definition: wlp6s0: Duplicate access point SSID 'mywifi'

The configuration is duplicated, but this shouldn't be a problem to Netplan.

Now update Netplan using the PPA https://launchpad.net/~danilogondolfo/+archive/ubuntu/netplan
and try the steps above again. Both the scenarios should work fine.

add-apt-repository ppa:danilogondolfo/netplan
apt update && apt -y upgrade

[ Where problems could occur ]

These changes affect how the parser handles access-points only so any issues
that it might introduce will manifest while parsing wifi interfaces.

Although, all the unit and integration tests are passing so we don't expect any
changes in behavior on scenarios that were working fine in Netplan. All NetworkManager's
autopkgtests are also passing (and they contain several tests related to Wifi).

[ Other Info ]

Note the this issue was reported back in 2018 and is still present in Netplan.
While it wasn't a priority, with the integration between Netplan and Network Manager
in Mantic, it became urgent.

---- Original bug description ----

1) # lsb_release -rd
Description: Ubuntu 18.04.1 LTS
Release: 18.04

2) # apt-cache policy netplan.io
netplan.io:
  Installed: 0.36.3
  Candidate: 0.40.1~18.04.3
  Version table:
     0.40.1~18.04.3 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
 *** 0.36.3 100
        100 /var/lib/dpkg/status
     0.36.1 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

3) netplan generate creates configuration for a bridge with single member and wifi accesspoint

4) netplan generate returns error, "Duplicate access point SSID"

# netplan --debug generate
DEBUG:command generate: running ['/lib/netplan/generate']
** (generate:5321): DEBUG: 20:44:08.972: Processing input file //etc/netplan/50-cloud-init.yaml..
** (generate:5321): DEBUG: 20:44:08.972: starting new processing pass
** (generate:5321): DEBUG: 20:44:08.972: recording missing yaml_node_t enp5s0
** (generate:5321): DEBUG: 20:44:08.973: wlp6s0: adding wifi AP 'palisades-guest'
** (generate:5321): DEBUG: 20:44:08.973: starting new processing pass
** (generate:5321): DEBUG: 20:44:08.973: wlp6s0: adding wifi AP 'palisades-guest'
Error in network definition //etc/netplan/50-cloud-init.yaml line 18 column 16: wlp6s0: Duplicate access point SSID 'palisades-guest'

% cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    bridges:
        mybridge:
            interfaces: [enp5s0]
            dhcp4: true
    ethernets:
        enp5s0:
            dhcp4: no
    wifis:
        wlp6s0:
            dhcp4: true
            optional: true
            access-points:
                "mywifi":
                    password: "mypass"
    version: 2

Note, if I move the ethernet definition of enp5s0 before the bridge, netplan can parse it.

This appears to be related to parse.c:add_missing_node() which tags it is missing a definition and then reparses. This then attempts to parse wifis a second time and runs into a condition check that's meant to prevent duplicate SSIDs.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: netplan.io 0.36.3
ProcVersionSignature: Ubuntu 4.15.0-36.39-generic 4.15.18
Uname: Linux 4.15.0-36-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.9-0ubuntu7.4
Architecture: amd64
Date: Fri Dec 28 20:40:57 2018
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: netplan.io
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Revision history for this message
Ryan Harper (raharper) wrote :
Changed in netplan.io (Ubuntu):
status: New → Triaged
importance: Undecided → High
assignee: nobody → Mathieu Trudel-Lapierre (cyphermox)
Revision history for this message
Nanzikambe (nanzikambe) wrote :

This impacts bonded interfaces on a fresh install of Ubuntu 18.04.3 LTS

# cat /etc/netplan/50-cloud-init.yaml
network:
    wifis:
        wlx00XXXXXXXXXXXX:
           access-points:
               "guest":
                   mode: ap
                   password: "lol netplan"
           addresses: [192.168.42.0/24]
           optional: true
           match:
              macaddress: xx:xx:xx:xx:xx:xx
    bonds:
        bond0:
            dhcp4: false
            dhcp6: false
            addresses: [192.168.0.2/24]
            nameservers:
               # search: [lab, home]
               addresses: [8.8.8.8, 1.1.1.1]
            interfaces:
            - enp3s0
            - enp5s0
            parameters:
                mode: broadcast
    ethernets:
        enp3s0: {}
        enp5s0: {}
    version: 2

# netplan --debug generate
DEBUG:command generate: running ['/lib/netplan/generate']
** (generate:1388): DEBUG: 17:40:48.166: Processing input file /etc/netplan/50-cloud-init.yaml..
** (generate:1388): DEBUG: 17:40:48.166: starting new processing pass
** (generate:1388): DEBUG: 17:40:48.166: wlx00XXXXXXXXXXXX: adding wifi AP 'guest'
** (generate:1388): DEBUG: 17:40:48.166: recording missing yaml_node_t enp3s0
** (generate:1388): DEBUG: 17:40:48.166: recording missing yaml_node_t enp5s0
** (generate:1388): DEBUG: 17:40:48.166: starting new processing pass
** (generate:1388): DEBUG: 17:40:48.166: wlx00XXXXXXXXXXXX: adding wifi AP 'guest'
/etc/netplan/50-cloud-init.yaml:12:16: Error in network definition: wlx00XXXXXXXXXXXX: Duplicate access point SSID 'guest'
               "guest":
               ^

Revision history for this message
Mikhail Baykov (rusxakep) wrote :

With bridges or vlans interfaces same problem here

Revision history for this message
Guillaume Hain (zedtux) wrote :

I can confirm that bug from my Chef cookbook: https://gitlab.com/pharmony/netplan.io-cookbook.

I fixed it using the given trick of moving the 'bridges' device to the end of the config file.

Lukas Märdian (slyon)
tags: added: fr-3791
Revision history for this message
@AT (tregubovav-dev) wrote :

cloud-init uses "netplan generate" command to convert cloud-init network configuration from data-source to /etc/cloudinit/50-cloud-init.yaml file. Keys in generated file are ordered in alphabetical order (bonds/bridgeas are at the top pf the file and wifis are at the end).
This issues make impossible to deploy devices required to set-up bridges, bonds and wifi interfaces using cloud-init. This is serious blocker.

Revision history for this message
Danilo Egea Gondolfo (danilogondolfo) wrote :

I can confirm it's still an issue.

tags: added: foundations-todo
Revision history for this message
Danilo Egea Gondolfo (danilogondolfo) wrote :

I prepared a fix for this issue https://github.com/canonical/netplan/pull/413

tags: removed: foundations-todo
Changed in netplan.io (Ubuntu):
status: Triaged → Fix Committed
Lukas Märdian (slyon)
Changed in netplan.io (Ubuntu Noble):
assignee: Mathieu Trudel-Lapierre (cyphermox) → nobody
description: updated
Lukas Märdian (slyon)
Changed in netplan.io (Ubuntu Mantic):
status: New → In Progress
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Ryan, or anyone else affected,

Accepted netplan.io into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.107-5ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in netplan.io (Ubuntu Mantic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-mantic
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (netplan.io/0.107-5ubuntu0.2)

All autopkgtests for the newly accepted netplan.io (0.107-5ubuntu0.2) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.142ubuntu15 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#netplan.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package netplan.io - 0.107-5ubuntu2

---------------
netplan.io (0.107-5ubuntu2) noble; urgency=medium

  * d/p/lp1809994/0010-parse-improve-the-parsing-of-access-points-LP-180999.patch.
    Improve access-points parsing. Properly handle access-points if a second
    pass if required in the parser and not report them as duplicates. (LP: #1809994)
  * d/p/lp1809994/0011-wifi-replace-the-previously-defined-AP-with-the-new-.patch.
    Overwrite the access-point object if another access-point with the same
    name is found instead of ignoring the new one. This behavior is closer
    to the merging process Netplan does.

 -- Danilo Egea Gondolfo <email address hidden> Tue, 14 Nov 2023 14:13:31 +0000

Changed in netplan.io (Ubuntu Noble):
status: Fix Committed → Fix Released
Revision history for this message
Danilo Egea Gondolfo (danilogondolfo) wrote :
Download full text (3.3 KiB)

SRU validation on Mantic

netplan.io installed is 0.107-5ubuntu0.1

---
root@mantic:~# dpkg --list | grep netplan
ii libnetplan0:amd64 0.107-5ubuntu0.1 amd64 YAML network configuration abstraction runtime library
ii netplan-generator 0.107-5ubuntu0.1 amd64 YAML network configuration abstraction systemd-generator
ii netplan.io 0.107-5ubuntu0.1 amd64 YAML network configuration abstraction for various backends
ii python3-netplan 0.107-5ubuntu0.1 amd64 YAML network configuration abstraction Python bindings
---

After adding the reproducer from the description, netplan will fail to generate configuration:

cat > /etc/netplan/10-config.yaml <<EOF
network:
  bridges:
    mybridge:
      interfaces: [enp5s0]
  ethernets:
    enp5s0: {}
  wifis:
    wlp6s0:
      access-points:
        "mywifi":
          password: "mypassword"
EOF

---
root@mantic:~# netplan generate

** (generate:2912): WARNING **: 11:31:27.671: Permissions for /etc/netplan/10-config.yaml are too open. Netplan configuration should NOT be accessible by others.
/etc/netplan/10-config.yaml:10:9: Error in network definition: wlp6s0: Duplicate access point SSID 'mywifi'
        "mywifi":
        ^
---

At this point, Network Manager will not work anymore (because it will also call netplan generate):

---
apt install network-manager
...

root@mantic:~# nmcli con add type dummy ifname dummy0
Error: Failed to add 'dummy-dummy0' connection: Message recipient disconnected from message bus without replying

root@mantic:~# journalctl -u NetworkManager
...
Nov 24 11:31:57 mantic NetworkManager[3389]: /etc/netplan/10-config.yaml:10:9: Error in network definition: wlp6s0: Duplicate access point SSID 'mywifi'
Nov 24 11:31:57 mantic NetworkManager[3389]: "mywifi":
Nov 24 11:31:57 mantic NetworkManager[3389]: ^
...
---

Enabling -proposed and upgrading to netplan.io 0.107-5ubuntu0.2:

cat <<EOF >/etc/apt/preferences.d/proposed-updates
Package: src:netplan.io
Pin: release a=$(lsb_release -cs)-proposed
Pin-Priority: 501
EOF

echo "deb http://archive.ubuntu.com/ubuntu mantic-proposed main" >> /etc/apt/sources.list

apt update
apt upgrade

---
root@mantic:~# dpkg --list | grep netplan
ii libnetplan0:amd64 0.107-5ubuntu0.2 amd64 YAML network configuration abstraction runtime library
ii netplan-generator 0.107-5ubuntu0.2 amd64 YAML network configuration abstraction systemd-generator
ii netplan.io 0.107-5ubuntu0.2 amd64 YAML network configuration abstraction for various backends
ii python3-netplan 0.107-5ubuntu0.2 amd64 YAML network configuration abstraction Python bindings
---

Check that netplan generate and nmcli will work fine

---
root@mantic:~# netplan generate

root@mantic:~# nmcli con add type dummy ifname dummy0
Connection 'dummy-dummy0' (2c97024c-35d3-4f2d-932c-33baccae43fb) successfully added.

root@mantic:...

Read more...

tags: added: verification-done-mantic
removed: verification-needed-mantic
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

I think this SRU got enough validation to be considered early.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package netplan.io - 0.107-5ubuntu0.2

---------------
netplan.io (0.107-5ubuntu0.2) mantic; urgency=medium

  * lp1809994/0010-parse-improve-the-parsing-of-access-points-LP-180999.patch:
    Improve access-points parsing. Properly handle access-points if a second
    pass if required in the parser and not report them as duplicates
    (LP: #1809994).
  * lp1809994/0011-wifi-replace-the-previously-defined-AP-with-the-new-.patch:
    Overwrite the access-point object if another access-point with the same
    name is found instead of ignoring the new one. This behavior is closer
    to the merging process Netplan does.

 -- Danilo Egea Gondolfo <email address hidden> Wed, 15 Nov 2023 10:58:49 +0000

Changed in netplan.io (Ubuntu Mantic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for netplan.io has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.