[SRU] Backport 0.107.1-3 to Ubuntu 23.10

[Danilo Egea Gondolfo]

Stable Release Update for netplan.io 0.107.1-3 to Mantic. This version has many bug fixes and new features we want to make available to users of Ubuntu 23.10. netplan.io has an SRU exception for backporting new versions with new features https://wiki.ubuntu.com/NetplanUpdates

Another SRU, for Jammy, was prepared in a separate LP bug: https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/2058031. As the package is slightly different (due to backwards compatibility patches) and on Mantic we have the integration with Network Manager (which can cause different problems and requires more tests) I thought it would be more appropriate to work on the Mantic SRU in a different LP bug.

[Impact]
This release contains both bug-fixes and new features and we would like to
make sure all of our supported customers have access to these improvements.
The notable ones are:

In terms of new features there were not big changes.
The changes below though will cause backwards compatibility issues if the
user upgrades to the new version of netplan.io, starts to use the new features and
then decides to downgrade for some reason:

* wifi: add support for WPA3-Enterprise
  https://github.com/canonical/netplan/pull/402
  https://bugs.launchpad.net/netplan/+bug/2029876

* wifi: make it possible to have a psk and an eap password simultaneously
  https://github.com/canonical/netplan/pull/416

* parse: allow COMMON_LINK_HANDLERS for VRFs
  https://github.com/canonical/netplan/pull/401

The change below generates a different backend configuration for
wpa-supplicant. Although is will not cause major backwards compatibility
problems:

* wifi: support WPA2 and WPA3 Personal simultaneously
https://github.com/canonical/netplan/pull/404

In addition to all the changes from netplan.io 0.107.1-3, a couple of bug fixes were imported from netplan.io 1.0-1:

* debian/patches/lp2041727: Check if ovsdb-server.service is active before displaying warning
  https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/2041727
  https://github.com/canonical/netplan/pull/421

  This fixes an annoying warning message that many users were seeing when they call "netplan apply".

* d/p/0004-tests-assert-generated-.service-files-in-assert_srio.patch, d/p/0005-tests-sriov-test-if-the-generated-netplan-rebind-ser.patch, d/p/0006-sriov-don-t-generate-duplicate-entries-in-the-rebind.patch: Don't generate duplicate entries in the netplan-sriov-rebind.service
  https://github.com/canonical/netplan/pull/437

  This fixes a duplication problem where "netplan rebind" could process the same SR-IOV interface multiple times.

See the changelog entry below for a full list of changes and bugs.

[Test Plan]
The following development and SRU process was followed:
https://wiki.ubuntu.com/NetplanUpdates

Netplan contains an extensive integration test suite that is ran using
the SRU package for each releases. This test suite's results are available here:
http://autopkgtest.ubuntu.com/packages/n/netplan.io

A successful run is required before the proposed netplan package
can be let into -updates.

The netplan team will be in charge of attaching the artifacts and console
output of the appropriate run to the bug. Netplan team members will not
mark ‘verification-done’ until this has happened.

Testing netplan upgrade

1) Launch a Mantic LXD VM

lxc launch ubuntu:mantic mantic-vm --vm
lxc shell mantic-vm

2) Check Netplan works
netplan get
netplan apply

3) Install Network Manager so we can test the netplan-everywhere integration

apt update
apt install network-manager

4) Change the default renderer to NetworkManager in /etc/netplan/50-cloud-init.yaml

network:
  renderer: NetworkManager
  version: 2
  ethernets:
    enp5s0:
      dhcp4: true

5) Apply the configuration

netplan apply
nmcli con show

6) Add a new connection with nmcli

nmcli con add type dummy ifname dummy0

7) Check that a new 'dummy-device' is present in Netplan

netplan get

8) Add the PPA with the new package

add-apt-repository ppa:danilogondolfo/netplan-sru

apt update ; apt -y upgrade

9) Restart Network Manager so it will load the new libnetplan

systemctl restart NetworkManager

10) Add a new interface using nmcli

nmcli con add type dummy ifname dummy1

11) Check it will show up in the Netplan configuration

netplan get

12) Delete both of the new connections

nmcli con del dummy-dummy0
nmcli con del dummy-dummy1

13) Check they are gone

netplan get

[Where problems could occur]
In order to mitigate the regression potential, the results of the
aforementioned integration tests are attached to this bug.

Regressions in this update could potentially impact the generation of networking configuration and also Network Manager. Note that all the autopkgtests for both netplan.io and network-manager don't show any regression. Also, this version of netplan.io was present in Noble for a few months now and no regressions were reported.

[Other Info]

[Changelog]

  * Backport netplan.io 0.107.1-3 to 23.10 (LP: #2058051):
    - wifi: add support for WPA3-Enterprise (LP: #2029876) (!402)
    - wifi: support WPA2 and WPA3 Personal simultaneously (!404)
    - added mii-monitor-interval example (!411)
    - docs: Add "Contribute Documentation" how-to
    - auth: add support for LEAP and EAP-PWD (!415)
    - tests: Add autopkgtest for (LP: #1959570) (!419)
    - wifi: make it possible to have a psk and an eap password simultaneously
      (!416)
    - doc: Set-up some basic Doxygen project (!423)
    - doc: Make Sphinx to handle autodoxygen project, using breathe (!423)
    - doc: create libnetplan apidoc structure (!423)
    - inc: Start documenting public API (!423)
    - doc: Update 'Netplan everywhere' for 23.10 (!418)
    Bug fixes:
    - test:ovs: Avoid NetworkManager taking contol, breaking a test
    - parse: allow COMMON_LINK_HANDLERS for VRFs (!401)
    - util: don't return a placeholder netdef in the iterator (!406)
    - tunnels/validation: do not error out if "local" is not defined (!407)
    - tests: add some integration tests without the local address (!407)
    - wireguard: ignore empty endpoints (LP: #2038811) (!414)
    - parse: improve the parsing of access-points (LP: #1809994) (!413)
    - wifi: replace the previously defined AP with the new one (!413)
    - doc: spelling check improvements (!417)
    - Fix permissions on folder '/run/NetworkManager/' (!422)
    - cli:try: avoid linting error for type hints (Closes: #1058524) (!422)
    - nm-parse: always read the PSK into the new psk variable (!416)
    - networkd: fix formatting (!424)
    - networkd: replace deprecated CriticalConnection= by KeepConfiguration=
      (!424)
    - networkd: move KeepConfiguration= into [Network] section
    - apply: bring "lo" back up if it's managed by NM (!408)
    - apply: don't assume the NM loopback connection is called "lo" (!408)
  * Drop patches merged upstream:
    - lp1809994/0010-parse-improve-the-parsing-of-access-points-LP-180999.patch
    - lp1809994/0011-wifi-replace-the-previously-defined-AP-with-the-new-.patch
    - lp2039821/0008-wireguard-ignore-empty-endpoints.patch
    - lp2039825/0009-auth-add-support-for-LEAP-and-EAP-PWD.patch
  * Add patches for bug fixes from netplan.io 1.0-1:
    - lp2041727/0004-cli-utils-introduce-systemctl_is_installed-helper.patch
      cli:utils: introduce systemctl_is_installed helper (LP: #2041727) (!421)
    - lp2041727/0005-Update-ovs.py-to-check-if-ovsdb-server.service-is-in.patch
      Check if ovsdb-server.service is active before displaying warning
      (LP: #2041727) (!421)
    - d/p/0007-tests-assert-generated-.service-files-in-assert_srio.patch,
      d/p/0008-tests-sriov-test-if-the-generated-netplan-rebind-ser.patch,
      d/p/0009-sriov-don-t-generate-duplicate-entries-in-the-rebind.patch:
      Don't generate duplicate entries in the netplan-sriov-rebind.service
      (!437)
  * Drop patches not required for 23.10:
    - debian/patches/python-limited-stable-api.patch
  * debian/netplan.io.preinst:
    - This preinst script is intended to cleanup the .pyc files from
      share/netplan/netplan. This directory is supposed to be removed after
      the upgrade from netplan.io 0.106.1 to 0.107, as the Python code
      was moved to it's own python3-netplan package, but it's left behind
      due to Python cached files.
  * Drop changes related to usr-merge and not required for 23.10:
    - debian/netplan-generator.dirs
    - debian/netplan-generator.install
    - debian/netplan-generator.postinst
    - debian/netplan-generator.preinst
  * debian/netplan-generator.lintian-overrides:
    - Drop overrides file. It wasn't really silencing any lintian warnings.

[Danilo Egea Gondolfo (danilogondolfo)]

Attaching autopkgtest logs from netplan.io and network-manager tests.