[SRU] Backport 0.107.1-3 to Ubuntu 22.04

[Danilo Egea Gondolfo]

Stable Release Update for netplan.io 0.107.1-3 to Jammy. This version has many bug fixes and new features we want to make available to users of Ubuntu 22.04. netplan.io has an SRU exception for backporting new versions with new features https://wiki.ubuntu.com/NetplanUpdates

Another SRU, for Mantic, was prepared in a separate LP bug: https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/2058051. As the package is slightly different (due to backwards compatibility patches) and on Mantic we have the integration with Network Manager (which can cause different problems and requires more tests) I thought it would be more appropriate to work on the Mantic SRU in a different LP bug.

[Impact]

This release contains both bug-fixes and new features and we would like to
make sure all of our supported customers have access to these improvements.
The notable ones are:

* The netplan.io package was split up into netplan.io, which contains the CLI tool, netplan-generator, which contains the systemd generator and python3-netplan which contains the libnetplan Python bindings.
  netplan.io was changed to depend on the new packages so they will be automatically installed during upgrades
https://salsa.debian.org/debian/netplan.io/-/commit/08311bc76adda512c9a0d353734d06231520a82e
https://salsa.debian.org/debian/netplan.io/-/commit/8ed803b320a52ec6364f02959cd67332a6553b87

* Support for dummy interfaces (dummy-devices)
  https://github.com/canonical/netplan/pull/361
  https://bugs.launchpad.net/netplan/+bug/1774203

* Support for veth interfaces (virtual-ethernets)
  https://github.com/canonical/netplan/pull/368

* The libnetplan Python bindings implementation was refactored to use python3-cffi and was moved to its own package, python3-netplan.
  https://github.com/canonical/netplan/pull/385

In addition to all the changes from netplan.io 0.107.1-3, a couple of bug fixes were imported from netplan.io 1.0-1:

* debian/patches/lp2041727: Check if ovsdb-server.service is active before displaying warning
  https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/2041727
  https://github.com/canonical/netplan/pull/421

  This fixes an annoying warning message that many users were seeing when they call "netplan apply".

* d/p/0004-tests-assert-generated-.service-files-in-assert_srio.patch, d/p/0005-tests-sriov-test-if-the-generated-netplan-rebind-ser.patch, d/p/0006-sriov-don-t-generate-duplicate-entries-in-the-rebind.patch: Don't generate duplicate entries in the netplan-sriov-rebind.service
  https://github.com/canonical/netplan/pull/437

  This fixes a duplication problem where "netplan rebind" could process the same SR-IOV interface multiple times.

See the changelog entry below for a full list of changes and bugs.

[Test Plan]
The following development and SRU process was followed:
https://wiki.ubuntu.com/NetplanUpdates

Netplan contains an extensive integration test suite that is ran using
the SRU package for each releases. This test suite's results are available here:
http://autopkgtest.ubuntu.com/packages/n/netplan.io

A successful run is required before the proposed netplan package
can be let into -updates.

The netplan team will be in charge of attaching the artifacts and console
output of the appropriate run to the bug. Netplan team members will not
mark ‘verification-done’ until this has happened.

NOTE: there are a few autopkgtests failing on Jammy, but they are not regressions:

  * ip6gre and vti6 tunnel types: after a kernel update on Ubuntu, systemd started to fail to setup these types of tunnels. The problem was narrowed down to systemd and there is a fix for that which is pending SRU: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2037667

  * veth activation with Network Manager: the Network Manager available in Jammy has a bug that prevents it to correctly activate veth interfaces. There is a fix for that pending analysis for a future SRU: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/2032824

We have been using patched packages for systemd and network-manager on Netplan's CI on Github and all the tests pass with the fixes. A PPA is available here https://launchpad.net/~slyon/+archive/ubuntu/netplan-ci

Testing netplan upgrade

1) Launch a LXD VM with Jammy

lxc launch ubuntu:jammy jammy-vm --vm
lxc shell jammy-vm

2) Check netplan is working

netplan apply

Note the annoying warning below:

WARNING:root:Cannot call Open vSwitch: ovsdb-server.service is not running.

The warnings about file permission can be suppressed with `chmod 600 /etc/netplan/*`.

netplan get

3) Add the PPA with the new package and upgrade your packages

add-apt-repository ppa:danilogondolfo/netplan-sru

apt update ; apt -y upgrade

4) Check netplan is still working

netplan apply

netplan get

Also check that the old /usr/share/netplan/netplan directory is gone

ls /usr/share/netplan/netplan

[Where problems could occur]
In order to mitigate the regression potential, the results of the
aforementioned integration tests are attached to this bug.

<TODO: attach test artifacts for every SRU release, not a link as links expire>

As this is a big update, there are changes across most of the code base. In addition to the existing distro patches to ensure backwards compatibility with Jammy, a new one was added: debian/patches/sru-compat/0014-Demote-lacp-rate-validation-error-to-warning-for-bac.patch. This patch converts a new validation error (for an option we were not validating correctly) to a simple warning so configuration files that have the option improperly set will not start to fail to validate after update.

But as usual, any backwards compatibility issue that prevents the new netplan to parse existing YAML files would leave the system without networking configuration after a reboot.

[Other Info]
With the python3-netplan package, the directory /usr/share/netplan/netplan will not be used anymore. Although, due to .pyc cache files, it would not be deleted during upgrade. A new netplan.io.preinst maintainer script was added to remove these files so dpkg will remove the old directory.

[Changelog]

  * Backport netplan.io 0.107.1-3 to 22.04 (LP: #2058031):
    - Support for "dummy" (`dummy-devices`) interfaces (LP: #1774203) (!361)
    - Support for "veth" (`virtual-ethernets`) interfaces (!368)
    - Add Python bindings for libnetplan (!385)
    - netplan: Handle command exceptions (!334)
    - WPA3 (personal) support (LP: #2023238) (!369)
    - Add all the commands to the bash completion file (LP: #1749869) (!326)
    - New submodule for state manipulation (!379)
    - commands/status: show routes from all routing tables (!390)
    - cli:status: Make rich pretty printing optional (!388)
    - libnetplan: expose dhcp4 and dhcp6 properties (!394)
    - Expose macaddress and DNS configuration from the netdef (!395)
    - libnetplan: expose the routes list in the netdef (!397)
    - NetworkManager: Wireguard private key flag support (!371)
    - Add a netplan_parser_load_keyfile() Python binding (!351)
    - keyfile parser: add support for all tunnel types (LP: #2016473) (!360)
    - parse-nm:wg: add support for reading the listen-port property (!372)
    - parse-nm: add support for VRF devices (!398)
    - Vlan keyfile parser support (!370)
    - Netplan docs rework (!333 & !337)
    - docs: Add a short netplan-everywhere howto (!325)
    - doc: make us of sphinx copybutton plugin (!354)
    - doc: Add Ubuntu Code of Conduct 2.0 (!355)
    - doc: Explanation about 00-network-manager-all.yaml (!378)
    - wifi: add support for WPA3-Enterprise (LP: #2029876) (!402)
    - wifi: support WPA2 and WPA3 Personal simultaneously (!404)
    - added mii-monitor-interval example (!411)
    - docs: Add "Contribute Documentation" how-to
    - auth: add support for LEAP and EAP-PWD (!415)
    - tests: Add autopkgtest for (LP: #1959570) (!419)
    - wifi: make it possible to have a psk and an eap password simultaneously
      (!416)
    - doc: Set-up some basic Doxygen project (!423)
    - doc: Make Sphinx to handle autodoxygen project, using breathe (!423)
    - doc: create libnetplan apidoc structure (!423)
    - inc: Start documenting public API (!423)
    - doc: Update 'Netplan everywhere' for 23.10 (!418)
    Bug fixes:
    - Fix FTBFS on Fedora and refresh RPM packaging (!323)
    - parser: validate lacp-rate properly (LP: #1745648) (!324)
    - use meson-make-symlink.sh helper instead of install_symlink() (!327)
    - netplan: cli: fix typo from 'unkown' to 'unknown' (!328)
    - Handle duplication during parser second pass (LP: #2007682) (!329)
    - parse:ovs: Ignore deprecated OpenFlow1.6 protocol (LP: #1963735) (!332)
    - dbus: Build the copy path correctly (!331)
    - tests: add new spread based snapd integration test (!330)
    - Use controlled execution environment, to avoid failure if PATH is unset
      (LP: #1959570) (!336)
    - Some refactoring (!338)
    - netplan: adjust the maximum buffer size to 1MB (!340)
    - parse: use "--" with systemd-escape (!347)
    - docs: fix bridge parameters types and add examples (!346)
    - vrfs: skip policies parsing if list is NULL (LP: #2016427) (!341)
    - networkd: plug a memory leak (!344)
    - libnetplan: don't try to read from a NULL file (!342)
    - nm: return if write_routes() fails (!345)
    - parse: plug a memory leak (!348)
    - parse: set the backend on nm-devices to NM (!349)
    - parse: don't point to the wrong node on validation (!343)
    - rtd: set the OS and Python versions explicitly (!357)
    - Fix 8021x eap method parsing (LP: #2016625) (!358)
    - CI: update canonical/setup-lxd to v0.1.1 (!359)
    - CI: fix dch after adding the new 0.106.1 tag (!364)
    - Provide frequency to wpa_supplicant in adhoc mode (LP: #2020754) (!363)
    - Improve the coverage of the memory leak tests (!365)
    - Fix keyfile parsing of wireguard config (!366)
    - routes: fix metric rendering (LP: #2023681) (!367)
    - CI: add DebCI integration test (!362)
    - CI: initial NetworkManager autopkgtests (!374)
    - parse-nm: handle cloned-mac-address special cases (LP: #2026230) (!376)
    - Improve autopkgtest stability with systemd 253 & iproute 6.4 (!377)
    - Fixes for minor issues (!380)
    - tests:integration: Adopt for systemd v254 (Closes: #1041310) (!381)
    - parse: Downgrade NM passthrough warning to debug (!384)
    - Don't drop files with just global values (LP: #2027584) (!382)
    - Fixing Coverity issues (!383)
    - CLI: Refactoring to avoid namespace clash with public bindings (!387)
    - tests: fix test coverage report with newer python-coverage (!389)
    - github: add a scheduled action to run Coverity (!391)
    - github: only run the coverity workflow on our repository (!392)
    - Addressing a few issues found (!393)
    - Wireguard fixes (!352)
    - Fix a memory leak, an assert and an error message (!350)
    - ovs: don't allow peers with the same name (!353)
    - CI: make use of the canonical/setup-lxd action (!356)
    - test:ovs: Avoid NetworkManager taking contol, breaking a test
    - parse: allow COMMON_LINK_HANDLERS for VRFs (!401)
    - util: don't return a placeholder netdef in the iterator (!406)
    - tunnels/validation: do not error out if "local" is not defined (!407)
    - tests: add some integration tests without the local address (!407)
    - wireguard: ignore empty endpoints (LP: #2038811) (!414)
    - parse: improve the parsing of access-points (LP: #1809994) (!413)
    - wifi: replace the previously defined AP with the new one (!413)
    - doc: spelling check improvements (!417)
    - Fix permissions on folder '/run/NetworkManager/' (!422)
    - cli:try: avoid linting error for type hints (Closes: #1058524) (!422)
    - nm-parse: always read the PSK into the new psk variable (!416)
    - networkd: fix formatting (!424)
    - networkd: replace deprecated CriticalConnection= by KeepConfiguration=
      (!424)
    - networkd: move KeepConfiguration= into [Network] section
    - apply: bring "lo" back up if it's managed by NM (!408)
    - apply: don't assume the NM loopback connection is called "lo" (!408)
    Packaging restructuring:
    - Split netplan-generator into separate package to make the Python
      dependency optional.
    - Split python3-netplan bindings into a separate package
  * Add patches for bug fixes from netplan.io 1.0-1:
    - debian/patches/lp2041727:
      Check if ovsdb-server.service is active before displaying warning
      (LP: #2041727) (!421)
    - d/p/0004-tests-assert-generated-.service-files-in-assert_srio.patch,
      d/p/0005-tests-sriov-test-if-the-generated-netplan-rebind-ser.patch,
      d/p/0006-sriov-don-t-generate-duplicate-entries-in-the-rebind.patch:
      Don't generate duplicate entries in the netplan-sriov-rebind.service
      (!437)
  * Drop patches not required for 22.04:
    - debian/patches/python-limited-stable-api.patch
  * Add patches for SRU backwards compatibility:
    - 0014-Demote-lacp-rate-validation-error-to-warning-for-bac.patch:
      Convert the error to a warning in a new validation for the option
      'lacp-rate' to prevent breaking existing setups
  * debian/control:
    - Drop python3-rich dependency to Suggests
    - Drop build dependency on systemd-dev
  * debian/netplan.io.preinst:
    - This preinst script is intended to cleanup the .pyc files from
      share/netplan/netplan. This directory is supposed to be removed after
      the upgrade from netplan.io 0.106.1 to 0.107.1, as the Python code
      was moved to it's own python3-netplan package, but it's left behind
      due to Python cached files.
  * Drop changes related to usr-merge and not required for 22.04
    - debian/netplan-generator.install
    - debian/netplan-generator.dirs
    - debian/netplan-generator.postinst
    - debian/netplan-generator.preinst
  * d/netplan-generator.lintian-overrides, d/netplan.io.lintian-overrides:
    - Drop overrides file. It wasn't really silencing any lintian warnings.

[Danilo Egea Gondolfo (danilogondolfo)]

Autopkgtests from Jammy. See the description above for an explanation about why there are test failures.

[Gustavo Sanchez (gustavosr98)]

Hi, is there an ETA to when this would land in Jammy?