[SRU] Update to netplan.io 0.104

[Lukas Märdian]

[Impact]
This release contains both bug-fixes and new features and we would like to make sure all of our supported customers have access to those improvements.

The most notable changes (besides usual bug-fixes) incoming in 0.104:
- Enable 'embedded-switch-mode' setting on SmartNICs
- Permit multiple patterns for the driver globs in match (LP: #1918421)
- Improve routing capabilities (LP: #1892272, LP: #1805038)
- Support additional link offload options for networkd (LP: #1771740)
- Handle differing 'ip6-privacy' default value for NetworkManager
- YAML state tracking for DBus API and 'netplan try' (LP: #1943120)
- Support ConfigureWithoutCarrier ('ignore-carrier') for networkd
- Cleanup Makefile, install only public headers
- Netplan 'get' to use the libnetplan parser
- libnetplan consolidation of YAML parsing into the library

ATTENTION:
- We restricted the symbols export to a determined public API starting with v0.104
- libnetplan 0.104 is still fully API & ABI compatible with any consumers that we are aware of in Ubuntu
- We dropped some internal symbols from the API (public headers and binary) that we know have no external consumers:
=> From 0.103: _serialize_yaml, contains_netdef_type, tmp, validate_default_route_consistency
=> From 0.102: cur_filename, netplan_netdef_new
=> From 0.100: address_option_handlers, is_hostname, validate_ovs_target, wireguard_peer_handlers
=> From 0.99: current_file, is_ip4_address, is_ip6_address, missing_id,
missing_ids_found, parser_error, validate_backend_rules, validate_netdef_grammar, yaml_error

See the changelog entry below for a full list of changes and bugs.

[Test Plan]
The following development and SRU process was followed:
https://wiki.ubuntu.com/NetplanUpdates

Netplan contains an extensive integration test suite that is run using
the SRU package for each release. This test suite's results are available here:
http://autopkgtest.ubuntu.com/packages/n/netplan.io

A successful run is required before the -proposed netplan.io package
can be let into -updates.

In addition to the autopkgtests, we want to make sure that on Focal/Core20 the NetworkManager netplan integration plugin (binary) keeps working using this new version of libnetplan and also can be compiled against the new headers without changes required.
We want to use `nmcli con add type etherent && netplan get` to check if new connections can still be created via NetworkManager, and run `make check` to execute NM's unit-tests.

The netplan team will be in charge of attaching the artifacts and console
output of the appropriate run to the bug. Netplan team members will not
mark ‘verification-done’ until this has happened.

[Where problems could occur]
Netplan being a core package, it could impact the whole networking stack of the operating system up to the point where servers would not be reachable anymore after a reboot, due to broken network config being generated by netplan at bootup.

In order to mitigate the regression potential, the results of the aforementioned integration tests are attached to this bug:

Impish:
https://git.launchpad.net/~slyon/+git/files/plain/LP1964481/impish_amd64.log
https://git.launchpad.net/~slyon/+git/files/plain/LP1964481/impish_arm64.log
https://git.launchpad.net/~slyon/+git/files/plain/LP1964481/impish_armhf.log
https://git.launchpad.net/~slyon/+git/files/plain/LP1964481/impish_ppc64el.log
https://git.launchpad.net/~slyon/+git/files/plain/LP1964481/impish_s390x.log

Focal:
https://git.launchpad.net/~slyon/+git/files/plain/LP1964481/focal_amd64.log
https://git.launchpad.net/~slyon/+git/files/plain/LP1964481/focal_arm64.log
https://git.launchpad.net/~slyon/+git/files/plain/LP1964481/focal_armhf.log
https://git.launchpad.net/~slyon/+git/files/plain/LP1964481/focal_ppc64el.log
https://git.launchpad.net/~slyon/+git/files/plain/LP1964481/focal_s390x.log

[Other Info]
The integration test logs will be attached to this bug, once the package has been accepted into -proposed and the tests have been executed on the real infrastructure.

[Changelog]
a4b70e7 cli:apply: fix potential race with rename/creation of netdevs and start networkd if off (LP: #1962095) (#260)
3e522b7 nm:parse-nm: handle differing ip6-privacy default value (#263)
7030ca9 Makefile: install public headers only
1f1e921 doc: avoid super long line in netplan-dbus.md
2e8fa56 Drop unused legacy/compat API (#262)
4d74e2e tests:integration:ethernets: improve eth_dhcp6_off stability
ef56b3a sriov: generate rebind systemd service if needed
57fb971 cli: add sriov rebind method
24a08f6 test:sriov: add eswitch_mode test
508ecac test:sriov: add get_pci_slot_name & PCIDevice unit tests
12fcbf9 test:sriov: fix tests when using libnetplan parser
4c6e430 cli:sriov: Allow setting of eswitch mode on SmartNICs
806a463 parse:netplan: handle 'embedded-switch-mode' & 'delay-virtual-functions-rebind'
6408aea doc: add sriov eswitch mode field descriptions
27a129c test:sriov: update _prepare_sysfs_dir_structure() to make it create PF/VF mocks dynamically
6d77a02 test:sriov: virtfns should be starting at index 0
ed7550d Fix removal of defunct vlan interfaces (LP #1959147) (#256)
ba88341 Have ConfigManager cleanup its temporary directory on destruction (LP #1959729) (#259)
49e5583 util: fix dump_yaml_subtree to not insert NULL at the actual end (#258)
9336443 parse: fix the handle_match_driver type
a42eb78 Permit multiple patterns for the driver globs in match (LP: #1918421) (#202)
331ca01 netplan:cli:get: remove the PyYAML dependency
9a036cc lib:netplan: output all the dirty fields in the YAML dump
0979b0c lib:parser: track which fields are dirty
2b24916 lib:parser: use a more flexible structure for the mapping handlers
0917c59 YAML consolidation prelude: new YAML dump APIs (FR-702) (#251)
d3f81b3 YAML consolidation prelude: High-level libnetplan bindings (FR-702) (#250)
095710a Improve routing capabilities (LP: #1892272) (LP: #1805038) (#248)
3e3c371 YAML consolidation prelude: OVS handling (FR-702) (#249)
519fb50 Consolidate the YAML parsing into libnetplan: low-hanging fruits (FR-702) (#241)
e0a07bc keyfile: do not try to write out unvalidated YAML (LP: #1952967) (#247)
2190567 Add basic integration test (autopkgtest) GA workflow
64e381e Disable temporary address generation when renderer is NetworkManager (LP: #1948027) – change in behavior (#244)
5361619 get/set: ignore empty YAML hints and delete files on "set network=null" (LP: #1946957) (#246)
e19441c dbus: wait for 'netplan try' to be ready (LP: #1949893) (#245)
c6ad8e6 test:bridge:base: give bridge some more time to reach 'state UP'
9a4c2e1 Makefile: fix warning about pandoc title
5a34a7e doc: fix WireGuard writing
9c448f1 Fix ethernets test with network-manage 1.32.10
715cb2c lib: use an explicit parser context (#233)
15c98fe generate:dbus:util: glib 2.70 compat (#235)
930d2a4 tests:regressions: make netplan_try autopkgtest more reliable
53c5a27 Makefile: 'which' is deprecated
a6e8366 cli:apply: initialize self.state (LP: #1949104) (#243)
15de777 Reset offload options (#242)
d4c713d networkd:wpa_supplicant: driver fallback to nl80211 and/or wext (LP: #1814012) (#240)
87c5b04 libnetplan: expose coherent generator APIs (#239)
fd1b3c7 parser: fix some error handling code (#234)
9531dad lib: introduce the notion of NetplanState (#232)
e205f06 parse-nm: Handle missing gateway in keyfile routes, keep dns-search fallback (#238)
bc6eb0b netplan: set: make it possible to unset a whole devtype subtree (LP: #1942930) (FR-1685) (#236)
467e88a API/ABI: restrict the symbol export to a determined public API (#227)
76ae706 Add support for additional `Link` options (#225) (LP: #1771740)
23cb913 lib: unify how constant names are exposed and used (#230)
730fbbd Implement YAML state tracking and use it in the DBus API and netplan-try (LP: #1943120) (FR-1745) (#231)
85ef928 tests: normalize multiple keys on a single dict (#229)
17d3848 parse: add 'default-routes' feature-flag
73d4024 NetDefinition: ownership and cleanup (FR-786) (#228)
d4884cf Add support for networkd ConfigureWithoutCarrier (ignore-carrier) option on interfaces (#215)
e533ddf clarify requirements for a link local route (#226)
c6613b5 Add `.editorconfig` for convenience (#224)
d909f5a GH actions: migrate master -> main
ee0d5df parse: cleaup whitespace
2f0ff65 parse-nm: fix 32bit format string

[Łukasz Zemczak (sil2100)]

This is a big SRU, especially with the API getting a nice refresh and cleanup. I like the idea of abi_compat.c for our stable series. Realistically speaking, we should not have any real users of libnetplan0 besides the bits in the NetworkManager snap, but I think it's good practice not to leave anyone 'hanging' just in case. Some of the symbols that got dropped, I think do not warrant an SONAME bump just yet.

After general review, this looks good to go - but there's one thing that we need to document in the debian/changelog. At least the impish upload includes the cherry-pick of 0002-cli-apply-fix-potential-race-with-rename-creation-of.patch which is not mentioned in the debian/changelog changes - and since it's a delta on top of 0.104, we need this documented.
From smaller nitpicks: changelog mentions changes to Standards-Version and debian/gbp.conf but none are done in this upload. Those could possibly get dropped?

Thanks!

[Łukasz Zemczak (sil2100)]

Okay, I discussed this with Lukas and I think we're good to go. The changelog entry mentions this being a backport from 0.104-0ubuntu2, which was the upload with the patched cherry-pick. So we should be good.

[Łukasz Zemczak (sil2100)]

Hello Lukas, or anyone else affected,

Accepted netplan.io into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.104-0ubuntu2~21.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Lukas, or anyone else affected,

Accepted netplan.io into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.104-0ubuntu2~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted netplan.io (0.104-0ubuntu2~21.10.1) for impish have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/unknown (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/impish/update_excuses.html#netplan.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted netplan.io (0.104-0ubuntu2~20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/unknown (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#netplan.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Lukas Märdian (slyon)]

I've tested netplan.io 0.104-0ubuntu2~21.10.1 from impish-proposed and attached the autopkgtest logs to the bug description. All looking good!

The additional testing of libnetplan/NetworkManager integration is not relevant for Impish as it only applies to Focal/Core20.

The regression in initramfs-tools (arm64) reported by the Ubuntu SRU Bot as a test environment failure and got fixed by retry.

[Lukas Märdian (slyon)]

Focal looking good as well. I've tested netplan.io 0.104-0ubuntu2~20.04.1 from focal-proposed and attached the autopkgtest logs to the bug description.

root@ff-nm:/x# apt list *netplan*
Listing... Done
libnetplan-dev/focal-proposed,now 0.104-0ubuntu2~20.04.1 amd64 [installed]
libnetplan0/focal-proposed,now 0.104-0ubuntu2~20.04.1 amd64 [installed,automatic]
netplan.io/focal-proposed,now 0.104-0ubuntu2~20.04.1 amd64 [installed]

Additional Focal/Core20 testing:
root@ff-nm:/x# make -j8 check # running test-suite on netplan enabled NetworkManager
[...]
============================================================================
Testsuite summary for NetworkManager 1.22.10
============================================================================
# TOTAL: 843
# PASS: 837
# SKIP: 6
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================

root@ff-nm:/x# nmcli con add type ethernet con-name SRU-test
Connection 'SRU-test' (b1938133-9749-4f20-878c-91442f95500c) successfully added.
root@ff-nm:/x# nmcli c
NAME UUID TYPE DEVICE
SRU-test b1938133-9749-4f20-878c-91442f95500c ethernet --
netplan-eth1 8bf25856-ca0b-388e-823c-b898666ab9d2 ethernet --
root@ff-nm:/x# netplan get
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
    eth1:
      renderer: NetworkManager
      dhcp4: true
    NM-b1938133-9749-4f20-878c-91442f95500c:
      renderer: NetworkManager
      match: {}
      dhcp4: true
      dhcp6: true
      ipv6-address-generation: "stable-privacy"
      wakeonlan: true
      networkmanager:
        uuid: "b1938133-9749-4f20-878c-91442f95500c"
        name: "SRU-test"
        passthrough:
          connection.permissions: ""
          ethernet.mac-address-blacklist: ""
          ipv4.dns-search: ""
          ipv6.dns-search: ""
          ipv6.ip6-privacy: "-1"
          proxy._: ""
root@ff-nm:/x# ll /etc/netplan/
total 20
drwxr-xr-x 2 root root 4096 Mar 23 12:07 ./
drwxr-xr-x 101 root root 4096 Mar 23 11:56 ../
-rw-r--r-- 1 root root 389 Oct 5 10:08 50-cloud-init.yaml
-rw-r--r-- 1 root root 81 Feb 22 14:16 70-netplan-set.yaml
-rw-r--r-- 1 root root 551 Mar 23 12:07 90-NM-b1938133-9749-4f20-878c-91442f95500c.yaml

The regression in initramfs-tools (arm64) reported by the Ubuntu SRU Bot as a test environment failure and got fixed by retry.

[Launchpad Janitor (janitor)]

This bug was fixed in the package netplan.io - 0.104-0ubuntu2~21.10.1

---------------
netplan.io (0.104-0ubuntu2~21.10.1) impish; urgency=medium

  * Backport netplan.io 0.104-0ubuntu2 to 20.04 (LP: #1964481)
    - Enable 'embedded-switch-mode' setting on SmartNICs
    - Permit multiple patterns for the driver globs in match
    - Improve routing capabilities
    - Support additional link offload options for networkd
    - Handle differing 'ip6-privacy' default value for NetworkManager
    - YAML state tracking for DBus API and 'netplan try'
    - Support ConfigureWithoutCarrier ('ignore-carrier') for networkd
    - Cleanup Makefile, install only public headers
    - Netplan 'get' to use the libnetplan parser
    - libnetplan:
      + introduce the notion of NetplanState
      + use an explicit parser context
      + expose coherent generator APIs
      + improve overall error handling
      + consolidation of YAML parsing into the library
    ATTENTION:
    - Restrict the symbol export to a determined public API
      + We dropped some internal symbols from the API that we know have no
        external consumers, see upstream changelog for list of dropped symbols
  * Update debian/gbp.conf
  * Use Standards-Version 4.5.1 in debian/control
  * Add d/p/multi-driver-match-compat.patch for match.driver backwards compat

 -- Lukas Märdian <email address hidden> Fri, 11 Mar 2022 18:04:51 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for netplan.io has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package netplan.io - 0.104-0ubuntu2~20.04.1

---------------
netplan.io (0.104-0ubuntu2~20.04.1) focal; urgency=medium

  * Backport netplan.io 0.104-0ubuntu2 to 20.04 (LP: #1964481)
    - Enable 'embedded-switch-mode' setting on SmartNICs
    - Permit multiple patterns for the driver globs in match
    - Improve routing capabilities
    - Support additional link offload options for networkd
    - Handle differing 'ip6-privacy' default value for NetworkManager
    - YAML state tracking for DBus API and 'netplan try'
    - Support ConfigureWithoutCarrier ('ignore-carrier') for networkd
    - Cleanup Makefile, install only public headers
    - Netplan 'get' to use the libnetplan parser
    - libnetplan:
      + introduce the notion of NetplanState
      + use an explicit parser context
      + expose coherent generator APIs
      + improve overall error handling
      + consolidation of YAML parsing into the library
    ATTENTION:
    - Restrict the symbol export to a determined public API
      + We dropped some internal symbols from the API that we know have no
        external consumers, see upstream changelog for list of dropped symbols
  * Add d/p/0003-Mute-gateway4-6-deprecation-warnings.patch:
    - Do not show a deprecation warning on the backported version
  * Add d/p/0006-cli-apply-give-some-extra-time-for-networkctl-reload.patch:
    - To support networkctl reload/reconfigure on older versions of sd-networkd
  * Add d/p/test-ethernets-stability.patch to improve autopkgtest reliability
  * Add d/p/multi-driver-match-compat.patch for match.driver backwards compat
  * Update systemd dependency to >= 245.4-4ubuntu3.8 for activation-mode
  * Update debian/gbp.conf
  * Use debhelper-compat 12 and Standards-Version 4.5.0 in debian/control
  * Keep riscv64 build-time tests disabled

 -- Lukas Märdian <email address hidden> Mon, 14 Mar 2022 12:41:44 +0100