netplan/systemd-networkd: route metric not applied to routes to the local subnet

[Alberto Contreras]

[SRU TEMPLATE]

[DESCRIPTION]

Cloud-init introduced a feature to configure policy routing on AWS EC2 instances with multiple NICs in
https://github.com/canonical/cloud-init/commit/0ca5f31043e2d98eab31a43d9dde9bdaef1435cb targeting v24.1.

Cloud-init generates the following netplan config:

```
$ cat /etc/netplan/50-cloud-init.yaml
network:
    ethernets:
        ens5:
            dhcp4: true
            dhcp4-overrides: &id001
                route-metric: 100
            dhcp6: true
            dhcp6-overrides: *id001
            match:
                macaddress: 0a:c8:ab:90:c2:fb
            set-name: ens5
        ens6:
            dhcp4: true
            dhcp4-overrides:
                route-metric: 200
                use-routes: true
            dhcp6: false
            match:
                macaddress: 0a:c6:55:a1:dc:3b
            routes:
            - table: 101
                to: 0.0.0.0/0
                via: 192.168.0.1
            - table: 101
                to: 192.168.0.0/20
            routing-policy:
            - from: 192.168.10.212
                table: 101
            set-name: ens6
    version: 2
```

Which renders the following systemd-networkd config files:

```
$ cat 10-netplan-ens5.link
[Match]
MACAddress=0a:c8:ab:90:c2:fb

[Link]
Name=ens5
WakeOnLan=off

$ cat 10-netplan-ens5.network
[Match]
MACAddress=0a:c8:ab:90:c2:fb
Name=ens5

[Network]
DHCP=yes
LinkLocalAddressing=ipv6

[DHCP]
RouteMetric=100
UseMTU=true

$ cat 10-netplan-ens6.link
[Match]
MACAddress=0a:c6:55:a1:dc:3b

[Link]
Name=ens6
WakeOnLan=off

$ cat 10-netplan-ens6.network
[Match]
MACAddress=0a:c6:55:a1:dc:3b
Name=ens6

[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6

[Route]
Destination=0.0.0.0/0
Gateway=192.168.0.1
Table=101

[Route]
Destination=192.168.0.0/20
Scope=link
Table=101

[RoutingPolicyRule]
From=192.168.10.212
Table=101

[DHCP]
RouteMetric=200
UseMTU=true
```

Which configures the instance with the following state in Ubuntu Focal:

```
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:c8:ab:90:c2:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.12.94/20 brd 192.168.15.255 scope global dynamic ens5
       valid_lft 2087sec preferred_lft 2087sec
    inet6 2a05:d012:ea0:c500:6d12:2b20:5fef:a502/128 scope global dynamic noprefixroute
       valid_lft 440sec preferred_lft 130sec
    inet6 fe80::8c8:abff:fe90:c2fb/64 scope link
       valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:c6:55:a1:dc:3b brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.212/20 brd 192.168.15.255 scope global dynamic ens6
       valid_lft 2083sec preferred_lft 2083sec
    inet6 fe80::8c6:55ff:fea1:dc3b/64 scope link
       valid_lft forever preferred_lft forever

$ ip route show
default via 192.168.0.1 dev ens5 proto dhcp src 192.168.12.94 metric 100
default via 192.168.0.1 dev ens6 proto dhcp src 192.168.10.212 metric 200
192.168.0.0/20 dev ens5 proto kernel scope link src 192.168.12.94
192.168.0.0/20 dev ens6 proto kernel scope link src 192.168.10.212
192.168.0.1 dev ens5 proto dhcp scope link src 192.168.12.94 metric 100
192.168.0.1 dev ens6 proto dhcp scope link src 192.168.10.212 metric 200

$ ip rule show
0: from all lookup local
0: from 192.168.10.212 lookup 101
32766: from all lookup main
32767: from all lookup default

$ ip route show table 101
default via 192.168.0.1 dev ens6 proto static onlink
192.168.0.0/20 dev ens6 proto static scope link
```

The issue here is that the instance is not reachable from the same subnet via the private ipv4 of the primary NIC,
packets are routed to egress via ens6 and dropped.

The cause is that interface metrics are not applied to local subnet routes with systemd 245 (245.4-4ubuntu3.23).
On newer systemd versions, as in Jammy, the metrics are correctly applied.
Correcting them manually fixes the issue in Focal.

Expected main route table:

default via 192.168.0.1 dev ens5 proto dhcp src 192.168.12.94 metric 100
default via 192.168.0.1 dev ens6 proto dhcp src 192.168.10.212 metric 200
192.168.0.0/20 dev ens5 proto kernel scope link src 192.168.12.94 metric 100
192.168.0.0/20 dev ens6 proto kernel scope link src 192.168.10.212 metric 200
192.168.0.1 dev ens5 proto dhcp scope link src 192.168.12.94 metric 100
192.168.0.1 dev ens6 proto dhcp scope link src 192.168.10.212 metric 200

It looks like the upstream systemd issue and PR fixing this problem are:

https://github.com/systemd/systemd/issues/928
https://github.com/systemd/systemd/pull/19344

[TESTING]

As described above.

[REGRESSION POTENTIAL]

The backport targets Focal.
The fixing patches are touching network related code, regression potential would regard networking part of systemd,
especially in address configuration.
In particualar:

* https://github.com/systemd/systemd/commit/aa550d2a51b025681ff8399e597338d35f540195
This patch adds sd_netlink_message_append_s* functions and types without modifying existing code.

* https://github.com/systemd/systemd/commit/0e7bb99ff919bf8e6030ab7c3c178b87caf166a2
This one just adds missing address types

* https://github.com/systemd/systemd/commit/c4ff0629dd450a40c5733b759eda08e6a032fae3
This one is adds the RouteMetric option for [Address]. Adds code to address_configure() function.

* https://github.com/systemd/systemd/commit/415deef9c3e97211c862f39aceabf8e1f1485a41#
This one adds the RouteMetric option to [DHCPv4]

[OTHER]

The upstream patches fixing this issue are the following :

https://github.com/systemd/systemd/commit/aa550d2a51b025681ff8399e597338d35f540195
https://github.com/systemd/systemd/commit/0e7bb99ff919bf8e6030ab7c3c178b87caf166a2
https://github.com/systemd/systemd/commit/c4ff0629dd450a40c5733b759eda08e6a032fae3
https://github.com/systemd/systemd/commit/415deef9c3e97211c862f39aceabf8e1f1485a41

They originate in PR [1] and backported for focal in MR [2].
There's also a test package in [3].

[1] https://github.com/systemd/systemd/pull/19344
[2] https://code.launchpad.net/~joalif/ubuntu/+source/systemd/+git/systemd/+ref/lp2055397
[3] https://launchpad.net/~joalif/+archive/ubuntu/systemd-focal

[Alberto Contreras (aciba)]

Marking cloud-init as invalid as I think there is no workaround to fix this issue.
Adding netplan.io for reference and awareness.

After confirmation from systemd-networkd, I wonder if it would be feasible / reasonable to backport https://github.com/systemd/systemd/pull/19344 to focal.

[Alberto Contreras (aciba)]

Attaching cloud-init logs, which include journal logs.

[Alberto Contreras (aciba)]

cloud-init upstream PR adding test coverage for this issue: https://github.com/canonical/cloud-init/pull/4982

[Lukas Märdian (slyon)]

Thanks for the heads-up for Netplan! IIUC this will be fixed by a systemd SRU, so closing it as "Invalid" for Netplan. Please re-open if you feel there is something to do on our side.

[Timo Aaltonen (tjaalton)]

Hello Alberto, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.24 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Alberto Contreras (aciba)]

Thanks for the work here. I can confirm that systemd 245.4-4ubuntu-3.24 properly renders route metrics and fixes the routing issue that this bug refers to. Applying the following diff to cloud-init to install systemd's version from focal-proposed and make extra assertions results in the test passing, see attached log.

```diff
diff --git a/tests/integration_tests/modules/test_hotplug.py b/tests/integration_tests/modules/test_hotplug.py
index 8c7bc7839..82d4a2cd1 100644
--- a/tests/integration_tests/modules/test_hotplug.py
+++ b/tests/integration_tests/modules/test_hotplug.py
@@ -299,7 +299,6 @@ def test_multi_nic_hotplug(setup_image, session_cloud: IntegrationCloud):
         verify_clean_log(log_content)

[e-mail](CURRENT_RELEASE <= FOCAL, reason="See LP: #2055397")
 @pytest.mark.skipif(PLATFORM != "ec2", reason="test is ec2 specific")
 def test_multi_nic_hotplug_vpc(setup_image, session_cloud: IntegrationCloud):
     """Tests that additional secondary NICs are routable from local
@@ -308,6 +307,19 @@ def test_multi_nic_hotplug_vpc(setup_image, session_cloud: IntegrationCloud):
     with session_cloud.launch(
         user_data=USER_DATA
     ) as client, session_cloud.launch() as bastion:
+ assert client.execute("""\
+sudo sh -c "echo 'deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe' >> /etc/apt/sources.list.d/proposed-repositories.list"
+ """).ok
+ assert client.execute("sudo apt update").ok
+ assert client.execute("sudo apt upgrade systemd -y").ok
+ systemd_resp = client.execute("apt policy systemd | grep Installed | cut -d ':' -f 2 | tr -d ' '")
+ assert systemd_resp.ok
+ assert systemd_resp.stdout == "245.4-4ubuntu3.24"
+
+ assert client.execute("cloud-init clean --logs")
+ client.restart()
+ wait_for_cloud_init(client)
+
         ips_before = _get_ip_addr(client)
         primary_priv_ip4 = ips_before[1].ip4
         primary_priv_ip6 = ips_before[1].ip6
@@ -343,18 +355,23 @@ def test_multi_nic_hotplug_vpc(setup_image, session_cloud: IntegrationCloud):
         assert r.ok, r.stdout
         r = bastion.execute(f"ping -c1 {secondary_priv_ip4}")
         assert r.ok, r.stdout
- r = bastion.execute(f"ping -c1 {primary_priv_ip6}")
+ r = bastion.execute(f"ping -c3 {primary_priv_ip6}")
         assert r.ok, r.stdout
- r = bastion.execute(f"ping -c1 {secondary_priv_ip6}")
+ r = bastion.execute(f"ping -c3 {secondary_priv_ip6}")
         assert r.ok, r.stdout

+ ip_route_show = client.execute("ip route show")
+ assert ip_route_show.ok, ip_route_show.stderr
+ for route in ip_route_show.splitlines():
+ assert "metric" in route, "Expected metric to be configured in route"
+
         # Remove new NIC
         client.instance.remove_network_interface(secondary_priv_ip4)
         _wait_till_hotplug_complete(client, expected_runs=2)

         # ping to primary NIC works
         assert bastion.execute(f"ping -c1 {primary_priv_ip4}").ok
- assert bastion.execute(f"ping -c1 {primary_priv_ip6}").ok
+ assert bastion.execute(f"ping -c3 {primary_...

[Alberto Contreras (aciba)]

Cloud-init PR to re-enable the test: https://github.com/canonical/cloud-init/pull/5492

[Ioanna Alifieraki (joalif)]

# VERIFICATION FOCAL

#### WITHOUT THE FIX

$ dpkg -l | grep systemd
ii dbus-user-session 1.12.16-2ubuntu2.3 amd64 simple interprocess messaging system (systemd --user integration)
ii libnss-systemd:amd64 245.4-4ubuntu3.23 amd64 nss module providing dynamic user and group name resolution
ii libpam-systemd:amd64 245.4-4ubuntu3.23 amd64 system and service manager - PAM module
ii libsystemd0:amd64 245.4-4ubuntu3.23 amd64 systemd utility library
ii systemd 245.4-4ubuntu3.23 amd64 system and service manager
ii systemd-sysv 245.4-4ubuntu3.23 amd64 system and service manager - SysV links
ii systemd-timesyncd 245.4-4ubuntu3.23 amd64 minimalistic service to synchronize local time with NTP servers

$ ip route show
default via 192.168.96.1 dev ens5 proto dhcp src 192.168.108.150 metric 100
default via 192.168.122.1 dev enp1s0 proto dhcp src 192.168.122.192 metric 100
default via 192.168.96.1 dev ens6 proto dhcp src 192.168.107.159 metric 200
192.168.96.0/20 dev ens5 proto kernel scope link src 192.168.108.150
192.168.96.0/20 dev ens6 proto kernel scope link src 192.168.107.159
192.168.96.1 dev ens5 proto dhcp scope link src 192.168.108.150 metric 100
192.168.96.1 dev ens6 proto dhcp scope link src 192.168.107.159 metric 200
192.168.122.0/24 dev enp1s0 proto kernel scope link src 192.168.122.192
192.168.122.1 dev enp1s0 proto dhcp scope link src 192.168.122.192 metric 100

Metric does not appear on subnet

#### WITH THE FIX

$ dpkg -l | grep systemd
ii dbus-user-session 1.12.16-2ubuntu2.3 amd64 simple interprocess messaging system (systemd --user integration)
ii libnss-systemd:amd64 245.4-4ubuntu3.24 amd64 nss module providing dynamic user and group name resolution
ii libpam-systemd:amd64 245.4-4ubuntu3.24 amd64 system and service manager - PAM module
ii libsystemd0:amd64 245.4-4ubuntu3.24 amd64 systemd utility library
ii systemd 245.4-4ubuntu3.24 amd64 system and service manager
ii systemd-sysv 245.4-4ubuntu3.24 amd64 system and service manager - SysV links
ii systemd-timesyncd 245.4-4ubuntu3.24 amd64 minimalistic service to synchronize local time with NTP servers

$ ip route show
default via 192.168.96.1 dev ens5 proto dhcp src 192.168.108.150 metric 100
default via 192.168.122.1 dev enp1s0 proto dhcp src 192.168.122.192 metric 100
default via 192.168.96.1 dev ens6 proto dhcp src 192.168.107.159 metric 200
192.168.96.0/20 dev ens5 proto kernel scope link src 192.168.108.150 metric 100
192.168.96.0/20 dev ens6 proto kernel scope link src 192.168.107.159 metric 200
192.168.96.1 dev ens5 proto dhcp scope link src 192.168....

[Mauricio Faria de Oliveira (mfo)]

The release of this SRU is blocked in failing autopkgtests [1, 2].

1) Ignore the `linux-*` "regressions" as these packages are denylisted (shouldn't run autopkgtests; this seems like a parsing/reporting issue).

2) Please look at the `gvfs` regressions.
It seems the `integreation` test finishes but does not exit, so it keeps running until a timeout. (previous passing runs finished <5 minutes, currently failing ones time out at 2-3 hours).

3) Please look at the `systemd/armhf` regression.
The test case failing is `test_resolved_domain_restricted_dns`, due to a `resolvectl query` failing.
This passes in other archs, so this could use test packages with instrumentation and autopkgtest runs against PPAs with them [3] (I'm happy to trigger those if needed) to figure out why the command fails (get its stdout/stderr in all arches and print it).

Thanks!

[1] https://ubuntu-archive-team.ubuntu.com/pending-sru.html # search for 245.4-4ubuntu3.24
[2] https://ubuntu-archive-team.ubuntu.com/proposed-migration/focal/update_excuses.html#systemd
[3] https://wiki.ubuntu.com/ProposedMigration#Testing_against_a_PPA

[Nick Rosbrook (enr0n)]

The non-linux autopkgtest regressions were resolved with test retries/baseline tests. As previously stated, the linux-* tests should be ignored as they are denylisted. With, I believe the update could be released.

[Mauricio Faria de Oliveira (mfo)]

Thanks, Nick.

There are several autopkgtests positives that are outdated (i.e., focal-updates has newer versions of the tested packages), thus listed in the update_excuses page as blocking [1].

I have requested reruns for all of those, which should clear the `Issues preventing migration` section if all goes well (not expecting failures, as they have all passed with the older/outdated version, but if they fail, the testing was worth it).

Once that is finished, this should be good for release, IMHO.

cheers,
Mauricio

[1] https://ubuntu-archive-team.ubuntu.com/proposed-migration/focal/update_excuses.html#systemd

Generated: 2024.09.09 18:58:58 +0000
...
systemd (245.4-4ubuntu3.23 to 245.4-4ubuntu3.24)
...
Issues preventing migration:
autopkgtest for exim4/4.93-13ubuntu1.10: amd64: Pass, arm64: Pass, armhf: Pass, ppc64el: Pass, s390x: Pass
autopkgtest for linux/5.4.0-189.209: amd64: Pass, arm64: Pass, armhf: Pass, ppc64el: Pass, s390x: Pass
autopkgtest for linux-aws/5.4.0-1128.138: amd64: Pass, arm64: Pass
autopkgtest for linux-aws-5.15/5.15.0-1065.71~20.04.1: amd64: Pass, arm64: Pass
autopkgtest for linux-azure/5.4.0-1133.140: amd64: Pass, arm64: Pass
autopkgtest for linux-azure-5.15/5.15.0-1068.77~20.04.1: amd64: Pass, arm64: Pass
autopkgtest for linux-azure-cvm/blacklisted: amd64: Regression ♻
autopkgtest for linux-bluefield/5.4.0-1088.95: arm64: Pass
autopkgtest for linux-gcp/5.4.0-1132.141: amd64: Pass, arm64: Pass
autopkgtest for linux-gcp-5.15/5.15.0-1062.70~20.04.1: amd64: Pass, arm64: Pass
autopkgtest for linux-gke-5.15/blacklisted: amd64: Regression ♻ , arm64: Regression ♻
autopkgtest for linux-gkeop/5.4.0-1095.99: amd64: Pass
autopkgtest for linux-gkeop-5.15/5.15.0-1048.55~20.04.1: amd64: Pass
autopkgtest for linux-hwe-5.15/5.15.0-116.126~20.04.1: amd64: Pass, arm64: Pass, armhf: Pass, ppc64el: Pass, s390x: Pass
autopkgtest for linux-ibm/5.4.0-1075.80: amd64: Pass
autopkgtest for linux-ibm-5.15/5.15.0-1056.59~20.04.1: amd64: Pass
autopkgtest for linux-intel-iotg-5.15/5.15.0-1060.66~20.04.1: amd64: Pass
autopkgtest for linux-iot/5.4.0-1040.41: amd64: Pass, arm64: Pass
autopkgtest for linux-kvm/5.4.0-1116.123: amd64: Pass
autopkgtest for linux-lowlatency-hwe-5.15/5.15.0-113.123~20.04.1: amd64: Pass, arm64: Pass
autopkgtest for linux-nvidia-tegra-5.15/5.15.0-1025.25~20.04.1: arm64: Pass
autopkgtest for linux-oem-5.14/blacklisted: amd64: Regression ♻
autopkgtest for linux-oracle/5.4.0-1127.136: amd64: Pass, arm64: Pass
autopkgtest for linux-oracle-5.15/5.15.0-1063.69~20.04.1: amd64: Pass, arm64: Pass
autopkgtest for linux-xilinx-zynqmp/5.4.0-1047.51: arm64: Pass
autopkgtest for postgresql-12/12.19-0ubuntu0.20.04.1: amd64: Pass, arm64: Pass, armhf: Pass, i386: Ignored failure, ppc64el: Pass, s390x: Pass
autopkgtest for prometheus-alertmanager/0.15.3+ds-3ubuntu1.1: amd64: Pass, arm64: Pass, armhf: Pass, ppc64el: Pass, s390x: Pass
autopkgtest for snapd/2.63+20.04: amd64: Pass, arm64: Pass, armhf: No test results, ppc64el: Pass, s390x: Pass
autopkgtest for zfs-linux/0.8.3-1ubuntu12.17: amd64: Pass, arm64: Pass, armhf: No test results, ppc64el: Pass, s390x: Pass

[Mauricio Faria de Oliveira (mfo)]

The non-denylisted regressions are now cleared:

systemd (245.4-4ubuntu3.23 to 245.4-4ubuntu3.24)
...
Issues preventing migration:
autopkgtest for linux-azure-cvm/blacklisted: amd64: Regression ♻
autopkgtest for linux-gke-5.15/blacklisted: amd64: Regression ♻ , arm64: Regression ♻
autopkgtest for linux-oem-5.14/blacklisted: amd64: Regression ♻
...

The 4 "Regression" links point to:
"This package is marked to never run. To get the entry removed, contact a member of the Ubuntu Release or Canonical Ubuntu QA team."

I plan to review this SRU in my next shift.

[Mauricio Faria de Oliveira (mfo)]

Today, another two (newer) packages need reruns on on latest versions (clamav, dovecot).
Triggered reruns for them.
Hopefully they may finish before the end of my shift today.

Issues preventing migration:
autopkgtest for clamav/0.103.11+dfsg-0ubuntu0.20.04.1: amd64: Pass, arm64: Pass, armhf: Pass, ppc64el: Pass, s390x: Pass
autopkgtest for dovecot/1:2.3.7.2-1ubuntu3.6: amd64: Pass, arm64: Pass, armhf: Pass, i386: Not a regression, ppc64el: Pass, s390x: Pass
autopkgtest for linux-azure-cvm/blacklisted: amd64: Regression ♻
autopkgtest for linux-gke-5.15/blacklisted: amd64: Regression ♻ , arm64: Regression ♻
autopkgtest for linux-oem-5.14/blacklisted: amd64: Regression ♻

[Mauricio Faria de Oliveira (mfo)]

This still looks good (same as above).

MR for the linux-* denylists in [1] (updates with missing linux-* packages, including the three above).

[1] https://code.launchpad.net/~mfo/britney/+git/britney/+merge/473318

[Mauricio Faria de Oliveira (mfo)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 245.4-4ubuntu3.24

---------------
systemd (245.4-4ubuntu3.24) focal; urgency=medium

  * network: add RouteMetric= setting in [Address] (LP: #2055397)
    This consists of the following upstream commits:
     * sd-netlink: introduce sd_netlink_message_append_s8() and friends
     * sd-netlink: add missing address types
     * network: add RouteMetric= setting in [Address] section
     * network: dhcp4: also apply RouteMetric= setting in [DHCPv4] to prefix route
    Files:
    - debian/patches/lp2055397/0001-sd-netlink-introduce-sd_netlink_message_append_s8-an.patch
    - debian/patches/lp2055397/0002-sd-netlink-add-missing-address-types.patch
    - debian/patches/lp2055397/0003-network-add-RouteMetric-setting-in-Address-section.patch
    - debian/patches/lp2055397/0004-network-dhcp4-also-apply-RouteMetric-setting-in-DHCP.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=10e86da62257cc03dc1d984478cb1c8efc45097d

 -- Ioanna Alifieraki <email address hidden> Mon, 17 Jun 2024 16:29:39 -0400