[Impact]
* Due to changes in systemd's networkctl output networkd-dispatcher 2.0 contains a bug, producing invalid keys/values inside the JSON it provides to its scripts
* e.g.: json={"A": ["ress: 119.224.106.22 (DHCP4)"], ...}
* Therefore the scripts cannot access the relevant network configuration state
[Test Plan]
* Run systemd v244+ (e.g. in a LXD container)
* Make sure you got an IP address via DHCP:
* # networkctl status eth0
● 120: eth0
Link File: /usr/lib/systemd/network/99-default.link
Network File: /run/systemd/network/10-netplan-eth0.network
Type: ether
State: routable (configured)
Driver: veth
HW Address: 00:16:3e:0b:d7:74 (Xensource, Inc.)
MTU: 1500 (min: 68, max: 65535)
Queue Length (Tx/Rx): 1/1
Auto negotiation: no
Speed: 10Gbps
Duplex: full
Port: tp
Address: 10.238.94.146 (DHCP4)
fd42:7213:f20e:bd74:216:3eff:fe0b:d774
fe80::216:3eff:fe0b:d774
Gateway: 10.238.94.1 (Xensource, Inc.)
fe80::216:3eff:fe0f:ee29 (Xensource, Inc.)
DNS: 10.238.94.1
fd42:7213:f20e:bd74::1
fe80::216:3eff:fe0f:ee29
Search Domains: lxd
Activation Policy: up
* # cat /etc/networkd-dispatcher/routable.d/00run.sh
#!/bin/bash
env > /tmp/out.log
* chmod +x /etc/networkd-dispatcher/routable.d/00run.sh
* # networkd-dispatcher --run-startup-triggers
* cat /tmp/out.log
* Check json= ENV to verify if it contains the correct IPv4 address (without "(DHCP4)" suffix).
[Where problems could occur]
* This was broken in 2.0 and the fix changes the behavior back to what was originally intended.
* If people adopted to the broken environment variable, to somehow parse the information they need from the broken JSON, this change will most probably break their networkd-dispatcher scripts (again).
[Other Info]
* The upstream version 2.1 just fixes the parsing of current networkctl output (in addition to a few testing and documentation improvements)
* The Debian revision -2 fixes the parsing of the "(DHCP4 ...)" suffix
* The 2.0.1 -> 2.1 version bump does not contain any relevant changes other than the fix for this bug:
The comparison (https://gitlab.com/craftyguy/networkd-dispatcher/-/compare/2.0.1...2.1) shows that it contains “7 changed files with 52 additions and 35 deletions”:
* Several fixes to Gitlab CI YAML – not relevant
* Increased logging in tests/ – not relevant
* Documentation of already existing features (README & manpage) – not relevant
* Update of copyright year – not relevant
* Update tox.ini & Gitlab CI for py38 test environment – not relevant
* fix get_networkctl_status parse error if align of output changed – the RELEVANT fix for this bug
=== Original description ===
I just reinstalled my RPI3 router from Eoan to Focal but a lot of things about networkd broke. I'm not sure whether they are related or not.
The first thing I noticed was, that I did not get my upstream DNS servers. I use a dispatcher script to extract the information from the json environment variable when my uplink becomes routable. Using a test script to dump the environment I found this:
IP6_ADDRS=
PWD=/
json={"A": ["ress: 119.224.106.22 (DHCP4)"], "AdministrativeState": "configuring", "Auto negot": ["tion: yes", "peed: 100Mbps", "plex: full", "Port: mii"], "G": ["eway: 119.224.106.21", "DNS: 202.180.64.10", "202.180.64.11", ""], "HW A": ["ress: b8:27:eb:b6:a5:f4 (Raspberry Pi Foundation)", "MTU: 1500 (max: 65535)"], "InterfaceName": "wan", "Jun 20 00:01:49": ["mnus systemd-networkd[534]: wan: netdev ready", "mnus systemd-networkd[534]: wan: Link UP", "mnus systemd-networkd[534]: wan: Gained carrier"], "Jun 20 00:01:50": ["mnus systemd-networkd[534]: wan: Lost carrier"], "Jun 20 00:01:51": ["mnus systemd-networkd[534]: wan: Gained carrier"], "Jun 20 00:01:53": ["mnus systemd-networkd[534]: wan: DHCPv4 address 119.224.106.22/30 via 119.224.106.21"], "Lin": ["File: /usr/lib/systemd/network/99-default.link"], "Networ": ["File: /etc/systemd/network/07-wan.network", "Type: vlan", "tate: routable (configured)", "iver: 802.1Q VLAN Support"], "OperationalState": "routable", "Queue Length (": ["/Rx): 1/1"], "State": "routable (configuring)", "Type": "vlan"}
AdministrativeState=configuring
IFACE=wan
LANG=en_NZ.UTF-8
INVOCATION_ID=384cc7de4e8d45e48ecb2ecf3cb55508
IP_ADDRS=
ESSID=
ADDR=
NOTIFY_SOCKET=/run/systemd/notify
SHLVL=1
STATE=routable
JOURNAL_STREAM=9:35403
OperationalState=routable
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
networkd_dispatcher_args=--run-startup-triggers
Note that the script worked perfectly fine on Eoan.
Also when I tried to get the dump above, I ran the following command after installing the test-script:
ip link delete wan type vlan; systemctl restart systemd-networkd
in order to force networkd to recreate and reconfigure my uplink interface. Doing this resulted in all connectivity with the router being lost (up- and downstream). After a restart, I found the following in the networkd journal indicating that networkd had in fact crashed.
Jun 19 23:59:58 tumnus systemd-networkd[539]: wan: Link DOWN
Jun 19 23:59:58 tumnus systemd-networkd[539]: wan: Lost carrier
Jun 19 23:59:58 tumnus systemd-networkd[539]: wan: DHCP lease lost
Jun 19 23:59:58 tumnus systemd[1]: Stopping Network Service...
Jun 19 23:59:58 tumnus systemd[1]: systemd-networkd.service: Succeeded.
Jun 19 23:59:58 tumnus systemd[1]: Stopped Network Service.
Jun 19 23:59:58 tumnus systemd[1]: Starting Network Service...
Jun 19 23:59:59 tumnus systemd-networkd[1986]: private: netdev ready
Jun 19 23:59:59 tumnus systemd-networkd[1986]: Enumeration completed
Jun 19 23:59:59 tumnus systemd[1]: Started Network Service.
Jun 19 23:59:59 tumnus systemd-networkd[1986]: private: netdev exists, using existing without changing its parameters
Jun 20 00:00:00 tumnus systemd[1]: systemd-networkd.service: Main process exited, code=dumped, status=11/SEGV
Jun 20 00:00:00 tumnus systemd[1]: systemd-networkd.service: Failed with result 'core-dump'.
Jun 20 00:00:00 tumnus systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 1.
Jun 20 00:00:00 tumnus systemd[1]: Stopped Network Service.
Jun 20 00:00:00 tumnus systemd[1]: Starting Network Service...
Jun 20 00:00:01 tumnus systemd-networkd[2034]: private: netdev ready
Jun 20 00:00:01 tumnus systemd-networkd[2034]: Enumeration completed
Jun 20 00:00:01 tumnus systemd[1]: Started Network Service.
Jun 20 00:00:01 tumnus systemd-networkd[2034]: private: netdev exists, using existing without changing its parameters
Jun 20 00:00:01 tumnus systemd-networkd[2034]: wan: Link UP
Jun 20 00:00:01 tumnus systemd-networkd[2034]: wan: Gained carrier
Jun 20 00:00:01 tumnus systemd-networkd[2034]: private: Could not join netdev: Can not enslave a bridge to a bridge. Too many levels of symbolic links
Jun 20 00:00:01 tumnus systemd-networkd[2034]: private: Failed
Jun 20 00:00:01 tumnus systemd-networkd[2034]: wan: DHCPv4 address 119.224.106.22/30 via 119.224.106.21
Jun 20 00:01:22 tumnus systemd[1]: Stopping Network Service...
Jun 20 00:01:22 tumnus systemd-networkd[2034]: wan: DHCP lease lost
Jun 20 00:01:22 tumnus systemd[1]: systemd-networkd.service: Succeeded.
Jun 20 00:01:22 tumnus systemd[1]: Stopped Network Service.
-- Reboot --
However, no crash file had been generated.
The system is running systemd 245.4-4ubuntu3.1 and networkd-dispatcher 2.0.1-1.
Ok, it looks like the networkd crash & error message about enslaving a bridge to a bridge was caused by the fact that the network file for one of the bridge ports had a Match on MACAddress and came before the bridge's own network file. Therefore, on restarting networkd matched the port network file to the bridge itself (which had inherited that ports mac address) and hence tried to enslave the bridge to itself with obviously bad consequences.
It did not used to do that on Eoan, but changing the ports Match section to match on the interface's name means I can now restart networkd without it crashing (and without me loosing comms).
However, the dispatcher json is still just as mangled as before.