Networkd fails to configure a ethernet network device on a 3.4 kernel

[Simon Fels]

Knowing that a 3.4 based kernel is unsupported we have to get it working for a specific customer.

Systemd itself boots up nicely already but networkd fails to configure a network device with the following configuration file in /etc/systemd/network

$ cat /etc/systemd/network/enp1s0.network
[Match]
Name=enp1s0

[Network]
DNS=8.8.8.8
Address=192.168.178.30
Gateway=192.168.178.1

This gives the following error messages

$ sudo /lib/systemd/systemd-networkd
Enumeration completed
enp1s0: Could not bring up interface: Invalid argument
enp1s0: Could not set route: Invalid argument

ifupdown or NetworkManager is reliable able to get the same interface up and configured.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Pitt (pitti)]

2784 readlinkat(AT_FDCWD, "/sys/devices/virtual/net", 0x40136358, 99) = -1 EINVAL (Invalid argument)
# plus a few more readlinkat()'s failing the same way
2784 writev(2, [{"enp1s0: Could not set route: Inv"..., 45}, {"\n", 1}], 2) = 46

[Martin Pitt (pitti)]

Just to collect all the info, can you please re-run with

  sudo SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd

I assume this is a missing kernel feature (systemd only officially supports kernels ≤ 2 years old), so we'd need some fallback. Can you please step through with gdb and figure out which call fails in particular? I don't immediately see where this comes from -- readlinkat() is only called in some wrapper functions in src/basic/fs-util.c and copy_tree_at, neither of which are being called by networkd. So I sugguest to build a debug version with CFLAGS="-g -O0" (from upstream or the Debian package) and gdb through networkd.

[Simon Fels (morphis)]

Output of

  sudo SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd

ubuntu@nirvana:~$ sudo SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd
sudo: unable to resolve host nirvana: Connection refused
Failed to connect to bus, trying again in 5s: No such file or directory
timestamp of '/etc/systemd/network' changed
p2p0: Flags change: +MULTICAST +BROADCAST
p2p0: Link 7 added
p2p0: udev initialized link
p2p0: Saved original MTU: 1500
sit0: MAC address not found for new device, continuing without
sit0: Flags change: +NOARP
sit0: Link 6 added
sit0: udev initialized link
sit0: Saved original MTU: 1480
wlan0: Flags change: +MULTICAST +BROADCAST
wlan0: Link 5 added
wlan0: udev initialized link
wlan0: Saved original MTU: 1500
enx00606e43a573: Flags change: +MULTICAST +BROADCAST
enx00606e43a573: Link 4 added
enx00606e43a573: udev initialized link
enx00606e43a573: Saved original MTU: 1500
enp1s0: Flags change: +MULTICAST +BROADCAST
enp1s0: Link 3 added
enp1s0: udev initialized link
enp1s0: Saved original MTU: 1500
dummy0: Flags change: +BROADCAST +NOARP
dummy0: Link 2 added
dummy0: udev initialized link
dummy0: Saved original MTU: 1500
lo: Flags change: +LOOPBACK +UP +LOWER_UP +RUNNING
lo: Link 1 added
lo: udev initialized link
lo: Saved original MTU: 16436
lo: Adding address: ::1/128 (valid forever)
lo: Adding address: 127.0.0.1/8 (valid forever)
Enumeration completed
p2p0: Link state is up-to-date
p2p0: Unmanaged
sit0: Link state is up-to-date
sit0: Unmanaged
wlan0: Link state is up-to-date
wlan0: Unmanaged
enx00606e43a573: Link state is up-to-date
enx00606e43a573: Unmanaged
enp1s0: Link state is up-to-date
enp1s0: found matching network '/etc/systemd/network/enp1s0.network'
enp1s0: Bringing link up
enp1s0: Setting addresses
dummy0: Link state is up-to-date
dummy0: Unmanaged
lo: Link state is up-to-date
lo: Unmanaged
enp1s0: Could not bring up interface: Invalid argument
enp1s0: Updating address: 192.168.178.30/24 (valid forever)
enp1s0: Addresses set
enp1s0: Setting routes
enp1s0: Could not set route: Invalid argument
enp1s0: Routes set

[Simon Fels (morphis)]

The actual error message comes from https://github.com/systemd/systemd/blob/95adafc428b5b4be0ddd4d43a7b96658390388bc/src/network/networkd-link.c#L1441 and due to the nature of this the problem only occurs when the network interface isn't already up.

If lines 1391 to 1420 are commented on https://github.com/systemd/systemd/blob/95adafc428b5b4be0ddd4d43a7b96658390388bc/src/network/networkd-link.c#L1391 the kernel doesn't return EINVAL and the interface is properly setup.

[Martin Pitt (pitti)]

Does that kernel support IPv6, i. e. does socket_ipv6_is_supported() succeed and the whole AF_INET6 setup is actually done?

I. e. is the problem the mere existence of the IFLA_AF_SPEC container in the netlink message (which already isn't supported by the kernel) or only trying to set the IPv6 settings?

I. e. it might help to not create the container at all if !socket_ipv6_is_supported(), i. e. move the open/close_container() inside that if()? (that would only help if the kernel has IPv6 disabled)

[Simon Fels (morphis)]

Yes, the kernel has IPv6 support enabled so socket_ipv6_is_supported() returns true and with that it tries to setup the IPv6 part of the interface.

The two problems which I've found so far are:

 - an empty IFLA_AF_SPEC seems to cause the kernel to return EINVAL
 - having IPv6 enabled and IFLA_AF_SPEC is filled with the ipv6ll mode we get EINVAL

So to fix this we have to do two things:

 1. disable IPv6 support in the 3.4 kernel; if that is not acceptable for the real product later than we have to find another solution.
 2. move the open/close_container() inside the if() statement to avoid an empty container.

For 2. we may can work around this on the kernel side too to allow empty IFLA_AF_SPEC containers.

[Martin Pitt (pitti)]

> an empty IFLA_AF_SPEC seems to cause the kernel to return EINVAL

This only seems to happen on older kernels. I tried it on 4.4 and an empty container works fine. In particular, I made socket_ipv6_is_supported() always return false, and now networkd only configures IPv4 on my interface without any error.

[Martin Pitt (pitti)]

As I said I cannot test this patch on a current kernel really as it works just fine with IPv6 disabled. Please test this, and if it works for you I can send this upstream and SRU.

[Simon Fels (morphis)]

That patch works but there are now more problems with the DHCP stack inside networkd. See https://paste.ubuntu.com/23178198/ Needs more investigation. To unblock our work we're now using a static ifupdown configuration but need to fix this bug here for real networkd support.