systemd-networkd brings eno1 up and down repeatedly

[Attie Grande]

[impact]

a system using dhcp for an interface where (1) the interface 'resets' itself during mtu setting (meaning it briefly drops carrier during mtu change), and (2) the dhcp server provides a non-standard (i.e. other than 1500) mtu, will continually loop changing mtu between 1500 and the dhcp-privded mtu.

[test case]

The e1000e nic is one such nic where the driver briefly drops carrier when changing the mtu, so this can be reproduced by creating a VM and setting the interface to the 'e1000e' emulated hardware. Then configure a dhcp server to provide dhcp service to the VM, and set the mtu to e.g. 9000 (anything other than the default 1500).

When the VM runs dhcp, it will loop with:

Feb 22 15:18:58 lp1881207-upstream systemd-networkd[992]: ens8: Gained carrier
Feb 22 15:18:58 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv4 address 1.2.3.133/24 via 1.2.3.1
Feb 22 15:18:59 lp1881207-upstream systemd-networkd[992]: ens8: Lost carrier
Feb 22 15:18:59 lp1881207-upstream systemd-networkd[992]: ens8: DHCP lease lost
Feb 22 15:18:59 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv6 lease lost
Feb 22 15:19:01 lp1881207-upstream systemd-networkd[992]: ens8: Gained carrier
Feb 22 15:19:01 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv4 address 1.2.3.133/24 via 1.2.3.1
Feb 22 15:19:02 lp1881207-upstream systemd-networkd[992]: ens8: Lost carrier
Feb 22 15:19:02 lp1881207-upstream systemd-networkd[992]: ens8: DHCP lease lost
Feb 22 15:19:02 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv6 lease lost
Feb 22 15:19:04 lp1881207-upstream systemd-networkd[992]: ens8: Gained carrier
Feb 22 15:19:04 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv4 address 1.2.3.133/24 via 1.2.3.1
Feb 22 15:19:05 lp1881207-upstream systemd-networkd[992]: ens8: Lost carrier
Feb 22 15:19:05 lp1881207-upstream systemd-networkd[992]: ens8: DHCP lease lost
Feb 22 15:19:05 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv6 lease lost

[regression potential]

any regression would likely involve an issue with dhcpv4 service; possibly not setting the mtu correctly, or possibly not completing dhcpv4 at all.

[scope]

this is not fixed upstream yet, so (once fixed upstream) it will be needed for b/f/g/h

[original description]

Running on a NUC (BOXNUC8i7BEH3).

After updating the `systemd` package past `237-3ubuntu10.32`, I see the onboard NIC link being taken down / up repeatedly, as shown below (dates are from my original notes, but the issue remains the same today).

The NIC fails to remain up, and all networking is out of action.

Running `systemctl stop systemd-netword` stops this and leaves the NIC in an unknown state.
Subsequently running `ifconfig eno1 up && dhclient eno1` allows networking to continue basic operation, albeit without systemd's oversight.

My original solution was to downgrade both the `libsystemd0` and `systemd` packages to `237-3ubuntu10.28`.

After attempting an `apt update && apt upgrade` again today, exactly the same thing is happening with `237-3ubuntu10.41`.

Good versions:
- 237-3ubuntu10.28
- 237-3ubuntu10.32 (currently in use, and held)

Bad versions:
- 237-3ubuntu10.33
- 237-3ubuntu10.38
- 237-3ubuntu10.41

dmesg:

[ 360.711367] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 360.714370] e1000e 0000:00:1f.6 eno1: changing MTU from 1500 to 9000
[ 360.726189] e1000e 0000:00:1f.6: Interrupt Throttle Rate off
[ 361.733073] e1000e 0000:00:1f.6 eno1: changing MTU from 9000 to 1500
[ 361.744146] e1000e 0000:00:1f.6: Interrupt Throttle Rate on
[ 366.760198] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 366.763375] e1000e 0000:00:1f.6 eno1: changing MTU from 1500 to 9000
[ 366.776060] e1000e 0000:00:1f.6: Interrupt Throttle Rate off
[ 367.781718] e1000e 0000:00:1f.6 eno1: changing MTU from 9000 to 1500
[ 367.792796] e1000e 0000:00:1f.6: Interrupt Throttle Rate on
[ 372.808660] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 372.811537] e1000e 0000:00:1f.6 eno1: changing MTU from 1500 to 9000
[ 372.823648] e1000e 0000:00:1f.6: Interrupt Throttle Rate off
[ 373.830436] e1000e 0000:00:1f.6 eno1: changing MTU from 9000 to 1500
[ 373.841512] e1000e 0000:00:1f.6: Interrupt Throttle Rate on

journalctl -u systemd-networkd:

Jan 06 18:04:05 patch systemd-networkd[755]: eno1: Gained carrier
Jan 06 18:04:05 patch systemd-networkd[755]: eno1: DHCPv4 address 10.42.0.5/24 via 10.42.0.3
Jan 06 18:04:05 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
Jan 06 18:04:05 patch systemd-networkd[755]: eno1: Configured
Jan 06 18:04:06 patch systemd-networkd[755]: eno1: Lost carrier
Jan 06 18:04:06 patch systemd-networkd[755]: eno1: DHCP lease lost
Jan 06 18:04:06 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
Jan 06 18:04:11 patch systemd-networkd[755]: eno1: Gained carrier
Jan 06 18:04:11 patch systemd-networkd[755]: eno1: DHCPv4 address 10.42.0.5/24 via 10.42.0.3
Jan 06 18:04:11 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
Jan 06 18:04:11 patch systemd-networkd[755]: eno1: Configured
Jan 06 18:04:12 patch systemd-networkd[755]: eno1: Lost carrier
Jan 06 18:04:12 patch systemd-networkd[755]: eno1: DHCP lease lost
Jan 06 18:04:12 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
Jan 06 18:04:17 patch systemd-networkd[755]: eno1: Gained carrier
Jan 06 18:04:17 patch systemd-networkd[755]: eno1: DHCPv4 address 10.42.0.5/24 via 10.42.0.3
Jan 06 18:04:17 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
Jan 06 18:04:17 patch systemd-networkd[755]: eno1: Configured
Jan 06 18:04:18 patch systemd-networkd[755]: eno1: Lost carrier
Jan 06 18:04:18 patch systemd-networkd[755]: eno1: DHCP lease lost
Jan 06 18:04:18 patch systemd-networkd[755]: eno1: IPv6 successfully enabled

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 237-3ubuntu10.32
ProcVersionSignature: Ubuntu 4.15.0-101.102-generic 4.15.18
Uname: Linux 4.15.0-101-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.15
Architecture: amd64
Date: Thu May 28 23:19:27 2020
InstallationDate: Installed on 2019-06-20 (343 days ago)
InstallationMedia: Ubuntu-Server 18.04.2 LTS "Bionic Beaver" - Release amd64 (20190210)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 003: ID 0dc6:3401 Precision Squared Technology Corp.
 Bus 001 Device 002: ID 0403:6014 Future Technology Devices International, Ltd FT232H Single HS USB-UART/FIFO IC
 Bus 001 Device 004: ID 8087:0aaa Intel Corp.
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Intel(R) Client Systems NUC8i7BEH
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-101-generic root=UUID=d03ca5b0-7a00-42da-a3a8-e065dcf6dd07 ro
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 02/13/2019
dmi.bios.vendor: Intel Corp.
dmi.bios.version: BECFL357.86A.0064.2019.0213.1122
dmi.board.name: NUC8BEB
dmi.board.vendor: Intel Corporation
dmi.board.version: J72688-305
dmi.chassis.type: 3
dmi.chassis.vendor: Intel Corporation
dmi.chassis.version: 2.0
dmi.modalias: dmi:bvnIntelCorp.:bvrBECFL357.86A.0064.2019.0213.1122:bd02/13/2019:svnIntel(R)ClientSystems:pnNUC8i7BEH:pvrJ72992-305:rvnIntelCorporation:rnNUC8BEB:rvrJ72688-305:cvnIntelCorporation:ct3:cvr2.0:
dmi.product.family: Intel NUC
dmi.product.name: NUC8i7BEH
dmi.product.version: J72992-305
dmi.sys.vendor: Intel(R) Client Systems

[Dan Streetman (ddstreet)]

Can you attach your network configuration - either netplan config or networkd config?

Also, can you try removing your MTU config for eno1 and see if that allows it to stay up?

[Attie Grande (attiegrande)]

Hi Dan,

- `/etc/systemd/network/` is empty.
- `/etc/netplan/01-netcfg.yaml` is attached. (I believe this is default, I don't think I've made any changes)
- MTU 9000 was being set by DHCP, I've removed this option and with the default (MTU=1500), the system comes up correctly.

Attie

[Attie Grande (attiegrande)]

To clarify:

- I'm now running version `237-3ubuntu10.41` of both `systemd` and `libsystemd0`.
- With the DHCP MTU option removed, the network interface comes up correctly and remains stable.

[Dan Streetman (ddstreet)]

I can reproduce this with a VM using the emulated 'e1000e' device, and unfortunately this is still a bug with upstream systemd code. I'll look at fixing this upstream but for now as a workaround, I'd recommend manually setting the mtu and ignoring the mtu from dhcp, for example by modifying the netplan yaml as such:

network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: yes
      dhcp4-overrides:
        use-mtu: false
      mtu: 9000

[Attie Grande (attiegrande)]

Thanks for the workaround.

I've reinstated the DHCP MTU option, and updated netplan's config.
I can confirm that this behaves for me too.

[Dan Streetman (ddstreet)]

marking Eoan as wontfix; it's EOL in weeks

[Dan Streetman (ddstreet)]

upstream fix still in progress

[Brian Murray (brian-murray)]

The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release

[Brian Murray (brian-murray)]

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

[Brian Murray (brian-murray)]

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

[Nick Rosbrook (enr0n)]

This has not actually been in progress for a while. IIUC, based on https://github.com/systemd/systemd/issues/18738, this has been resolved upstream since v250. At this time I do not think we will do an SRU for this to Focal or Bionic.