systemd-networkd brings eno1 up and down repeatedly

Bug #1881207 reported by Attie Grande on 2020-05-28
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd
New
Unknown
systemd (Ubuntu)
Medium
Dan Streetman
Bionic
Medium
Dan Streetman
Eoan
Medium
Dan Streetman
Focal
Medium
Dan Streetman
Groovy
Medium
Dan Streetman
Hirsute
Medium
Dan Streetman

Bug Description

[impact]

a system using dhcp for an interface where (1) the interface 'resets' itself during mtu setting (meaning it briefly drops carrier during mtu change), and (2) the dhcp server provides a non-standard (i.e. other than 1500) mtu, will continually loop changing mtu between 1500 and the dhcp-privded mtu.

[test case]

The e1000e nic is one such nic where the driver briefly drops carrier when changing the mtu, so this can be reproduced by creating a VM and setting the interface to the 'e1000e' emulated hardware. Then configure a dhcp server to provide dhcp service to the VM, and set the mtu to e.g. 9000 (anything other than the default 1500).

When the VM runs dhcp, it will loop with:

Feb 22 15:18:58 lp1881207-upstream systemd-networkd[992]: ens8: Gained carrier
Feb 22 15:18:58 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv4 address 1.2.3.133/24 via 1.2.3.1
Feb 22 15:18:59 lp1881207-upstream systemd-networkd[992]: ens8: Lost carrier
Feb 22 15:18:59 lp1881207-upstream systemd-networkd[992]: ens8: DHCP lease lost
Feb 22 15:18:59 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv6 lease lost
Feb 22 15:19:01 lp1881207-upstream systemd-networkd[992]: ens8: Gained carrier
Feb 22 15:19:01 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv4 address 1.2.3.133/24 via 1.2.3.1
Feb 22 15:19:02 lp1881207-upstream systemd-networkd[992]: ens8: Lost carrier
Feb 22 15:19:02 lp1881207-upstream systemd-networkd[992]: ens8: DHCP lease lost
Feb 22 15:19:02 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv6 lease lost
Feb 22 15:19:04 lp1881207-upstream systemd-networkd[992]: ens8: Gained carrier
Feb 22 15:19:04 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv4 address 1.2.3.133/24 via 1.2.3.1
Feb 22 15:19:05 lp1881207-upstream systemd-networkd[992]: ens8: Lost carrier
Feb 22 15:19:05 lp1881207-upstream systemd-networkd[992]: ens8: DHCP lease lost
Feb 22 15:19:05 lp1881207-upstream systemd-networkd[992]: ens8: DHCPv6 lease lost

[regression potential]

any regression would likely involve an issue with dhcpv4 service; possibly not setting the mtu correctly, or possibly not completing dhcpv4 at all.

[scope]

this is not fixed upstream yet, so (once fixed upstream) it will be needed for b/f/g/h

[original description]

Running on a NUC (BOXNUC8i7BEH3).

After updating the `systemd` package past `237-3ubuntu10.32`, I see the onboard NIC link being taken down / up repeatedly, as shown below (dates are from my original notes, but the issue remains the same today).

The NIC fails to remain up, and all networking is out of action.

Running `systemctl stop systemd-netword` stops this and leaves the NIC in an unknown state.
Subsequently running `ifconfig eno1 up && dhclient eno1` allows networking to continue basic operation, albeit without systemd's oversight.

My original solution was to downgrade both the `libsystemd0` and `systemd` packages to `237-3ubuntu10.28`.

After attempting an `apt update && apt upgrade` again today, exactly the same thing is happening with `237-3ubuntu10.41`.

Good versions:
- 237-3ubuntu10.28
- 237-3ubuntu10.32 (currently in use, and held)

Bad versions:
- 237-3ubuntu10.33
- 237-3ubuntu10.38
- 237-3ubuntu10.41

dmesg:

[ 360.711367] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 360.714370] e1000e 0000:00:1f.6 eno1: changing MTU from 1500 to 9000
[ 360.726189] e1000e 0000:00:1f.6: Interrupt Throttle Rate off
[ 361.733073] e1000e 0000:00:1f.6 eno1: changing MTU from 9000 to 1500
[ 361.744146] e1000e 0000:00:1f.6: Interrupt Throttle Rate on
[ 366.760198] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 366.763375] e1000e 0000:00:1f.6 eno1: changing MTU from 1500 to 9000
[ 366.776060] e1000e 0000:00:1f.6: Interrupt Throttle Rate off
[ 367.781718] e1000e 0000:00:1f.6 eno1: changing MTU from 9000 to 1500
[ 367.792796] e1000e 0000:00:1f.6: Interrupt Throttle Rate on
[ 372.808660] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[ 372.811537] e1000e 0000:00:1f.6 eno1: changing MTU from 1500 to 9000
[ 372.823648] e1000e 0000:00:1f.6: Interrupt Throttle Rate off
[ 373.830436] e1000e 0000:00:1f.6 eno1: changing MTU from 9000 to 1500
[ 373.841512] e1000e 0000:00:1f.6: Interrupt Throttle Rate on

journalctl -u systemd-networkd:

Jan 06 18:04:05 patch systemd-networkd[755]: eno1: Gained carrier
Jan 06 18:04:05 patch systemd-networkd[755]: eno1: DHCPv4 address 10.42.0.5/24 via 10.42.0.3
Jan 06 18:04:05 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
Jan 06 18:04:05 patch systemd-networkd[755]: eno1: Configured
Jan 06 18:04:06 patch systemd-networkd[755]: eno1: Lost carrier
Jan 06 18:04:06 patch systemd-networkd[755]: eno1: DHCP lease lost
Jan 06 18:04:06 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
Jan 06 18:04:11 patch systemd-networkd[755]: eno1: Gained carrier
Jan 06 18:04:11 patch systemd-networkd[755]: eno1: DHCPv4 address 10.42.0.5/24 via 10.42.0.3
Jan 06 18:04:11 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
Jan 06 18:04:11 patch systemd-networkd[755]: eno1: Configured
Jan 06 18:04:12 patch systemd-networkd[755]: eno1: Lost carrier
Jan 06 18:04:12 patch systemd-networkd[755]: eno1: DHCP lease lost
Jan 06 18:04:12 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
Jan 06 18:04:17 patch systemd-networkd[755]: eno1: Gained carrier
Jan 06 18:04:17 patch systemd-networkd[755]: eno1: DHCPv4 address 10.42.0.5/24 via 10.42.0.3
Jan 06 18:04:17 patch systemd-networkd[755]: eno1: IPv6 successfully enabled
Jan 06 18:04:17 patch systemd-networkd[755]: eno1: Configured
Jan 06 18:04:18 patch systemd-networkd[755]: eno1: Lost carrier
Jan 06 18:04:18 patch systemd-networkd[755]: eno1: DHCP lease lost
Jan 06 18:04:18 patch systemd-networkd[755]: eno1: IPv6 successfully enabled

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 237-3ubuntu10.32
ProcVersionSignature: Ubuntu 4.15.0-101.102-generic 4.15.18
Uname: Linux 4.15.0-101-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.15
Architecture: amd64
Date: Thu May 28 23:19:27 2020
InstallationDate: Installed on 2019-06-20 (343 days ago)
InstallationMedia: Ubuntu-Server 18.04.2 LTS "Bionic Beaver" - Release amd64 (20190210)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 003: ID 0dc6:3401 Precision Squared Technology Corp.
 Bus 001 Device 002: ID 0403:6014 Future Technology Devices International, Ltd FT232H Single HS USB-UART/FIFO IC
 Bus 001 Device 004: ID 8087:0aaa Intel Corp.
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Intel(R) Client Systems NUC8i7BEH
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-101-generic root=UUID=d03ca5b0-7a00-42da-a3a8-e065dcf6dd07 ro
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 02/13/2019
dmi.bios.vendor: Intel Corp.
dmi.bios.version: BECFL357.86A.0064.2019.0213.1122
dmi.board.name: NUC8BEB
dmi.board.vendor: Intel Corporation
dmi.board.version: J72688-305
dmi.chassis.type: 3
dmi.chassis.vendor: Intel Corporation
dmi.chassis.version: 2.0
dmi.modalias: dmi:bvnIntelCorp.:bvrBECFL357.86A.0064.2019.0213.1122:bd02/13/2019:svnIntel(R)ClientSystems:pnNUC8i7BEH:pvrJ72992-305:rvnIntelCorporation:rnNUC8BEB:rvrJ72688-305:cvnIntelCorporation:ct3:cvr2.0:
dmi.product.family: Intel NUC
dmi.product.name: NUC8i7BEH
dmi.product.version: J72992-305
dmi.sys.vendor: Intel(R) Client Systems

Revision history for this message
Attie Grande (attiegrande) wrote :
description: updated
Revision history for this message
Dan Streetman (ddstreet) wrote :

Can you attach your network configuration - either netplan config or networkd config?

Also, can you try removing your MTU config for eno1 and see if that allows it to stay up?

Revision history for this message
Attie Grande (attiegrande) wrote :

Hi Dan,

- `/etc/systemd/network/` is empty.
- `/etc/netplan/01-netcfg.yaml` is attached. (I believe this is default, I don't think I've made any changes)
- MTU 9000 was being set by DHCP, I've removed this option and with the default (MTU=1500), the system comes up correctly.

Attie

Revision history for this message
Attie Grande (attiegrande) wrote :

To clarify:

- I'm now running version `237-3ubuntu10.41` of both `systemd` and `libsystemd0`.
- With the DHCP MTU option removed, the network interface comes up correctly and remains stable.

Revision history for this message
Dan Streetman (ddstreet) wrote :

I can reproduce this with a VM using the emulated 'e1000e' device, and unfortunately this is still a bug with upstream systemd code. I'll look at fixing this upstream but for now as a workaround, I'd recommend manually setting the mtu and ignoring the mtu from dhcp, for example by modifying the netplan yaml as such:

network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: yes
      dhcp4-overrides:
        use-mtu: false
      mtu: 9000

Changed in systemd (Ubuntu Groovy):
importance: Undecided → Medium
Changed in systemd (Ubuntu Focal):
importance: Undecided → Medium
Changed in systemd (Ubuntu Eoan):
importance: Undecided → Medium
Changed in systemd (Ubuntu Bionic):
importance: Undecided → Medium
Changed in systemd (Ubuntu Groovy):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Focal):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Eoan):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Groovy):
status: New → In Progress
Changed in systemd (Ubuntu Focal):
status: New → In Progress
Changed in systemd (Ubuntu Eoan):
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
status: New → In Progress
tags: added: seg
Revision history for this message
Attie Grande (attiegrande) wrote :

Thanks for the workaround.

I've reinstated the DHCP MTU option, and updated netplan's config.
I can confirm that this behaves for me too.

Revision history for this message
Dan Streetman (ddstreet) wrote :

marking Eoan as wontfix; it's EOL in weeks

Changed in systemd (Ubuntu Eoan):
status: In Progress → Won't Fix
Dan Streetman (ddstreet) on 2021-02-22
description: updated
Changed in systemd:
status: Unknown → New
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.