Netplan and Intel e1000 Driver / I219-V Adapter

[Jimothy]

Since the following change earlier in the year, I have been seeing issues with Intel's I219-V adapter. It's hard to say whether the problem is specifically Netplan or down to another change with the e1000 driver a month or so before. The main reason I am posting this here is because when machines were switched to Network Manager, the problem seemed to go away. (Replicated this work-around around with ~10 machines).

https://changelogs.ubuntu.com/changelogs/pool/main/n/netplan.io/netplan.io_0.106.1-7ubuntu0.22.04.4/changelog

  * SECURITY REGRESSION: failure on systems without dbus
    - debian/netplan.io.postinst: Don't call the generator if no networkd
      configuration file exists. (LP: #2071333)

I have had quite a few machines on our network loosing networking after a period of time. The organisation's network is software defined (Cisco) and uses 802.1x to authenticate machines to various sub-nets.

Machines get an IP address at boot but loose connection after 3-6 hours. The syslog reports a constant stream of the following message. The Cisco logs seem to report "unable to obtain an IP address from DHCP". The machines seems to believe it still has the same IP address but is unable to communicate.

[60689.477031] e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang:
                 TDH <13>
                 TDT <15>
                 next_to_use <15>
                 next_to_clean <11>
               buffer_info[next_to_clean]:
                 time_stamp <100e65bc0>
                 next_to_watch <14>
                 jiffies <100e65d58>
                 next_to_watch.status <0>
               MAC Status <40080083>
               PHY Status <796d>
               PHY 1000BASE-T Status <3800>
               PHY Extended Status <3000>
               PCI Status <10>

There is a similiar bug reported on the following post although this e1000e driver seems to have quite phases where it fails for people. It seemed that it's a long lived NIC installed over many years, there are quite a few firmware versions to support over it's lifetime.

https://bugzilla.kernel.org/show_bug.cgi?id=118721

There was also a e1000e driver update via the Kernel package, this was around the same time range the Netplan changed.

https://changelogs.ubuntu.com/changelogs/pool/main/l/linux-hwe-6.8/linux-hwe-6.8_6.8.0-60.63~22.04.1/changelog

* Noble update: upstream stable patchset 2025-02-03 (LP: #2097301)
 - e1000e: change I219 (19) devices to ADP
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu82.8
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: locadmin 7922 F.... pulseaudio
 /dev/snd/controlC0: locadmin 7922 F.... pulseaudio
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2025-06-09 (36 days ago)
InstallationMedia: Ubuntu 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809.1)
IwConfig:
 lo no wireless extensions.

 eno1 no wireless extensions.
MachineType: ASUS BOAMOT-524
Package: netplan.io 0.106.1-7ubuntu0.22.04.4
PackageArchitecture: amd64
ProcFB: 0 nouveaudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-6.8.0-60-generic root=UUID=9d15962f-075b-4d5c-9b40-5cd9b8f11c8a ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 6.8.0-60.63~22.04.1-generic 6.8.12
RelatedPackageVersions:
 linux-restricted-modules-6.8.0-60-generic N/A
 linux-backports-modules-6.8.0-60-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3.37
RfKill:

Tags: wayland-session jammy wayland-session
Uname: Linux 6.8.0-60-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 05/14/2025
dmi.bios.release: 38.1
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 3801
dmi.board.asset.tag: Default string
dmi.board.name: PRIME H610M-A D4
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3801:bd05/14/2025:br38.1:svnASUS:pnBOAMOT-524:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnPRIMEH610M-AD4:rvrRev1.xx:cvnDefaultstring:ct3:cvrDefaultstring:skuH610i5M16S1DG:
dmi.product.family: StonePC Lite
dmi.product.name: BOAMOT-524
dmi.product.sku: H610i5M16S1DG
dmi.product.version: System Version
dmi.sys.vendor: ASUS
modified.conffile..etc.default.apport: enabled=0
mtime.conffile..etc.default.apport: 2025-06-09T13:41:20.855630
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu82.8
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: locadmin 1648 F.... pulseaudio
 /dev/snd/controlC0: locadmin 1648 F.... pulseaudio
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2025-06-09 (36 days ago)
InstallationMedia: Ubuntu 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809.1)
IwConfig:
 lo no wireless extensions.

 eno1 no wireless extensions.
MachineType: ASUS BOAMOT-524
Package: netplan.io 0.106.1-7ubuntu0.22.04.4
PackageArchitecture: amd64
ProcFB: 0 nouveaudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-6.8.0-52-generic root=UUID=9d15962f-075b-4d5c-9b40-5cd9b8f11c8a ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 6.8.0-52.53~22.04.1-generic 6.8.12
RelatedPackageVersions:
 linux-restricted-modules-6.8.0-52-generic N/A
 linux-backports-modules-6.8.0-52-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3.37
RfKill:

Tags: wayland-session jammy wayland-session
Uname: Linux 6.8.0-52-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 05/14/2025
dmi.bios.release: 38.1
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 3801
dmi.board.asset.tag: Default string
dmi.board.name: PRIME H610M-A D4
dmi.board.vendor: ASUSTeK COMPUTER INC.
dmi.board.version: Rev 1.xx
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3801:bd05/14/2025:br38.1:svnASUS:pnBOAMOT-524:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnPRIMEH610M-AD4:rvrRev1.xx:cvnDefaultstring:ct3:cvrDefaultstring:skuH610i5M16S1DG:
dmi.product.family: StonePC Lite
dmi.product.name: BOAMOT-524
dmi.product.sku: H610i5M16S1DG
dmi.product.version: System Version
dmi.sys.vendor: ASUS
modified.conffile..etc.default.apport: enabled=0
mtime.conffile..etc.default.apport: 2025-06-09T13:41:20.855630

[Lukas Märdian (slyon)]

Hi, thanks for your bug report!

From the version strings I conclude that this is happening on a Ubuntu 22.04 LTS ("Jammy") system, can you confirm this?

The netplan.io version 0.106.1-7ubuntu0.22.04.4 that you mentioned above was released in June 2024, about a year ago. If you say the issue happened earlier this year (2025), this is probably unrelated.

But:
There is a major new Netplan version in jammy-proposed (v0.107), have you already tried if using that is making any difference? (https://launchpad.net/ubuntu/+source/netplan.io/0.107.1-3ubuntu0.22.04.2)

Also, can you please show the contents of Netplan runtime directories after the systems lose their connection and provide journalctl logs, potentially with systemd-networkd debugging enabled [1], to see what's going on.

$ tree /run/systemd/network
$ tree /run/NetworkManager/system-connections/
$ journalctl -g systemd

[1] https://gist.github.com/rkalkani/817edb3d335d7bc17857be81c746e9c9

[Jimothy (jambonum5)]

Sorry Lukas, my mistake there I read Jan rather Jun on the changelog. You're probably right that it isn't to do with Netplan, especially given the kernel update that included the e1000e driver update.

This issue was mainly on Ubuntu 22.04, also had an instance on 24.04 but I hadn't looked too far into that version. Again, moving to network manager seemed to work around the issue on 24.04 too.

Thanks for the suggestions, I'll take a look a the proposed version. I'll also enable debugging for more info.

I'll get back to you when I have move info.

[Juerg Haefliger (juergh)]

@Jimothy Can you attach a journal log showing the error, including kernel messages? Also the output of `sudo lspci -vvv` so we know what HW you have.

[Jimothy (jambonum5)]

I have had to revert the config on my test machines, just in the process of recreaing the error with the debugging enabled.

These two test machines are running with the following Intel NIC, model and firmware below:

00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (17) I219-V (rev 11)
 DeviceName: Onboard - Ethernet
 Subsystem: ASUSTeK Computer Inc. Ethernet Connection (17) I219-V
 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
 Latency: 0
 Interrupt: pin D routed to IRQ 133
 IOMMU group: 9
 Region 0: Memory at 82300000 (32-bit, non-prefetchable) [size=128K]
 Capabilities: [c8] Power Management version 3
  Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
  Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
 Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
  Address: 00000000fee002d8 Data: 0000
 Kernel driver in use: e1000e
 Kernel modules: e1000e

lshw -c network
WARNING: you should run this program as super-user.
  *-network
       description: Ethernet interface
       product: Ethernet Connection (17) I219-V
       vendor: Intel Corporation
       physical id: 1f.6
       bus info: pci@0000:00:1f.6
       logical name: eno1
       version: 11
       serial: a0:36:bc:e6:05:61
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=6.8.0-60-generic duplex=full firmware=2.3-4 ip=XXX.XXX.XXX.XXX latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:133 memory:82300000-8231ffff

More information to follow...

[Jimothy (jambonum5)]

Thanks for the prompt response, I have attached the systemd / networkd logs that were requested. I have managed to repeating this issue on one machine, I might be able to get further logs from another machine that is on test.

This was the output of the tree command, there didn't seem to be anything in the Network Manager directory that was mentioned.

/run/systemd/network/
└── 10-netplan-eno1.network

0 directories, 1 file

[Andreas Hasenack (ahasenack)]

Logs were attached, so I'm moving the bug status away from incomplete.

[Massimiliano Pellizzer (mpellizzer)]

Thank you for taking the time to report this bug and contributing to Ubuntu.
To allow me to debug the issue from the kernel side, can you please run this command on a machine which is affected by the bug?

$ apport-collect 2115044

Is the bug not reproducible in some Ubuntu kernels?

[Jimothy (jambonum5)]

No problem Massimiliano, many thanks for your assistance.

I had a lab of 25 identical machines running 22.04, these machines seemed to be working with 6.8.0-52 or lower. It seemed the following kernel update contained the e1000e driver update in the link above.

Over these 25 machines, I tried advice from various forums:

* Disabling TSO options via ethtools hasn't helped
* Disabling WOL via the BIOS and ethtools hasn't helped
* Disabling Active State Power Management (via grub pcie_aspm=off) hasn't helped

Unfortuniately, it only seemed to be resovled by moving to Network Manager.

The network side is complaining about the DHCP negociation, there seems to be something Network Manager is able to do that Netplan is struggling with.

We might be a bit of a corner case here, given we are using a software defined network and 802.1x. We have seem weirdness in other senses, VMs not able to bridge virtual adapters. Machines not being accessible until it calls out to the network.

What add's further confusion is that we have had a few machines that appear to have the same OS/kernel/NIC firmware which appear to be ok. There was another system that we upgraded to 24.04 with the latest kernel which still seemed to have the issue.

I'll look at raising the apport-collect, is it best to run this after the error has happened?

[Massimiliano Pellizzer (mpellizzer)]

Yes please run apport-collect once you have "executed the reproducer" which gives you the error.
Please use apport-collect both with a working kernel (6.8.0-52 or lower) and with a kernel where you are able to reproduce the bug.
Thanks

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

apport information

[Jimothy (jambonum5)]

Sorry for the delay, I have just uploaded two apport-collects, one for 6.8.0-60 and one for working kernel 6.8.0-52.

For the 6.8.0-60 upload, the machine had suffered the issue and lost networking. However I had to reconnect the network cable to recover the connection so I could send the report. I'm not sure if this will make it confusing to go through the apport files that were uploaded.

Hope this helps, let me know if anything needs clarifying.

[Massimiliano Pellizzer (mpellizzer)]

Unfortunately the kernel buffer of (dmesg) of 6.8.0-60 has not been uploaded completely.
Can you please attach the output of:
$ sudo dmesg > dmesg60.txt
after reproducing the bug?

[Jimothy (jambonum5)]

I'm not sure how useful this log will be... because of the flood of messages, it seems to have cropped off the point the issue likely happened. There isn't much else in the /var/log/dmesg logs to draw on, sorry.

[Massimiliano Pellizzer (mpellizzer)]

I see ... thanks anyway for attaching logs.

Can you please try the new linux-hwe-6.8 [1] from jammy-proposed [2] which includes fixes for e1000? Is the bug still reproducible?

[1] https://launchpad.net/ubuntu/+source/linux-hwe-6.8/6.8.0-72.72~22.04.1
[2] https://wiki.ubuntu.com/Testing/EnableProposed

[Lukas Märdian (slyon)]

Thanks for picking up the kernel triage process. I'm marking this "Invalid" for Netplan, according to comment #2

[Jimothy (jambonum5)]

Hi Lukas, I was trying to get some more logs bundled as the updated kernel hasn't fixed the issue. The curious thing was that Network Manager doesn't have this issue.

Is it still worth collecting addditional logs for this issue?