igb driver crashes kernel when receiving jumbo frame UDP packets with all zeros

[Felix Ruess]

With a high MTU (9000) set on the interface, sending packets containing over ~3000 zero bytes to a network card using the igb driver results in broken packets or in the worst case for some packet sizes even leads to kernel panics or full freeze.

So far I observed this for Intel I350 and I210 network cards which use the igb driver.

This is easy to reproduce:
* set MTU to 9000
* send ping of death: ping -M do -p 00 -s 3016 <ip>
-> kernel panic or complete freeze of system

If the data is not zero, it works fine:
ping -M do -p ff -s 3016 <ip>

Depending on packet length:
* < 3000: fine
* 3016: panic/freeze
* > 3030: packet not complete

I also tested this with the following kernel versions:
* 5.4.0-131-generic -> OK
* 5.8.0-63-generic -> OK
* 5.11.0-46-generic -> OK
* 5.13.0-52-generic -> broken
* 5.15.0-41-generic -> broken
* 5.19.5-051905-generic mainline -> broken

We noticed that since we have GigE Vision cameras which send the data via UDP with packet sizes up to 9000. When we turned off the light (and hence many bytes in the payload were zero), the packets would not be complete anymore (checked with tcpdump/wireshark) or even lead to kernel panics.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-5.15.0-53-generic 5.15.0-53.59~20.04.1
ProcVersionSignature: Ubuntu 5.15.0-53.59~20.04.1-generic 5.15.64
Uname: Linux 5.15.0-53-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.25
Architecture: amd64
CasperMD5CheckResult: skip
Date: Thu Nov 24 13:16:00 2022
SourcePackage: linux-signed-hwe-5.15
UpgradeStatus: No upgrade log present (probably fresh install)

[Felix Ruess (flixr)]

here a dmesg output before the system freezes

[Stefan Battmer (stefanbat)]

Since we observed a similar issue I would like to add, that it actually seems like the actual content of the network packet towards the last bytes seems to cause this issue. As stated by Felix it seems to matter that the last 32 or so bytes of the packet contain zeros. One crashdump we observed actually showed one of these zeros on the instruction stack thus it looked to me as the processor tried to execute one of these zeros as a cpu instruction. This can also be seen on the dmesg output of Felix's kernel_bug.txt:

invalid opcode: 0000 [#1] SMP NOPTI

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Felix Ruess (flixr)]

So the kernel panic only seems to happen with some firmware versions for these network cards:
* I210 firmware=3.16
* I350 firmware=1.63

I could not reproduce this with a 4port I210 card with firmware=3.20.

Nevertheless this is a regression in the kernel.