tcpreplay and veth device; extra transmit bytes on 4.4.0-130-generic kernel

[Stuart Websper]

Issue found in a test harness for some code running on Ubuntu 16.04.4 LTS where we use a veth device to play in packets to the code under test. Essentially we see an extra 14 bytes being transmitted at the end of every packet that is just random garbage. Problem came in with the 4.4.0-130 kernel; if I revert to the 4.4.0-128 kernel the issue goes away. I have verified the issue on several different boxes running 16.04

The following steps reproduce the problem:

1) Veth setup

sudo ip link add name nic1 type veth peer name nic2
sudo sysctl -q -w net.ipv6.conf.nic1.disable_ipv6=1
sudo sysctl -q -w net.ipv6.conf.nic2.disable_ipv6=1
sudo ip link set nic1 up
sudo ip link set nic2 up

2) Configure tcpdump to collect packets on one interface:

sudo tcpdump -i nic1 -w outPkts.pcap

3) Play in packets on the other (Any pcap will do):

sudo tcpreplay -q -i nic2 inPkts.pcap

When I check ifconfig I can see nic2/nic1 transmitting/receiving more bytes than expected. Opening the capture seen by tcpdump shows an extra 14 bytes received for every packet when compared to the input.

tcpreplay version info:
  tcpreplay version: 3.4.4 (build 2450) (debug)
  Copyright 2000-2010 by Aaron Turner <aturner at synfin dot net>
  Cache file supported: 04
  Not compiled with libdnet.
  Compiled against libpcap: 1.7.4
  64 bit packet counters: enabled
  Verbose printing via tcpdump: enabled
  Packet editing: disabled
  Fragroute engine: disabled
  Injection method: PF_PACKET send()

tcpdump version info
  tcpdump version 4.9.2
  libpcap version 1.7.4
  OpenSSL 1.0.2g 1 Mar 2016

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-130-generic 4.4.0-130.156
ProcVersionSignature: Ubuntu 4.4.0-130.156-generic 4.4.134
Uname: Linux 4.4.0-130-generic x86_64
NonfreeKernelModules: nvidia_uvm nvidia
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: spw 3336 F.... pulseaudio
 /dev/snd/controlC0: spw 3336 F.... pulseaudio
CurrentDesktop: Unity
Date: Thu Jul 19 10:30:07 2018
HibernationDevice: RESUME=UUID=032369ab-18b8-41c3-a386-4ed08706f461
InstallationDate: Installed on 2016-11-03 (622 days ago)
InstallationMedia: Ubuntu 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
MachineType: Dell Inc. OptiPlex 7010
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-130-generic root=UUID=7b20f97c-4edc-4af1-bdbb-72ccb26cfbec ro quiet splash intel_iommu=on
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-130-generic N/A
 linux-backports-modules-4.4.0-130-generic N/A
 linux-firmware 1.157.20
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 08/12/2013
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A15
dmi.board.name: 0KRC95
dmi.board.vendor: Dell Inc.
dmi.board.version: A02
dmi.chassis.type: 6
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA15:bd08/12/2013:svnDellInc.:pnOptiPlex7010:pvr01:rvnDellInc.:rn0KRC95:rvrA02:cvnDellInc.:ct6:cvr:
dmi.product.name: OptiPlex 7010
dmi.product.version: 01
dmi.sys.vendor: Dell Inc.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream stable kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.4 stable kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.141

[Stuart Websper (spwcts)]

Tested on http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.142 and issue no longer present.

NOTE: I am not around next week, so if you have further things you want me to test and I don't respond, please don't take this that I have lost interest in getting this bug resolved. I will pick it up again on 30/7/2018.

[Stuart Websper (spwcts)]

Bug also present in the 4.4.0-131-generic kernel

[dajose22 (dajose22)]

I just upgraded my system and the bug is there.

kernel 4.15.0-33-generic
Ubuntu 18.04.1 LTS

[Andy Fingerhut (andy-fingerhut)]

I have Ubuntu 16.04.5 with kernel 4.15.0-33-generic and the bug is there.

I tried using ukuu to install several other "nearby" versions of the Linux kernel, and did not see the problem with any of them. A list of results at the end of this page: https://github.com/jafingerhut/p4-guide/tree/master/linux-veth-bug

[S B (sbourland)]

Can confirm bug returns moving from kernel 4.15.0-32 to 4.15.0-33, tested on both 16.04 Server and 18.04 Server. Tested with program using pcap_inject (libpcap 1.7.4 (16.04) and libpcap 1.8.1 (18.04)), no issues seen with 4.15.0-32 (or 4.15.0-29), additional 14 bytes being transmitted with 4.15.0-33 and 4.15.0-34.

[Daniel Baines (dnlbaines)]

Confirming the bug is present on Ubuntu 16.04 using kernel 4.15.0-33 when using pcap_inject. An extra 14 bytes are sent on the end of packets causing them to appear invalid in Wireshark.

[Daniel Baines (dnlbaines)]

Upgraded to the latest mainline kernel (4.18.7-041807-generic) and the issue is not present.

[Andy Fingerhut (andy-fingerhut)]

I saw the issue on both Ubuntu 16.04 and 18.04 with kernel versions 4.15.0-33-generic and 4.15.0-34-generic. The problem is not there for the latest released kernel version 4.15.0-36-generic.

I plan to continue testing for several more kernel releases for Ubuntu 16.04 and 18.04 for a while, to see if the issue reappears: https://github.com/jafingerhut/p4-guide/tree/master/linux-veth-bug