IPsec VTI functionality broken in 3.16.0-39

[Clemens Schrimpe]

Gentlepeople - this is my very first bug-report to/about Ubuntu, so please forgive any failings regarding "form" on my side!

After upgrading from 3.16.0-38-generic to 3.16.0-39-generic I noticed a number of my IPsec VTIs were no longer working:
All crypto parts appear to work fine (I can run tcpdump on the VTIs and I correct cleartext-packets in both directions), but incoming packets are not being "processed further" (they are simply ignored). It is like there is no IP stack listening on the inbound end of the VTI. I can ping devices on the other side and do see the packets w/ tcpdump/wireshark all over the place (locally, remote-router, remote-device), the targets respond and I again see the packets all the way, but the ping application pretends it never heard or saw a thing.

This is true for all VTIs, except those where I put complicated mangle and nat rules in place in order to overcome address-space collisions (damn RFC1918, damn, damn, damn!!!) - but then again source-NAT (masquerading) no longer works on these VTIs either.

I tested around by leaving *everything* (StrongSwan config, etc.) the same and only switching kernels and 3.16.0-38 ist the last one fully working and everything after and including 3.16.0-39 is broken in the way described above.

I am willing to test further and dig deeper unless you tell me that it is a known problem with an upcoming fix ... :-)

Thanks, Clemens

ProblemType: Bug
DistroRelease: Ubuntu 14.10
Package: linux-image-3.16.0-39-generic (not installed)
ProcVersionSignature: Ubuntu 3.16.0-38.52-generic 3.16.7-ckt10
Uname: Linux 3.16.0-38-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access /dev/snd/: No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.7-0ubuntu8.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
CRDA: Error: [Errno 2] No such file or directory: 'iw'
Date: Mon Jun 22 16:48:33 2015
HibernationDevice: RESUME=UUID=e0eb93cf-68f6-4c6b-b4f1-288db4b33df2
InstallationDate: Installed on 2015-02-15 (126 days ago)
InstallationMedia: Ubuntu-Server 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.3)
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
PciMultimedia:

ProcEnviron:
 LANGUAGE=en_US:en
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/usr/bin/tcsh
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.16.0-38-generic root=UUID=bb995ded-003a-4ae3-aa21-0cf188bdba17 ro
RelatedPackageVersions:
 linux-restricted-modules-3.16.0-38-generic N/A
 linux-backports-modules-3.16.0-38-generic N/A
 linux-firmware 1.138.1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to utopic on 2015-02-15 (126 days ago)
dmi.bios.date: 01/01/2011
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2011:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-trusty:cvnBochs:ct1:cvr:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-trusty
dmi.sys.vendor: QEMU

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.1 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.1-unstable/

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Simon Déziel (sdeziel)]

Someone on the Strongswan mailing list [1] mentioned that 3.19 was also affected.

I quickly skimmed the changelog between 3.16.0-38-generic to 3.16.0-39-generic and a possible culprit could be:

  * ip_forward: Drop frames with attached skb->sk

Clemens, would you be able to just revert the corresponding commit and see it if helps?

1: https://lists.strongswan.org/pipermail/users/2015-August/008644.html

[Tom Harbert (tommyh)]

I ran a git bisect with:

# bad: [291395b47cff7cf1c2ef3f51ea10ff1859888876] UBUNTU: Ubuntu-lts-3.16.0-39.53~14.04.1
# good: [991bc91294525e4fb701f2c9a435215b2223d81a] UBUNTU: Ubuntu-lts-3.16.0-38.52~14.04.1

I believe the bug was introduced with:

# first bad commit: [07cb1b8e7b70f7a0a0afe4657e9854fe85e1bd23] skbuff: Do not scrub skb mark within the same name space

I am going to test the upstream kernel and will post the results.

[Clemens Schrimpe (clemens-schrimpe)]

> I am going to test the upstream kernel and will post the results.

Thanks. I just did not find the time to set up a test machine. The one where I discovered this is „in production“ (hence not available for testing), I’m afraid.

Greetings,

 Clemens

[Simon Déziel (sdeziel)]

Marking as confirmed thanks to Tom's bisection results.

[Emre Eraltan (eeraltan)]

> I am going to test the upstream kernel and will post the results.

I did the same ping test between two Ubuntu machines using VTI interfaces running the same kernel versions on each side and here are the results:

- 3.19.0-25-generic: ping doesnt work but ICMP echo/replies can be observed thru tcpdump on the VTI interface (also encrypted packet on the physical NIC)
- 4.2.0-18-generic: ping works as expected (encryption still works)

I hope it helps

[Emre Eraltan (eeraltan)]

Another test using 3.19.0-64-generic shows that the ping works again on this version.

Again hope it helps anyone having the same issues