IPv6 with LVS Performance issue in latest 3.13LTS kernels

[Marc Hasson]

We experienced a major performance regression between 12.04's 3.2 kernels
and 14.04's 3.13 kernels when using IPv6 with the LVS load-balancing
facility. Through analysis of perf events and a workaround we've
determined that an upstream fix is available which addresses the issue.
Ubuntu has picked up the "IPv6: remove rt6i_genid" fix that appears to
address our issue in their 3.16 and later kernels. This was checked
into the upstream 3.16 kernel back in late 2014. This fix addressed an
issue introduced by the (late 2012) "ipv6: use net->rt_genid to check dst
validity", which is the source of our issue due to the mismatch between
a dst/route instantiation's rt6i_genid and the IPv6 rt_genid field.

Since we have drivers and other software tied to the 3.13 kernels in
the field this report is requesting that the backport of that upstream
fix be done to the 3.13 LTS kernels as well since we were planning on
several more years for those deployed systems. It seems relatively
straight-forward.

In our 3.13 kernel we used the "systemtap" facility in a test system
to temporarily address the "obsolete" determination mistakenly made by
the ip6_dst_check() function to check dst validity on behalf of the LVS
(ip_vs) code. By updating rt6i_genid to the current global value we were
able to restore our test systems to the previous performance obtained
with the 3.2 kernels. But clearly we want the official upstream fix
incorporated, which pulls the troublesome rt6i_genid field out altogether
since its mishandling affected more than IPv6/LVS support based on the
upstream mail threads. Those were made in mid-2014 to the upstream
kernel folks, such as a per-socket route caching issue.

Note that the apport-bug collected info from a 3.13.0-87 system but we see the
issue on all 3.13.0-xxx kernels.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-87-generic 3.13.0-87.133
ProcVersionSignature: Ubuntu 3.13.0-87.133-generic 3.13.11-ckt39
Uname: Linux 3.13.0-87-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.21
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: marc 2433 F.... pulseaudio
Date: Mon Aug 29 20:05:57 2016
HibernationDevice: RESUME=UUID=c4187d86-ea40-4f53-af39-1b7e83964502
InstallationDate: Installed on 2014-04-30 (852 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
Lsusb:
 Bus 001 Device 003: ID 0e0f:0002 VMware, Inc. Virtual USB Hub
 Bus 001 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: VMware, Inc. VMware Virtual Platform
ProcEnviron:
 LANGUAGE=en_US
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 svgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-87-generic root=UUID=92211e82-1c0b-42e6-bb12-0003b7f6db54 ro quiet splash crashkernel=384M-:128M
PulseList:
 Error: command ['pacmd', 'list'] failed with exit code 1: Home directory not accessible: Permission denied
 No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-87-generic N/A
 linux-backports-modules-3.13.0-87-generic N/A
 linux-firmware 1.127.22
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/20/2014
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd05/20/2014:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

I built a Trusty test kernel with a cherry-pick of commit: 705f1c8. This commit also required commit 0c3584d58 as a prereq. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1618299/

Can you test this kernel and see if it resolves this bug?

Thanks in advance!

[Marc Hasson (mhassonsuspect)]

Thanks so much Joseph! My apologies on not seeing this earlier that you had a test kernel ready, I was offsite almost all of last week and swamped. Will download it now and test it later this evening or tomorrow morning in my test rig. By any chance, so as to aid my systemtap monitoring, do you have a matching dbgsyms package for your test kernel available?

[Marc Hasson (mhassonsuspect)]

Joseph, the commits you've included appear to be correct to me. I've tested this kernel as best I can without the dbgsym package. I really could use that package so I could verify the performance with "perf" as well as use "systemtap" to verify stuff is going down the right paths. But I did use the /proc/net/ipv6_route's lookups counter as an indication that excessive lookups, when routing generation changes were injected, were no longer occurring with your test kernel.

Bottom line: This test kernel looks like it has what we've been needing for our production servers, we'd appreciate any/all efforts for getting this into the earliest 3.13 kernel maintenance release you guys can manage.

Thanks!

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Marc Hasson (mhassonsuspect)]

Thanks so much for the rapid turnaround on this report guys. I've modified the tag to the verification-done-trusty, as requested. I pulled down all the linux-image, source, and dbgsyms for the 3.13.0-97.144 kernel from proposed for installing/testing. I verified the source code manually as well.

All looks good to me as well as I can determine before actual production here, lets go with it.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-98.145

---------------
linux (3.13.0-98.145) trusty; urgency=low

  * Fix GRO recursion overflow for tunneling protocols (LP: #1631287)
    - tunnels: Don't apply GRO to multiple layers of encapsulation.

  * CVE-2016-7039
    - SAUCE: net: add recursion limit to GRO

linux (3.13.0-97.144) trusty; urgency=low

  [ Joseph Salisbury ]

  * Release Tracking Bug
    - LP: #1626604

  * Altering use_tempaddr drops all IPv6 addresses (LP: #994931)
    - Revert "UBUNTU: SAUCE: (no-up) ipv6: Fix net.ipv6.conf.all.use_tempaddr
      sysctl"
    - Revert "UBUNTU: SAUCE: (no-up) ipv6: make the net.ipv6.conf.all.use_tempaddr
      sysctl propagate to interface settings"
    - neigh: convert parms to an array
    - neigh: wrap proc dointvec functions
    - neigh: use tbl->family to distinguish ipv4 from ipv6
    - neigh: restore old behaviour of default parms values
    - neigh: ipv6: respect default values set before an address is assigned to
      device

  * PCI quirk required for correct access to configuration space of NFP
    4000/6000 (LP: #1624267)
    - PCI: Support PCIe devices with short cfg_size
    - PCI: Add Netronome vendor and device IDs
    - PCI: Limit config space size for Netronome NFP6000 family
    - PCI: Add Netronome NFP4000 PF device ID
    - PCI: Limit config space size for Netronome NFP4000

  * CVE-2016-6136
    - audit: fix a double fetch in audit_log_single_execve_arg()

  * CVE-2016-6480
    - aacraid: Check size values after double-fetch from user

  * CVE-2016-6828
    - tcp: fix use after free in tcp_xmit_retransmit_queue()

  * IPv6 with LVS Performance issue in latest 3.13LTS kernels (LP: #1618299)
    - ipv6: remove prune parameter for fib6_clean_all
    - ipv6: remove rt6i_genid

  * lsattr 32bit does not work on 64bit kernel (Inappropriate ioctl error)
    (LP: #1619918)
    - btrfs: bugfix: handle FS_IOC32_{GETFLAGS, SETFLAGS, GETVERSION} in
      btrfs_ioctl

  * Miscellaneous upstream changes
    - powerpc/pseries: use pci_host_bridge.release_fn() to kfree(phb)

 -- Seth Forshee <email address hidden> Fri, 07 Oct 2016 20:08:32 -0500