Setting net.ipv4.neigh.default.gc_thresh1/2/3 on 3.13.0-97.144 or later causes 'invalid argument' error

[Sam Kottler]

Setting net.ipv4.neigh.default.gc_thresh1/2/3 on the 3.13.0-96-generic kernel and below worked fine using the following:

$ sudo /sbin/sysctl -w net.ipv4.neigh.default.gc_thresh1=128
128

As of 3.13.0-97-generic, though, it no longer works as expected:

$ sudo /sbin/sysctl -w net.ipv4.neigh.default.gc_thresh1=128
error: "Invalid argument" setting key "net.ipv4.neigh.default.gc_thresh1"

There are two potentially related revisions in the -97 release - http://kernel.ubuntu.com/git/ubuntu/ubuntu-precise.git/commit/?h=lts-backport-trusty&id=563cf19389d8e999e69d6c94995966aeaf7c3a08 and http://kernel.ubuntu.com/git/ubuntu/ubuntu-precise.git/commit/?h=lts-backport-trusty&id=7b82096b0ebc9bf487b390fe970d66ffa5a5774e.

The expected behavior is that changing neighbor table settings works using 'sysctl -w' with the same values as before the -97 kernel.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.13.0-98-generic 3.13.0-98.145~precise1
ProcVersionSignature: Ubuntu 3.13.0-98.145~precise1-generic 3.13.11-ckt39
Uname: Linux 3.13.0-98-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version k3.13.0-98-generic.
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu17.13
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D1p', '/dev/snd/midiC0D0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info: Error: [Errno 2] No such file or directory
Card0.Amixer.values: Error: [Errno 2] No such file or directory
CurrentDmesg: [ 4.586115] NET: Registered protocol family 40
Date: Wed Oct 19 06:13:36 2016
HibernationDevice: RESUME=UUID=d32abfeb-623f-4c11-ae4a-a7babe8c2139
InstallationMedia: Ubuntu-Server 12.04.5 LTS "Precise Pangolin" - Release amd64 (20140806.1)
IwConfig: Error: [Errno 2] No such file or directory
MachineType: VMware, Inc. VMware Virtual Platform
MarkForUpload: True
ProcEnviron:
 LANGUAGE=en_US:
 TERM=screen-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 svgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-98-generic root=UUID=febea164-78c3-4d3a-a0ce-0f09990df608 ro find_preseed=/preseed.cfg noprompt quiet
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-98-generic N/A
 linux-backports-modules-3.13.0-98-generic N/A
 linux-firmware 1.79.16
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux-lts-trusty
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/02/2015
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd07/02/2015:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.

[Thomas Copeland (copeland-tj)]

I've bisected Ubuntu-3.13.0-96.143 to Ubuntu3.13.0-97.144 and it looks like 7b82096b0ebc9bf487b390fe970d66ffa5a5774e introduced the problem.

thomasco@dev:/storage/ubuntu-trusty (7b82096...|BISECTING) $ git bisect bad
7b82096b0ebc9bf487b390fe970d66ffa5a5774e is the first bad commit
commit 7b82096b0ebc9bf487b390fe970d66ffa5a5774e
Author: Jiri Pirko <email address hidden>
Date: Sat Dec 7 19:26:56 2013 +0100

    neigh: restore old behaviour of default parms values

    commit 1d4c8c29841b9991cdf3c7cc4ba7f96a94f104ca upstream.

    Previously inet devices were only constructed when addresses are added.
    Therefore the default neigh parms values they get are the ones at the
    time of these operations.

    Now that we're creating inet devices earlier, this changes the behaviour
    of default neigh parms values in an incompatible way (see bug #8519).

    This patch creates a compromise by setting the default values at the
    same point as before but only for those that have not been explicitly
    set by the user since the inet device's creation.

    Introduced by:
    commit 8030f54499925d073a88c09f30d5d844fb1b3190
    Author: Herbert Xu <email address hidden>
    Date: Thu Feb 22 01:53:47 2007 +0900

        [IPV4] devinet: Register inetdev earlier.

    Signed-off-by: Jiri Pirko <email address hidden>
    Signed-off-by: David S. Miller <email address hidden>
    BugLink: https://bugs.launchpad.net/bugs/994931
    Signed-off-by: Kamal Mostafa <email address hidden>
    Acked-by: Tim Gardner <email address hidden>
    Acked-by: Brad Figg <email address hidden>

:040000 040000 b95e2490d8d94ffb3c4b5818b0ca05e2cdff17d2 7296def1ea38ae1c46ec1a6ba4df3d26fe568545 M include
:040000 040000 e78478aead22c4947b2b4f816a244272613fdc08 12e3e33ddae9c141ae77b6eb471ab5b9d5214db8 M net

thomasco@dev:/storage/ubuntu-trusty (7b82096...|BISECTING) $ git bisect log
# bad: [002556d23100065c43d60bfdab94fedf0a30cca6] UBUNTU: Ubuntu-3.13.0-97.144
# good: [d8590fec6790ab84c56f483bef6ce0caf71a40af] UBUNTU: Ubuntu-3.13.0-96.143
git bisect start 'Ubuntu-3.13.0-97.144' 'Ubuntu-3.13.0-96.143'
# good: [e5e45738175efd66844287ddcdf75e0e756fbd52] PCI: Add Netronome vendor and device IDs
git bisect good e5e45738175efd66844287ddcdf75e0e756fbd52
# good: [a70a25682944ac342dae0add0a4bb4c443e93411] Revert "UBUNTU: SAUCE: (no-up) ipv6: make the net.ipv6.conf.all.use_tempaddr sysctl propagate to interface settings"
git bisect good a70a25682944ac342dae0add0a4bb4c443e93411
# good: [6235189019f6136cb28646c2e637b76217a8d837] neigh: use tbl->family to distinguish ipv4 from ipv6
git bisect good 6235189019f6136cb28646c2e637b76217a8d837
# bad: [68a4a771678dd24d5eb30801d43412afb48723f5] neigh: ipv6: respect default values set before an address is assigned to device
git bisect bad 68a4a771678dd24d5eb30801d43412afb48723f5
# bad: [7b82096b0ebc9bf487b390fe970d66ffa5a5774e] neigh: restore old behaviour of default parms values
git bisect bad 7b82096b0ebc9bf487b390fe970d66ffa5a5774e
# first bad commit: [7b82096b0ebc9bf487b390fe970d66ffa5a5774e] neigh: restore old behaviour of default parms values

[Joseph Salisbury (jsalisbury)]

I built a Trusty test kernel with a revert of mainline commit 1d4c8c2. This also required that I revert commit bba24896f.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1634892/

Can you test this kernel and see if it resolves this bug?

[Joseph Salisbury (jsalisbury)]

Note, for this test kernel, you need to install both the linux-image and linux-image-extra .deb packages.

[Thomas Copeland (copeland-tj)]

Works for me

thomasco@dev:/proc/sys/net/ipv4/neigh/default$ uname -a
Linux kerneltest-uswest1adevc 3.13.0-100-generic #147~lp1634892Commit1d4c8c2Reverted SMP Mon Oct 24 18:04:20 UTC x86_64 x86_64 x86_64 GNU/Linux
thomasco@dev:/proc/sys/net/ipv4/neigh/default$ echo 4096 | sudo tee gc_thresh3
4096
thomasco@dev:/proc/sys/net/ipv4/neigh/default$ echo 8192 | sudo tee gc_thresh3
8192

[Sam Kottler (skottler)]

The kernel built above work fine for me too.

[Yoshi Kadokawa (yoshikadokawa)]

It worked for me too.

[Joseph Salisbury (jsalisbury)]

I built another Trusty test kernel. However, this kernel is with a cherry pick of commit 9ecf07a1d8f7 and without the reverts.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1634892/

Can you test this kernel and see if it resolves this bug?

[Yoshi Kadokawa (yoshikadokawa)]

I have tried it with commit9ecf07a1d8 version, but got "Invalid argument" again.
Here is the output.

ubuntu@testkernel:~$ uname -a
Linux testkernel 3.13.0-100-generic #147~lp1634892WithCommit9ecf07a1d8 SMP Wed Oct 26 18:36:02 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@testkernel:~$ sudo sysctl -w net.ipv4.neigh.default.gc_thresh1=8192
sysctl: setting key "net.ipv4.neigh.default.gc_thresh1": Invalid argument
net.ipv4.neigh.default.gc_thresh1 = 8192

[Joseph Salisbury (jsalisbury)]

We might need to perform a "Reverse" bisect to identify the fix that prevented this commit from being reverted in newer kernel versions.

Can you first confirm that the bug also exists in upstream v3.14-rc1? It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.14-rc1-trusty/

If the bug DOES also exist in v3.14-rc1, then test v3.15:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.15-utopic/

We can decide the next step based on the results of these two test.

[Yoshi Kadokawa (yoshikadokawa)]

Had the same Invalid argument error in 3.14-rc1 as well,
but for 3.15.0-031500, I could set the parameter successfully.

[Joseph Salisbury (jsalisbury)]

Thanks for testing, Yoshi. That is good news and means there is a fix for this already, without requiring a revert. We just need to narrow down which version the fix landed in, and then identify that commit.

Lets first test some of the 3.14 release candidates, newer than -rc1. Can you test the following:

v3.14-rc4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.14-rc4-trusty/

If v3.14-rc4 does NOT exhibit the bug then test v3.14-rc3:
v3.14-rc3: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.14-rc3-trusty/

If v3.14-rc4 DOES exhibit the bug then test v3.14-rc6:
v3.14-rc6: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.14-rc6-trusty/

Thanks!

[Thomas Copeland (copeland-tj)]

rc4 has the same problem:

thomasco@dev:/proc/sys/net/ipv4/neigh/default$ uname -a
Linux dev 3.14.0-031400rc4-generic #201402232235 SMP Mon Feb 24 03:36:35 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
thomasco@dev:/proc/sys/net/ipv4/neigh/default$ sudo sysctl -w net.ipv4.neigh.default.gc_thresh3=8192
sysctl: setting key "net.ipv4.neigh.default.gc_thresh3": Invalid argument
net.ipv4.neigh.default.gc_thresh3 = 8192

rc6 works though

thomasco@dev:~$ uname -a
Linux dev 3.14.0-031400rc6-generic #201403100035 SMP Mon Mar 10 04:36:54 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
thomasco@dev:~$ sudo sysctl -w net.ipv4.neigh.default.gc_thresh3=8192
net.ipv4.neigh.default.gc_thresh3 = 8192
thomasco@dev:~$ cat /proc/sys/net/ipv4/neigh/default/gc_thresh3
8192

[Thomas Copeland (copeland-tj)]

This sysctl also fails in rc5, so the fix must have landed in rc6.

thomasco@dev:~$ uname -a
Linux dev 3.14.0-031400rc5-generic #201403022235 SMP Mon Mar 3 03:36:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
thomasco@dev:~$ sudo sysctl -w net.ipv4.neigh.default.gc_thresh3=8192
sysctl: setting key "net.ipv4.neigh.default.gc_thresh3": Invalid argument
net.ipv4.neigh.default.gc_thresh3 = 8192

[Joseph Salisbury (jsalisbury)]

Great, thanks for testing and finding the version! I'll review the commit logs for rc6 and see if anything sticks out as the fix. If not. we can reverse bisect between rc5 and rc6 to identify the fix.

[Joseph Salisbury (jsalisbury)]

Well that was easy, now that you identified rc6 as working. I'll be willing to bet this is the fix:

commit b194c1f1dbd5f2671e49e0ac801b1b78dc7de93b
Author: Jiri Pirko <email address hidden>
Date: Fri Feb 21 14:52:57 2014 +0100

    neigh: fix setting of default gc_* values

    This patch fixes bug introduced by:
    commit 1d4c8c29841b9991cdf3c7cc4ba7f96a94f104ca
    "neigh: restore old behaviour of default parms values"

    The thing is that in neigh_sysctl_register, extra1 and extra2 which were
    previously set for NEIGH_VAR_GC_* are overwritten. That leads to
    nonsense int limits for gc_* variables. So fix this by not touching
    extra* fields for gc_* variables.

    Signed-off-by: Jiri Pirko <email address hidden>
    Signed-off-by: David S. Miller <email address hidden>

I'll build a Trusty test kernel with a pick of this commit and post a link to it shortly.

[Joseph Salisbury (jsalisbury)]

I built another Trusty test kernel. However, this kernel is with a cherry pick of commit b194c1f1dbd5f2671e49e0ac801b1b78dc7de93b and without the reverts.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1634892/

Can you test this kernel and see if it resolves this bug?

[Thomas Copeland (copeland-tj)]

Sure enough, that fixes it

thomasco@dev:~$ uname -a
Linux dev 3.13.0-100-generic #147~lp1634892WithCommitb194c1f1d SMP Fri Oct 28 19:23:46 UTC 20 x86_64 x86_64 x86_64 GNU/Linux
thomasco@dev:~$ sudo sysctl -w net.ipv4.neigh.default.gc_thresh3=8192
net.ipv4.neigh.default.gc_thresh3 = 8192
thomasco@dev:~$ cat /proc/sys/net/ipv4/neigh/default/gc_thresh3
8192

[Joseph Salisbury (jsalisbury)]

Great news! Thanks again for testing. I'll submit an SRU request to have this commit included in 16.04(Trusty).

[Ryan Cox (ryan-a-cox)]

Thanks for the quick work on this. Any idea when this fix might land in the linux-image-generic package?

[Thomas Copeland (copeland-tj)]

Did this ever get submitted?

[Joseph Salisbury (jsalisbury)]

Yes, and SRU request has now been submitted to the kernel-team mailing list. This bug will be marked as "Fix Commited" when the fix lands in the -proposed repository.

[Ryan Cox (ryan-a-cox)]

Will it follow the schedule found here: https://lists.ubuntu.com/archives/kernel-sru-announce/2016-October/000076.html ?

cycle: 04-Oct through 26-Nov
====================================================================
         04-Nov Last day for kernel commits for this cycle
07-Nov - 12-Nov Kernel prep week.
13-Nov - 25-Nov Bug verification & Regression testing..
         28-Nov Release to -updates.

[Hajo Locke (hajo-locke)]

When this fix will appear in official kernel-release?
Yesterday i updated Trusty Tahr/14.04 to 3.13.0-103-generic and this problem is still present.
This is a problem for all people who use 14.04 in high frequented environments, just like nameservers.

[Hajo Locke (hajo-locke)]

Today we updated 14.04 to 3.13.0-105-generic, bug still present.

[Thomas Copeland (copeland-tj)]

Looks like this got punted to 3.13.0-106, which I installed it from the canonical-kernel-team ppa. This bug appears to be solved in -106.

thomasco@dev:~$ uname -a
Linux dev 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
thomasco@dev:~$ sudo sysctl -w net.ipv4.neigh.default.gc_thresh3=4096
net.ipv4.neigh.default.gc_thresh3 = 4096
thomasco@dev:~$ cat /proc/sys/net/ipv4/neigh/default/gc_thresh3
4096
thomasco@dev:~$

[Richard Arends (l-lauuchpad-s)]

What is the current planned release date of this kernel? We also have issues that we must solve and downgrading is not really an option.

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'. If the problem still exists, change the tag 'verification-needed-trusty' to 'verification-failed-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Hajo Locke (hajo-locke)]

I also upgraded to 3.13.0-106-generic from canonical-kernel-team ppa and can confirm it is working again:

# sysctl -w net.ipv4.neigh.default.gc_thresh1=8192 net.ipv4.neigh.default.gc_thresh1 = 8192

# uname -a
Linux myhostname 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

How to add tags?

[Richard Arends (l-lauuchpad-s)]

Installed linux-image-3.13.0-106-generic_3.13.0-106.153_amd64.deb and linux-image-extra-3.13.0-106-generic_3.13.0-106.153_amd64.deb and can confirm that the sysctl settings are working again

$ uname -a
Linux server1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ sudo sysctl -a |egrep net.ipv4.neigh.default.gc
net.ipv4.neigh.default.gc_interval = 120
net.ipv4.neigh.default.gc_stale_time = 240
net.ipv4.neigh.default.gc_thresh1 = 256
net.ipv4.neigh.default.gc_thresh2 = 1024
net.ipv4.neigh.default.gc_thresh3 = 2048

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-lts-trusty - 3.13.0-106.153~precise1

---------------
linux-lts-trusty (3.13.0-106.153~precise1) precise; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1647755

  * CVE-2016-7916
    - proc: prevent accessing /proc/<PID>/environ until it's ready

  * CVE-2016-6213
    - mnt: Add a per mount namespace limit on the number of mounts

  * aio completions are dropped (LP: #1641129)
    - aio: fix reqs_available handling

  * [Hyper-V] do not lose pending heartbeat vmbus packets (LP: #1632786)
    - hv: do not lose pending heartbeat vmbus packets

  * ipv6: connected routes are missing after a down/up cycle on the loopback
    (LP: #1634545)
    - ipv6: reallocate addrconf router for ipv6 address when lo device up
    - ipv6: correctly add local routes when lo goes up

  * audit: prevent a new auditd to stop an old auditd still alive (LP: #1633404)
    - audit: stop an old auditd being starved out by a new auditd

  * Setting net.ipv4.neigh.default.gc_thresh1/2/3 on 3.13.0-97.144 or later
    causes 'invalid argument' error (LP: #1634892)
    - neigh: fix setting of default gc_* values

  * move nvme driver to linux-image (LP: #1640275)
    - [Config] Add nvme to the generic inclusion list

 -- Luis Henriques <email address hidden> Tue, 06 Dec 2016 15:50:24 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-106.153

---------------
linux (3.13.0-106.153) trusty; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1647749

  * CVE-2016-7916
    - proc: prevent accessing /proc/<PID>/environ until it's ready

  * CVE-2016-6213
    - mnt: Add a per mount namespace limit on the number of mounts

  * aio completions are dropped (LP: #1641129)
    - aio: fix reqs_available handling

  * [Hyper-V] do not lose pending heartbeat vmbus packets (LP: #1632786)
    - hv: do not lose pending heartbeat vmbus packets

  * ipv6: connected routes are missing after a down/up cycle on the loopback
    (LP: #1634545)
    - ipv6: reallocate addrconf router for ipv6 address when lo device up
    - ipv6: correctly add local routes when lo goes up

  * audit: prevent a new auditd to stop an old auditd still alive (LP: #1633404)
    - audit: stop an old auditd being starved out by a new auditd

  * Setting net.ipv4.neigh.default.gc_thresh1/2/3 on 3.13.0-97.144 or later
    causes 'invalid argument' error (LP: #1634892)
    - neigh: fix setting of default gc_* values

  * move nvme driver to linux-image (LP: #1640275)
    - [Config] Add nvme to the generic inclusion list

 -- Luis Henriques <email address hidden> Tue, 06 Dec 2016 15:00:27 +0000