ip command becomes slower with large number of network devices

Bug #1281366 reported by Chris J Arges on 2014-02-18
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
iproute (Ubuntu)
Undecided
Unassigned
Precise
Medium
Chris J Arges
Quantal
Medium
Unassigned

Bug Description

[Impact]
When using iproute commands with a large number of network devices (virtual ethernet adapters, etc) iproute performance becomes much slower.

[Test Case]
1) Download attached 'make_fake_routers.sh' and 'parse.py' scripts.
2) ./make_fake_routers.sh 3000 ip > results
3) ./parse.py results

You should be able to graph and clearly see a performance improvement with the fix than without it.

[Regression Potential]
This patch fixes the issue:
https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/commit/?id=f0124b0f0aa0e5b9288114eb8e6ff9b4f8c33ec8
This patch is upstream and is currently in saucy and beyond.

Chris J Arges (arges) wrote :
Changed in iproute (Ubuntu Precise):
assignee: nobody → Chris J Arges (arges)
Changed in iproute (Ubuntu Quantal):
assignee: nobody → Chris J Arges (arges)
Changed in iproute (Ubuntu Precise):
importance: Undecided → Medium
Changed in iproute (Ubuntu Quantal):
importance: Undecided → Medium
Changed in iproute (Ubuntu Precise):
status: New → In Progress
Changed in iproute (Ubuntu Quantal):
status: New → In Progress
Changed in iproute (Ubuntu):
status: New → Fix Released
Chris J Arges (arges) wrote :
Chris J Arges (arges) wrote :

uploaded for p/q

Hello Chris, or anyone else affected,

Accepted iproute into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/iproute/20120521-3ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in iproute (Ubuntu Quantal):
status: In Progress → Fix Committed
tags: added: verification-needed
Brian Murray (brian-murray) wrote :

Hello Chris, or anyone else affected,

Accepted iproute into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/iproute/20111117-1ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in iproute (Ubuntu Precise):
status: In Progress → Fix Committed
ThorstenK (tkundoch) wrote :

Hello Brian,

just upgraded from 20111117-1ubuntu2.1 to 20111117-1ubuntu2.2 an noticed that 'ip route' reports wrong interface device names ( if2 instead of eth0):

1ubuntu2.1:
root@maumau:/etc/apt# ip route show
default via abc.def.39.254 dev eth0 metric 100
abc.def.32.0/21 dev eth0 proto kernel scope link src abc.def.32.251
169.254.0.0/16 dev eth0 scope link metric 1000

1ubuntu2.2:
root@maumau:/etc/apt# ip route show
default via abc.def.39.254 dev if2 metric 100
abc.def.32.0/21 dev if2 proto kernel scope link src abc.def.32.251
169.254.0.0/16 dev if2 scope link metric 1000

Looks like the fix is not complete. Thanks. Thorsten

Chris J Arges (arges) wrote :

I can confirm this problem. I'll look into why this patch is causing this.

tags: added: verification-failed
removed: verification-needed
Ian! D. Allen (idallen) wrote :

Ubuntu 12.04.4 LTS 3.8.0-36-generic #52~precise1-Ubuntu SMP Mon Feb 3 21:54:46 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I just said "this bug affects me too", but what I really meant to say was "the fix is broken for me too" and shows "if" numbers instead of interface device names. I had to downgrade back to the previous version.

Chris J Arges (arges) wrote :

The ll_init_map removal code is dependent on a few others changes which I didn't backport correctly. There are many changes to the ll_map code that make this not a trivial backport as I had anticipated.

Adam Conrad (adconrad) wrote :

These SRUs have been removed from both precise-proposed and quantal-proposed due to failed verification.

Changed in iproute (Ubuntu Quantal):
status: Fix Committed → In Progress
Changed in iproute (Ubuntu Precise):
status: Fix Committed → In Progress
Rafael David Tinoco (inaddy) wrote :
Chris J Arges (arges) on 2014-04-15
Changed in iproute (Ubuntu Quantal):
assignee: Chris J Arges (arges) → nobody
status: In Progress → Won't Fix
Brian Murray (brian-murray) wrote :

Hello Chris, or anyone else affected,

Accepted iproute into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/iproute/20111117-1ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in iproute (Ubuntu Precise):
status: In Progress → Fix Committed
tags: removed: verification-failed
tags: added: verification-needed
Bryan Quigley (bryanquigley) wrote :

With package from #12:
$ ./parse.py results
5.6818
3.9063
2.4752
1.4535
1.1521
1.004
0.8621
0.7375
0.6477
0.5828
0.523
0.4762

With stock package, parse results:
5.2083
3.7313
2.381
1.4881
1.1848
1.0
0.8333
0.731
0.6378
0.5747
0.5133
0.4647

Bryan Quigley (bryanquigley) wrote :

Confirmed with time ./make_fake_routers.sh; the latest ubuntu2.3 package does not seem to solve the original issue as they both take the same amount of time.

tags: added: verification-failed
removed: verification-needed
Rafael David Tinoco (inaddy) wrote :

Bryan

Id like you to reconsider this test if possible. Better results have also been confirmed by others (including customers).

Fake routers creation performance depends on other issues (not only covered by this bug) like CPU scalability (since netlink generated work is delayed by kernel and offloaded to RCU callbacks). There are some other issues being investigated for fake router creation but for this specific bug I can confirm results several times.

Keep in mind that results are sometimes less then 10% better because of kernel RCU offload characteristics (like for example when using only 1 CPU for the test).

My testcase:

I'm using last good commit generated kernel for this case (34ed6246). After this specific commit (upstream kernel) performance for fake router creation brakes (I'll put other BUG information and # at the end of this case, when appropriate).

By now, from kernel 3.2 to kernel 3.9 (up to this commit), general behavior is to have RCU callbacks enabled (no configuration on what cpu to offload up to this point) and this is what "NOCB" stands for on the attached file.

Independently of the kernel (running on precise), iproute userland tool + this patch gives better results.

Please take a look at the attached file specifically on "average" fields.

Thanks in advance.

Mark Russell (marrusl) wrote :

I can also confirm that this patch works as hoped. It's been happily working in a very large production environment for a few weeks as well. Please promote! Thanks.

tags: added: verification-done
removed: verification-failed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package iproute - 20111117-1ubuntu2.3

---------------
iproute (20111117-1ubuntu2.3) precise; urgency=medium

  * Fix ip netns scaling issues for 1000s of routers. (LP: #1281366)
 -- Rafael David Tinoco <email address hidden> Tue, 15 Apr 2014 14:05:33 -0500

Changed in iproute (Ubuntu Precise):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for iproute has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers