network performance and cpu usage in gutsy vs feisty

[BullCreek]

Binary package hint: kernel-image-2.6.22-14-generic-di

Gigabit ethernet adapters use 400-500% more CPU in Gutsy compared to Feisty for the same load.

To reproduce:

1. Find two computers with gigabit NICs (I've tested with e1000, rt8169, tg3, sky2, forcedeth) plugged in to a suitable gigabit switch or connect with a suitable cross over cable.

2. Boot Feisty LiveCD in both computers, enable universe, and "sudo apt-get install iperf" on both computers. Then run iperf -s on one and iperf -c on the other and observe CPU usage for iperf (in the 10-20% range).

3. Repeat same procedure with Gutsy LiveCD. Note CPU usage for iperf (90-100+%).

This should go in the Gutsy caveats in a very visible place as I don't think very many people will want to upgrade their servers if they know this. Just a little searching revealed this possibly related discussion on the LKML:

http://<email address hidden>/msg213242.html

[Emanuel Fonseca (emdfonseca)]

I can confirm this bug... I get this issue when running iperf with any bandwidth (i.e. -b 1M) or device (broadcom 3c59x, ipw2200, atheros)...

[Brian Murray (brian-murray)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug is fixed in the development release of Ubuntu, Hardy Heron.

[Brian Murray (brian-murray)]

I found a patch in the lkml thread at http://<email address hidden>/msg213559.html. This is essentially a one line patch to the kernel that has been well tested since it is Hardy also. I think it'd be worthwhile to get this fixed in Gutsy. The patch follows:

Problem spotted by Denys Fedoryshchenko

Signed-off-by: Eric Dumazet <[EMAIL PROTECTED]>

--- linux-2.6.22/drivers/char/random.c 2007-10-01 10:18:42.000000000 +0200
+++ linux-2.6.22-ed/drivers/char/random.c 2007-10-01 21:47:58.000000000
+0200
@@ -1550,11 +1550,13 @@ __u32 secure_tcp_sequence_number(__be32
         * As close as possible to RFC 793, which
         * suggests using a 250 kHz clock.
         * Further reading shows this assumes 2 Mb/s networks.
- * For 10 Gb/s Ethernet, a 1 GHz clock is appropriate.
- * That's funny, Linux has one built in! Use it!
- * (Networks are faster now - should this be increased?)
+ * For 10 Mb/s Ethernet, a 1 MHz clock is appropriate.
+ * For 10 Gb/s Ethernet, a 1 GHz clock should be ok, but
+ * we also need to limit the resolution so that the u32 seq
+ * overlaps less than one time per MSL (2 minutes).
+ * Choosing a clock of 64 ns period is OK. (period of 274 s)
         */
- seq += ktime_get_real().tv64;
+ seq += ktime_get_real().tv64 >> 6;
 #if 0
        printk("init_seq(%lx, %lx, %d, %d) = %d\n",
               saddr, daddr, sport, dport, seq);

[Leann Ogasawara (leannogasawara)]

Adding git commit id and description just for the kernel team to reference as well. Thanks.

commit 9b42c336d06411e6463949d2dac63949f66ff70b
Author: Eric Dumazet <email address hidden>
Date: Mon Oct 1 13:58:36 2007 -0700

    [TCP]: secure_tcp_sequence_number() should not use a too fast clock

    TCP V4 sequence numbers are 32bits, and RFC 793 assumed a 250 KHz clock.
    In order to follow network speed increase, we can use a faster clock, but
    we should limit this clock so that the delay between two rollovers is
    greater than MSL (TCP Maximum Segment Lifetime : 2 minutes)

    Choosing a 64 nsec clock should be OK, since the rollovers occur every
    274 seconds.

    Problem spotted by Denys Fedoryshchenko

    [ This bug was introduced by f85958151900f9d30fa5ff941b0ce71eaa45a7de ]

    Signed-off-by: Eric Dumazet <email address hidden>
    Signed-off-by: David S. Miller <email address hidden>

[Tim Gardner (timg-tpi)]

SRU Justification:

Impact: Network traffic consumes an inordinate amount of CPU

Fix Description: Reduce the number of TCP sequence number rollovers. This same code is in Hardy and subsequent versions.

Patch: http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-gutsy.git;a=commit;h=503a509b951363785ffdc8942d28c51b180ea27f

TEST CASE: See first comment.

[Tim Gardner (timg-tpi)]

http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-gutsy.git;a=commit;h=503a509b951363785ffdc8942d28c51b180ea27f

[Tim Gardner (timg-tpi)]

linux-source-2.6.22 (2.6.22-15.54) gutsy-security; urgency=low

  [Colin Ian King]

  * usb-storage: always set the allow_restart flag
    - LP: #193154

  [Upstream Kernel Changes]

  * [IA64] Fix unaligned handler for floating point instructions with base
    update
  * CVE-2007-6694: [POWERPC] CHRP: Fix possible NULL pointer dereference
  * vm audit: add VM_DONTEXPAND to mmap for drivers that need it
    (CVE-2008-0007)
  * fix SMP ordering hole in fcntl_setlk() (CVE-2008-1669)
  * Fix dnotify/close race (CVE-2008-1375)
  * [TCP]: secure_tcp_sequence_number() should not use a too fast clock
  * hrtimer: check relative timeouts for overflow
  * netdrvr: natsemi: Fix device removal bug

 -- Ben Collins <email address hidden> Mon, 19 May 2008 23:03:41 +0000