Thread library bug for kernel >= 2.6.21

[Mike Carpenter]

Binary package hint: iperf

Taken from http://dast.nlanr.net/Projects/Iperf2.0/patch-iperf-linux-2.6.21.txt :

The latest version of Linux kernel aggravates a pre-existing iperf thread
library bug. The iperf thread library assumes that calling usleep(0) will
cause
the thread to yield so that other threads will run. This has never been a
documented
behavior of Linux/Unix. The new high resolution timer option in the kernel
causes
usleep(0) to be a nop so the thread keeps running (until it's quanta is
exhausted).

Without this fix, iperf will get poor performance because the monitoring
thread
may hog the cpu, keeping the sender/receiver threads from running.

The fix to iperf is easy, just use sched_yield() instead. The manual page
for sched_yield
says to test for POSIX_PRIORITY_SCHEDULING as a Posix option.

--- compat/Thread.c.orig 2005-05-03 08:15:51.000000000 -0700
+++ compat/Thread.c 2007-06-04 10:41:11.000000000 -0700
@@ -405,9 +405,13 @@
 void thread_rest ( void ) {
 #if defined( HAVE_THREAD )
 #if defined( HAVE_POSIX_THREAD )
- // TODO add checks for sched_yield or pthread_yield and call that
- // if available
+
+#if defined( _POSIX_PRIORITY_SCHEDULING )
+ sched_yield();
+#else
     usleep( 0 );
+#endif
+
 #else // Win32
     SwitchToThread( );
 #endif

[Antonio Barbuzzi (antoniob82)]

Same problem for me...
Solved with patch of Ingo Molnar from:
http://kerneltrap.org/Linux/Finding_Bugs_With_CFS

---
 compat/Thread.c | 3 ---
 src/Reporter.c | 13 +++++++++----
 src/main.cpp | 2 ++
 3 files changed, 11 insertions(+), 7 deletions(-)

Index: iperf-2.0.2/compat/Thread.c
===================================================================
--- iperf-2.0.2.orig/compat/Thread.c
+++ iperf-2.0.2/compat/Thread.c
@@ -405,9 +405,6 @@ int thread_numuserthreads( void ) {
 void thread_rest ( void ) {
 #if defined( HAVE_THREAD )
 #if defined( HAVE_POSIX_THREAD )
- // TODO add checks for sched_yield or pthread_yield and call that
- // if available
- usleep( 0 );
 #else // Win32
     SwitchToThread( );
 #endif
Index: iperf-2.0.2/src/Reporter.c
===================================================================
--- iperf-2.0.2.orig/src/Reporter.c
+++ iperf-2.0.2/src/Reporter.c
@@ -111,6 +111,7 @@ report_statistics multiple_reports[kRepo
 char buffer[64]; // Buffer for printing
 ReportHeader *ReportRoot = NULL;
 extern Condition ReportCond;
+extern Condition ReportDoneCond;
 int reporter_process_report ( ReportHeader *report );
 void process_report ( ReportHeader *report );
 int reporter_handle_packet( ReportHeader *report );
@@ -338,7 +339,7 @@ void ReportPacket( ReportHeader* agent,
             // item
             while ( index == 0 ) {
                 Condition_Signal( &ReportCond );
- thread_rest();
+ Condition_Wait( &ReportDoneCond );
                 index = agent->reporterindex;
             }
             agent->agentindex = 0;
@@ -346,7 +347,7 @@ void ReportPacket( ReportHeader* agent,
         // Need to make sure that reporter is not about to be "lapped"
         while ( index - 1 == agent->agentindex ) {
             Condition_Signal( &ReportCond );
- thread_rest();
+ Condition_Wait( &ReportDoneCond );
             index = agent->reporterindex;
         }

@@ -553,6 +554,7 @@ void reporter_spawn( thread_Settings *th
         }
         Condition_Unlock ( ReportCond );

+again:
         if ( ReportRoot != NULL ) {
             ReportHeader *temp = ReportRoot;
             //Condition_Unlock ( ReportCond );
@@ -575,9 +577,12 @@ void reporter_spawn( thread_Settings *th
                 // finished with report so free it
                 free( temp );
                 Condition_Unlock ( ReportCond );
+ Condition_Signal( &ReportDoneCond );
+ if (ReportRoot)
+ goto again;
             }
- // yield control of CPU is another thread is waiting
- thread_rest();
+ Condition_Signal( &ReportDoneCond );
+ usleep(10000);
         } else {
             //Condition_Unlock ( ReportCond );
         }
Index: iperf-2.0.2/src/main.cpp
===================================================================
--- iperf-2.0.2.orig/src/main.cpp
+++ iperf-2.0.2/src/main.cpp
@@ -96,6 +96,7 @@ extern "C" {
     // records being accessed in a report and also to
     // serialize modification of the report list
     Condition ReportCond;
+ Condition ReportDoneCond;
 }

 // global variables only accessed within this file
@@...

[Bernhard Schmidt (berni)]

This is a pretty serious performance issue. I've attached a patch against the original debian package to incorporate the Ingo Molnar fix (see Antonio above). It has great effect on system usage during iperf runs

bschmidt@lxbsc01:~/tmp$ time iperf -V -c ping.lrz-muenchen.de
------------------------------------------------------------
Client connecting to ping.lrz-muenchen.de, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 2001:4ca0:0:f000:211:43ff:fe7e:3a76 port 39596 connected with 2001:4ca0:0:101:250:56ff:fea9:56a1 port 5001
[ 3] 0.0-10.0 sec 1.03 GBytes 886 Mbits/sec

real 0m10.007s
user 0m2.244s
sys 0m13.769s

[System is very unresponsive during the test and iperf uses up to 180% cpu according to top]

bschmidt@lxbsc01:~/tmp$ sudo dpkg -i iperf_2.0.2-3.1_i386.deb
(Lese Datenbank ... 102562 Dateien und Verzeichnisse sind derzeit installiert.)
Vorbereiten zum Ersetzen von iperf 2.0.2-3 (durch iperf_2.0.2-3.1_i386.deb) ...
Entpacke Ersatz für iperf ...
Richte iperf ein (2.0.2-3.1) ...

bschmidt@lxbsc01:~/tmp$ time iperf -V -c ping.lrz-muenchen.de
------------------------------------------------------------
Client connecting to ping.lrz-muenchen.de, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 2001:4ca0:0:f000:211:43ff:fe7e:3a76 port 46070 connected with 2001:4ca0:0:101:250:56ff:fea9:56a1 port 5001
[ 3] 0.0-10.0 sec 1.06 GBytes 914 Mbits/sec

real 0m10.040s
user 0m0.120s
sys 0m4.748s

System is responsive, about 30% CPU usage

All of that on Ubuntu hardy. It would be highly appreciated if the Debian bug could be monitored and the bugfix be backported or integrated by Ubuntu before the hardy release.

[Bernhard Schmidt (berni)]

Bug has been fixed in Debian package version 2.0.2-4, can this be imported into Ubuntu before the hardy release?

[Daniel Hahler (blueyed)]

Thank you for reporting this bug.
I've filed a sync request at bug 204948.

[Daniel Hahler (blueyed)]

The package has been synced.