x86_64 SMP load balancing broken

[Dalle]

To reproduce the problem:

1) Find yourself an X86_64 machine with 2 CPUs. In my case it is Feisty on 2 AMD Opteron(tm) Processor 252.
    uname -a reports: Linux dalburk 2.6.20-16-generic #2 SMP Thu Jun 7 19:00:28 UTC 2007 x86_64 GNU/Linux

2) Write a program that consumes CPU cycles and does little or no I/O. E.g

#include <stdio.h>
int main()
{
  int i, junk = 0;
  for (i = 1; i < 1000000000; ++i)
    junk += i * i + i / i;
  return junk;
}

3) Compile it (without optimization, we want it to take a noticeable amount of time). On my machine it completes in about 20 seconds which is enough to illustrate the problem. You might need to adjust the upper limit in the loop to get suitable timing.

4) Start three copies of the program:
    ./a.out & ./a.out & ./a.out &

5) Check the running processes with top. I see one process consuming close to 100% CPU on one processor, lets call that process A, and two processes consuming about 50% CPU each on the other processor, lets call the B and C.

6) Either wait until process A terminates or kill it. At this point one expects the processes B and C to get one processor each, but they still execute on the same processor getting only 50% each. (It is possible to force them onto another processor by changing the affinity with taskset, but that is the kernels job, isn't it?)

I've tried repeating it on the feisty Fawn x86_64 Live CD and it behaves the same way, so it shouldn't be a problem with anything I've done after installation. I've tried to repeat it on the Feisty Fawn i386 live CD but the i386 version seems to handle the rebalancing correctly.

[Launchpad Janitor (janitor)]

Beginning with the Hardy Heron 8.04 development cycle, all open Ubuntu kernel bugs need to be reported against the "linux" kernel package. We are automatically migrating this bug to the new "linux" package. However, development has already began for the upcoming Intrepid Ibex 8.10 release. It would be helpful if you could test the upcoming release and verify if this is still an issue - http://www.ubuntu.com/testing . If the issue still exists, please update this report by changing the Status of the "linux" task from "Incomplete" to "New". We appreciate your patience and understanding as we make this transition. Thanks!

[Leann Ogasawara (leannogasawara)]

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

[Dalle (jan-erik-dahlin)]

I no longer have the Opteron box, so I'm afraid I cannot help you with
testing.

Regards,

/Dalle

[Leann Ogasawara (leannogasawara)]

Hi Dalle,

Thanks for the update. I'm going to close this report since you commented you no longer have access to the hardware to help debug/test. If anyone else is experiencing this issue and is able to test and can verify the issue still exists with 2.6.27, please reopen this report by setting the status back to "New". Thanks.