Kernel bug (soft lockup)

[Ashwin]

[ 342.917812] TCP: too many of orphaned sockets
[ 346.359618] TCP: too many of orphaned sockets
[ 349.806952] TCP: too many of orphaned sockets
[ 926.602159] TCP: too many of orphaned sockets
[ 933.725660] TCP: too many of orphaned sockets
[ 1090.441880] BUG: soft lockup detected on CPU#3!
[ 1090.442146]
[ 1090.442146] Call Trace:
[ 1090.442148] <IRQ> [<ffffffff802bc809>] softlockup_tick+0xf9/0x120
[ 1090.442166] [<ffffffff80211bff>] __do_softirq+0x5f/0xd0
[ 1090.442170] [<ffffffff802997d7>] update_process_times+0x57/0x90
[ 1090.442174] [<ffffffff8027a864>] smp_local_timer_interrupt+0x34/0x60
[ 1090.442177] [<ffffffff8027afd9>] smp_apic_timer_interrupt+0x59/0x80
[ 1090.442181] [<ffffffff80261ce6>] apic_timer_interrupt+0x66/0x70
[ 1090.442183] <EOI> [<ffffffff80227150>] tcp_sendmsg+0x0/0xb00
[ 1090.442190] [<ffffffff8024553c>] sk_stream_mem_schedule+0x6c/0x1d0
[ 1090.442194] [<ffffffff802272f8>] tcp_sendmsg+0x1a8/0xb00
[ 1090.442198] [<ffffffff80257114>] sock_sendmsg+0x104/0x140
[ 1090.442204] [<ffffffff80249415>] sock_aio_write+0x145/0x180
[ 1090.442209] [<ffffffff8022e084>] mntput_no_expire+0x24/0xb0
[ 1090.442213] [<ffffffff80217e1f>] do_sync_write+0xcf/0x120
[ 1090.442217] [<ffffffff80270c14>] do_gettimeofday+0x44/0x90
[ 1090.442222] [<ffffffff802a3820>] autoremove_wake_function+0x0/0x30
[ 1090.442225] [<ffffffff802a5ca4>] enqueue_hrtimer+0x54/0x80
[ 1090.442228] [<ffffffff8023e481>] hrtimer_start+0xd1/0x100
[ 1090.442234] [<ffffffff802165c4>] vfs_write+0xf4/0x1a0
[ 1090.442237] [<ffffffff802170e3>] sys_write+0x53/0x90
[ 1090.442241] [<ffffffff8026111e>] system_call+0x7e/0x83
[ 1090.442245]

OS: Linux ****** 2.6.20-15-server #2 SMP Sun Apr 15 06:22:36 UTC 2007 x86_64 GNU/Linux
This is the Feisty kernel.

I have quad processor x64 machine with 4gb:

From /proc/cpu
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU X3220 @ 2.40GHz
stepping : 7
cpu MHz : 2400.086
cache size : 4096 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips : 4803.52
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

I was stress testing the system. Edit /etc/apache2/apache.conf:
<IfModule worker.c>
ServerLimit 400
ThreadLimit 4000
StartServers 8
MaxClients 20000
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 2000
MaxRequestsPerChild 0

Make sure apache2-mpm-worker and ab (apache bench) is installed.

Then do this:
$ sudo su
# ulimit -n 65535
# ab -c 20000 -n 40000 https://not.127.0.addr.but.real.ip.adddress/

Wait.

[Launchpad Janitor (janitor)]

Beginning with the Hardy Heron 8.04 development cycle, all open Ubuntu kernel bugs need to be reported against the "linux" kernel package. We are automatically migrating this bug to the new "linux" package. However, development has already began for the upcoming Intrepid Ibex 8.10 release. It would be helpful if you could test the upcoming release and verify if this is still an issue - http://www.ubuntu.com/testing . If the issue still exists, please update this report by changing the Status of the "linux" task from "Incomplete" to "New". We appreciate your patience and understanding as we make this transition. Thanks!

[Leann Ogasawara (leannogasawara)]

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

[Michele Mangili (mangilimic)]

We are closing this bug report because it lacks the information we need to investigate the problem, as described in the previous comments. Please reopen it if you can give us the missing information, and don't hesitate to submit bug reports in the future. To reopen the bug report you can click on the current status, under the Status column, and change the Status back to "New". Thanks again!