BUG: soft lockup - CPU#7 stuck for 23s! [netstat:12121]

[Lluise]

Ubuntu 12.04 System running on Vmware ESXi 4.1.0
No graphical gui installed, only cli
Used as apache 2.2 server

Sometimes system has very slow responsiveness ant it appears to be stuck....
sometimes bash commands are very slow to execute, but no high cpu workload.

WORKAROUND: Removed irqbalance daemon from ubuntu server and unchecked "Synchronize guest time with host" from VM setting (VMWare esx).

[152312.575871] BUG: soft lockup - CPU#7 stuck for 23s! [netstat:12121]
[152313.161642] Modules linked in: xt_multiport pcnet32 vmxnet(O) vmblock(O) vmsync(O) vmhgfs(O) ext2 dm_crypt ip6t_LOG xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT ipt_LOG xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat ppdev nf_conntrack_ipv4 nf_defrag_ipv4 vmw_balloon nf_conntrack_ftp nf_conntrack psmouse iptable_filter ip_tables serio_raw x_tables parport_pc acpi_memhotplug lp parport vmci(O) vmxnet3 vmw_pvscsi floppy
[152314.013007] CPU 7
[152314.013010] Modules linked in: xt_multiport pcnet32 vmxnet(O) vmblock(O) vmsync(O) vmhgfs(O) ext2 dm_crypt ip6t_LOG xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 ipt_REJECT ipt_LOG xt_limit xt_tcpudp xt_addrtype xt_state ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat ppdev nf_conntrack_ipv4 nf_defrag_ipv4 vmw_balloon nf_conntrack_ftp nf_conntrack psmouse iptable_filter ip_tables serio_raw x_tables parport_pc acpi_memhotplug lp parport vmci(O) vmxnet3 vmw_pvscsi floppy
[152314.059941]
[152314.167091] Pid: 12121, comm: netstat Tainted: G W O 3.2.0-29-virtual #46-Ubuntu VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
[152314.176641] RIP: 0010:[<ffffffff8158f80a>] [<ffffffff8158f80a>] established_get_first+0x4a/0x130
[152314.571870] RSP: 0018:ffff8801604bfdd8 EFLAGS: 00010286
[152314.571874] RAX: ffffc90011625000 RBX: ffff0014ff0a0000 RCX: 000000000007a300
[152314.571876] RDX: 000000000007a300 RSI: ffff88017570b000 RDI: 00000000000003ff
[152314.571878] RBP: ffff8801604bfdf8 R08: 000000000000000a R09: 000000000000ffff
[152314.571881] R10: 0000000000000000 R11: 000000000000000f R12: ffff88016e6c736f
[152314.571883] R13: 0000000000000c91 R14: 0000000000000001 R15: 0000000000000064
[152314.571898] FS: 00007f2a1bc11700(0000) GS:ffff88017fce0000(0000) knlGS:0000000000000000
[152314.571901] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[152314.571903] CR2: 0000000000415f95 CR3: 0000000173f8a000 CR4: 00000000000006e0
[152314.575447] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[152314.575455] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[152314.575460] Process netstat (pid: 12121, threadinfo ffff8801604be000, task ffff88016009db80)
[152314.575469] Stack:
[152314.575707] ffff880010ad7e80 ffff880176706080 ffff880010ad7e80 ffff8801604bfe70
[152314.575717] ffff8801604bfe28 ffffffff8158fbcd ffff880176706080 ffff880162ac4100
[152314.625435] ffff880176706080 ffff880160a56c00 ffff8801604bfea8 ffffffff81197040
[152314.625442] Call Trace:
[152314.689030] [<ffffffff8158fbcd>] tcp_seq_next+0x8d/0xa0
[152314.862618] [<ffffffff81197040>] seq_read+0x280/0x3d0
[152314.865463] [<ffffffff81196dc0>] ? seq_lseek+0x100/0x100
[152315.105409] [<ffffffff811d3b42>] proc_reg_read+0x82/0xc0
[152315.218963] [<ffffffff81174dd0>] vfs_read+0xb0/0x180
[152315.218981] [<ffffffff81174eea>] sys_read+0x4a/0x90
[152315.304275] [<ffffffff8165ad42>] system_call_fastpath+0x16/0x1b
[152315.304287] Code: 98 00 4c 8b 23 c7 43 1c 00 00 00 00 89 d1 72 41 48 63 c1 48 8b 35 cf d7 98 00 8b 3d d5 d7 98 00 48 c1 e0 04 48 03 05 b6 d7 98 00 <f6> 00 01 74 31 f6 40 08 01 74 2b 8d 4a 01 3b 0d b2 d7 98 00 89
[152315.694936] Call Trace:
[152315.703635] [<ffffffff8158fbcd>] tcp_seq_next+0x8d/0xa0
[152315.703644] [<ffffffff81197040>] seq_read+0x280/0x3d0
[152315.703648] [<ffffffff81196dc0>] ? seq_lseek+0x100/0x100
[152315.761426] [<ffffffff811d3b42>] proc_reg_read+0x82/0xc0
[152315.820263] [<ffffffff81174dd0>] vfs_read+0xb0/0x180
[152315.820279] [<ffffffff81174eea>] sys_read+0x4a/0x90
[152315.844874] [<ffffffff8165ad42>] system_call_fastpath+0x16/0x1b
[

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-29-virtual 3.2.0-29.46
ProcVersionSignature: Ubuntu 3.2.0-29.46-virtual 3.2.24
Uname: Linux 3.2.0-29-virtual x86_64
AlsaDevices:
 total 0
 crw-rw---T 1 root audio 116, 1 ago 10 16:01 seq
 crw-rw---T 1 root audio 116, 33 ago 10 16:01 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu12
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Sun Aug 12 13:58:22 2012
HibernationDevice: RESUME=UUID=81e999c9-8d50-47cf-8d54-a25629143f72
InstallationMedia: Ubuntu-Server 10.10 "Maverick Meerkat" - Release amd64 (20101007)
IwConfig: Error: [Errno 2] No such file or directory
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: VMware, Inc. VMware Virtual Platform
PciMultimedia:

ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-29-virtual root=/dev/mapper/h30server2-root ro crashkernel=384M-2G:64M,2G-:128M quiet
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-29-virtual N/A
 linux-backports-modules-3.2.0-29-virtual N/A
 linux-firmware 1.79
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: Upgraded to precise on 2012-05-10 (93 days ago)
dmi.bios.date: 04/15/2011
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd04/15/2011:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.5kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. Please only remove that one tag and leave the other tags. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.6-rc1-quantal/

[Lluise (loris-luise)]

Additional log file /var/log/kern.log
containing more
BUG: soft lockup - CPU ....

[Lluise (loris-luise)]

Testing mainline kernel on a prod machine

Linux h3oserver2 3.5.1-030501-generic #201208091310 SMP

[Lluise (loris-luise)]

oops kernel.log for mainline kernel 3.5.1

[Lluise (loris-luise)]

I made 2 modifications and currently no more soft lockup has happened

1) removed irqbalance daemon from ubuntu server
2) unchecked "Synchronize guest time with host" from VM setting (VMWare esx)

Modification 1) most probably solved the problem.

[tolostoi (tolostoi)]

Same here with esxi 5.1 with virtual and server kernel on guest ubuntu 12.04.1 3.2.0-36 x86_64 I solved (think solved from 40 minutes there is no stucks) vmware tools installed version:
~$ grep buildNr /usr/bin/vmware-config-tools.pl
  my $buildNr;
  $buildNr = '9.0.0 build-782409';
  return remove_whitespaces($buildNr);
Justh on vsphere settings edit the cpu configuration. First was 1 socket with 2 CPU's, now (mean this solve the problem) 2 sockets with 1 CPU per socket.
Hardware where host esxi 5.1 installed is HP Proliant ML350 G5 with 5 GB RAM with just one CPU Intel(R) Xeon(R) CPU 5130 @ 2.00GHz

[penalvch (penalvch)]

Lluise, this bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? If so, could you also please test the latest upstream kernel available via http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.13-rc6-trusty/ and advise to the results?

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]