Hardy sun-java 5 and 6 SIGSEGV during GC

[drplix]

I have several Hardy guests running under KVM on a Jaunty host. All fully patched. Since this recent update "updating upgrade sun-java6-bin 6-16-0ubuntu1.8.04 6-17-0ubuntu1.8.04" we are seeing persistent Java instability on application servers. Crash with SIGSEGV - looking at the hs_err_log files it is happening during Garbage Collection (GC).

Sometimes the crash happens within 20s of booting the JVM othertimes it can be stable for days.

I will attach a typical copy of the Hotspot dump below. Here's a summary...

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f16e533235c, pid=4111, tid=1104398672
#
# JRE version: 6.0_17-b04
# Java VM: Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode linux-amd64 )
# Problematic frame:
# V [libjvm.so+0x56135c]

VM_Operation (0x0000000040982620): GenCollectFull, mode: safepoint, requested by thread 0x000000004031a800

--------------- S Y S T E M ---------------

OS:lenny/sid

uname:Linux 2.6.24-24-server #1 SMP Fri Sep 18 16:47:05 UTC 2009 x86_64
libc:glibc 2.7 NPTL 2.7
rlimit: STACK 8192k, CORE 0k, NPROC 2047, NOFILE 1024, AS infinity
load average:0.00 0.00 0.00

CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 2 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3

Memory: 4k page, physical 252376k(3140k free), swap 124984k(117308k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (14.3-b01) for linux-amd64 JRE (1.6.0_17-b04), built on Oct 11 2009 01:08:48 by "java_re" with gcc 3.2.2 (SuSE Linux)

time: Wed Jan 27 20:18:14 2010
elapsed time: 2880 seconds

[drplix (pjr-1060)]

I should also say. We have tried the latest 1.6.0_18 JRE from sun and see the same crashes. Also we've tried switching to java-5 64bit and also see the crashes. This suggests it could be something in the low level native libs called by libjvm.so.

Since these are production critical machines I'm trying a temporary switch to the ia32 bit JVM packages like this...

sudo apt-get install ia32-sun-java6-bin
sudo update-alternatives --config java and select ia32 java

Will report if this is a good long term workaround - so far so good.

[drplix (pjr-1060)]

Switching to 32bit Java didn't help. We have ended up doing a host upgrade to Karmic - turns out that all this involved was switching to a new kernel and was painless. Now running host on ...

Linux version 2.6.28-13-server (buildd@yellow) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #45-Ubuntu SMP Tue Jun 30 22:56:18 UTC 2009

Hardy virtual guests are unchanged and fully up to date. Stability has improved considerably. I can only guess that somehow the JVM GC code was triggering something unstable in the kernel KVM memory virtualization?