JVM crash with SIGSEGV as tomcat start with 7u181

[Stefan Huehner]

Hello,
with latest openjdk update
# JRE version: OpenJDK Runtime Environment (7.0_181-b01) (build 1.7.0_181-b01)

we are seeing quite frequent crash on starting tomcat with our web application (openbravo).

Checking 2 of the created hs_err_pid log they seem to point to same stacktrace for the crash:
Excerpt:
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x418d57]
V [libjvm.so+0x647253] JVM_FindClassFromClass+0x283
C [libverify.so+0x4a32]
C [libverify.so+0x5a99]
C [libverify.so+0x99fd]
C [libverify.so+0xb742] VerifyClassForMajorVersion+0x15b2
V [libjvm.so+0x94a7cf]
V [libjvm.so+0x95961c]
V [libjvm.so+0x5b9905]
V [libjvm.so+0x5bb243]
V [libjvm.so+0x5bb9e1]
V [libjvm.so+0x5eea00]
j org.hibernate.type.TypeResolver.<init>()V+8
j org.hibernate.cfg.Configuration.<init>(Lorg/hibernate/cfg/SettingsFactory;)V+9
j org.hibernate.cfg.Configuration.<init>()V+8
j org.openbravo.base.session.SessionFactoryController.initialize()V+21
j org.openbravo.base.session.SessionFactoryController.getSessionFactory()Lorg/hibernate/SessionFactory;+1
j org.openbravo.base.model.ModelProvider.initialize()V+23

2* Full hs_errr_pid files will be attached to this issue.

System running is:
- Ubuntu 14.04 server in 64bit hvm Amazon instance.
- OpenJdk package version is:
  # Distribution: Ubuntu 14.04 LTS, package 7u181-2.6.14-0ubuntu0.1

[Tiago Stürmer Daitx (tdaitx)]

I installed tomcat7 and tested a couple of very simple hibernate samples from around the web, none were able to trigger this issue.

Please see if you can find a simpler reproducer and update the bug with the required steps to reproduce it. I'm setting it as incomplete for now.

[Stefan Huehner (stefan-huehner)]

Hell Tiago,
on your recommedation on irc i also reported that upstream to icedtea here:
https://icedtea.classpath.org/bugzilla/show_bug.cgi?id=3604

[Tiago Stürmer Daitx (tdaitx)]

Attaching the bt full backtrace provided by the Huehner to the IcedTea bug.

It indicates that the bug is triggered in the file hotspot/src/share/vm/classfile/classLoaderDependencies.cpp:43 which is exclusive to jdk7u [1] and was introduced by the backport of "8189123: More consistent classloading" [2] to jdk7u [3].

[1] http://hg.openjdk.java.net/jdk7u/jdk7u/hotspot/file/327ea6f9647c/src/share/vm/classfile/classLoaderDependencies.cpp
[2] http://hg.openjdk.java.net/jdk/jdk/rev/0a8a11767c3d
[3] http://hg.openjdk.java.net/jdk7u/jdk7u/hotspot/rev/594f46270048

[Andrew John Hughes (ahughes)]

Thanks Tiago. The fix for 8189123 in the last security update had to be pretty much re-written for OpenJDK 7 due to the removal of permgen in 8 and up. I'll point the developer of the patch at this bug and hopefully we can get a resolution to this.

[GGrandes (ggrandes)]

We have this problem too, tomcat 7 and production. You know any workaround?

[Stefan Huehner (stefan-huehner)]

Hello Guillermo,

i don't know of any workaround. In our case it only triggered on application start and not 100% of the time -> so just retrying a few times.

Apart there is an developer from RedHat already working on debugging & fixing the problem.

I asked him to share an update here when he has new information.

[Martin Balao (mbalao)]

We've analyzed this bug and have a good understanding of what's going on. A fix will be ready soon.

[GGrandes (ggrandes)]

Thanks, in case it gives any other clue, the breaks do not happen every time, but it seems to match a FullGC.

2018-07-10T12:49:13.290+0200: 12.925: [Full GC2018-07-10T12:49:13.290+0200: 12.925: [Tenured: 29930K->28388K(40012K), 0.1007850 secs] 42589K->28388K(58060K), [Perm : 26687K->26679K(26688K)], 0.1009500 secs] [Times: user=0.09 sys=0.00, real=0.10 secs]
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fc964265d57, pid=17770, tid=140502113179392
#
# JRE version: OpenJDK Runtime Environment (7.0_181-b01) (build 1.7.0_181-b01)
# Java VM: OpenJDK 64-Bit Server VM (24.181-b01 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 2.6.14
# Distribution: Ubuntu 14.04 LTS, package 7u181-2.6.14-0ubuntu0.1
# Problematic frame:
# V [libjvm.so+0x418d57]

[Martin Balao (mbalao)]

OpenJDK upstream bug report: https://bugs.openjdk.java.net/browse/JDK-8207151
Fix proposal: http://mail.openjdk.java.net/pipermail/jdk7u-dev/2018-July/010787.html

[RAG (rgollub)]

In an attempt to contribute to similar cases and show interest in the eventual fix, herewith attached goes the log of the "segfault" in my system.

Although the java VM is launched by Eclipse, the fault lies clearly with jvm.

I also confirm that the fault occurs randomly while starting Eclipse. I also note that the fault started to occur only after recently having updated to this offending version of the java "environment".

Available for additional related data, if necessary, of course.

[Jean-Sebastien Bevilacqua (realitix)]

Hello,
The patch has been accepted upstream (http://icedtea.classpath.org//hg/release/icedtea7-forest-2.6/hotspot?cmd=changeset;node=e012acbfcd40)

It's a critical bug, when will this fix be available in ubuntu 14.04 ?

Thanks,
Jean-Sébastien

[Gianfranco Costamagna (costamagnagianfranco)]

please test the patch in https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/locutusofborg-ppa/+packages

and somebody do the paperwork mentioned here
https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

Since this is a regression in a security upload, this should probably be handled by security team.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openjdk-7 - 7u181-2.6.14-0ubuntu0.2

---------------
openjdk-7 (7u181-2.6.14-0ubuntu0.2) trusty-security; urgency=medium

  * Apply 8u181 security backports.
  * Security fixes:
    - CVE-2018-2938, S8197871: Support Derby connections.
    - CVE-2018-2952, S8199547: Exception to Pattern Syntax.
    - S8191239: Improve desktop file usage.
    - S8193419: Better Internet address support.
    - S8197925: Better stack walking.
    - S8200666: Improve LDAP support.
  * debian/patches/hotspot-S8207151-fix-bad-klassoop.patch: fix bug introduced
    by the backport of S8189123. LP: #1778930.

 -- Tiago Stürmer Daitx <email address hidden> Mon, 23 Jul 2018 20:03:03 +0000