JRE Crashes in JDK 11.0.19+7

[Aruna]

After deploying the application to EC2 instance, we are facing repeated 502 bad gateway error with below core dump created:
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (sharedRuntime.cpp:1261), pid=6746, tid=6910
# guarantee((retry_count++ < 100)) failed: Could not resolve to latest version of redefined method
#
# JRE version: OpenJDK Runtime Environment (11.0.19+7) (build 11.0.19+7-post-Ubuntu-0ubuntu120.04.1)
# Java VM: OpenJDK 64-Bit Server VM (11.0.19+7-post-Ubuntu-0ubuntu120.04.1, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/ubuntu/projects/frontend/code/core.6746)
#
# If you would like to submit a bug report, please visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-lts

PLEASE NOTE: THE above errors don't show up in local testing.

Please help resolve ASAP

[Vladimir Petko (vpa1977)]

Hi, thank you for reporting the bug!!!

The crash happens due to the failing assertion - a method that is being called by the virtual machine keeps getting redefined. It may be happening due to multiple reasons including a bug in the Java VM or just insufficient RAM.

You may want to try to tweak –XX:ProfiledCodeHeapSize/-XX:NonProfiledCodeHeapSize[1], but this is a wild guess as the bug report does not contain sufficient information. Another wild guess - it could be related to the crash bug[2] that is getting multiple reports currently.

Would it be possible to provide a core dump or hs_err.pid file or ideally a small sample that reproduces the crash?

[1] https://docs.oracle.com/en/java/javase/11/vm/java-hotspot-virtual-machine-performance-enhancements.html#GUID-1D9B26AD-8E0A-4771-90DA-A81A2C1F5B55
[2] https://bugs.openjdk.org/browse/JDK-8307683

[Aruna (aruna-geek-2023)]

Please fund core dump logs;

--------------- T H R E A D ---------------

Current thread (0x00007fb1fc53a000): JavaThread "http-nio-80-exec-16" daemon [_thread_in_vm, id=5867, stack(0x00007fb1f6c7f000,0x00007fb1f6d80000)]

Stack: [0x00007fb1f6c7f000,0x00007fb1f6d80000], sp=0x00007fb1f6d78920, free space=998k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xe9e869] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x199 V [libjvm.so+0xe9f7a5] VMError::report_and_die(Thread*, void*, char const*, int, char const*, char const*, __va_list_tag*)+0x35 V [libjvm.so+0x67ca9b] report_vm_error(char const*, int, char const*, char const*, ...)+0xfb V [libjvm.so+0xcb77f9] SharedRuntime::resolve_helper(JavaThread*, bool, bool, Thread*)+0x109 V [libjvm.so+0xcb7b25] SharedRuntime::resolve_virtual_call_C(JavaThread*)+0x45
v ~RuntimeStub::resolve_virtual_call
J 24701 c1 org.springframework.beans.CachedIntrospectionResults.clearClassLoader(Ljava/lang/ClassLoader;)V (56 bytes) @ 0x00007fb23e271d1c [0x00007fb23e270bc0+0x000000000000115c]
J 23512 c1 org.springframework.context.support.AbstractApplicationContext.refresh()V (190 bytes) @ 0x00007fb23f755774 [0x00007fb23f754500+0x0000000000001274]
J 23525 c1 org.springframework.context.support.ClassPathXmlApplicationContext.<init>(Ljava/lang/String;)V (15 bytes) @ 0x00007fb23f7614e4 [0x00007fb23f760ec0+0x0000000000000624]

[Vladimir Petko (vpa1977)]

Hi, this is the stack of the thread that fails the assertion. We can try to guess that it indeed may have something to do with the JDK-8307683 (branch prediction bug) since we are looking at conditional loops:
------------------cut----------------------
/**
  * Clear the introspection cache for the given ClassLoader, removing the
  * introspection results for all classes underneath that ClassLoader, and
  * removing the ClassLoader (and its children) from the acceptance list.
  * @param classLoader the ClassLoader to clear the cache for
  */
 public static void clearClassLoader(@Nullable ClassLoader classLoader) {
  acceptedClassLoaders.removeIf(registeredLoader ->
    isUnderneathClassLoader(registeredLoader, classLoader));
  strongClassCache.keySet().removeIf(beanClass ->
    isUnderneathClassLoader(beanClass.getClassLoader(), classLoader));
  softClassCache.keySet().removeIf(beanClass ->
    isUnderneathClassLoader(beanClass.getClassLoader(), classLoader));
 }
--------------------cut-------------------------

I have uploaded openjdk with the patch[1] applied to the ppa[2].

Could you please try to reproduce the issue with openjdk-lts 11.0.19+7~us1-0ubuntu2~ppa1?

In order to install it run
-----cut--------------
sudo apt-get install -y software-properties-common
sudo add-apt-repository ppa:vpa1977/openjdk-11-april
sudo apt update
sudo apt-get install openjdk-11-jdk=11.0.19+7~us1-0ubuntu2~ppa
---------------cut-------------
[1] https://github.com/openjdk/jdk11u/commit/1000b1c0074af4b954c5f210d1684b5623e9723b
[2] https://launchpad.net/~vpa1977/+archive/ubuntu/openjdk-11-april

[Aruna (aruna-geek-2023)]

Hi,
while running the command sudo add-apt-repository ppa:openjdk-r/ppa, getting below error:

err1.PNG

While running the command sudo apt-get install openjdk-11-jdk=11.0.19+7~us1-0ubuntu2~ppa, got below error
err2.png

see attachment of the error screenshot.

We are currently using ubuntu version 20.04
Please help resolve.

[Vladimir Petko (vpa1977)]

The ppa only had 23.10 version uploaded. I apologise for confusion it may have caused. The version for 23.10 can be installed with
-----------cut------------
 sudo apt install openjdk-11-jdk=11.0.19+7~us1-0ubuntu2~ppa1
----------cut-------------
please notice '1' at the end of the version, it was missing from the original paste.

I have added 20.04 (focal) version there.

please run
-----------cut------------
sudo add-apt-repository ppa:vpa1977/openjdk-11-april
-----------cut------------
rather than ppa:openjdk-r/ppa

In order to install 20.04 version after adding ppa
----cut----------
sudo apt-get install openjdk-11-jdk=11.0.19+7~us1-0ubuntu~20.04~ppal1
----cut-------

[Vladimir Petko (vpa1977)]

@Aruna, I apologise for asking, but did using ppa version resolve the problem or it had no effect?

[Aruna (aruna-geek-2023)]

Hi Vladimir,
After running the command sudo apt install openjdk-11-jdk=11.0.19+7~us1-0ubuntu2~ppa1, we get below error:

Pls find attached the png image.

WE are planning to move to JDK 13. Does this version JDK 13 have the fix you are aiming for?
Please help resolve.

[Vladimir Petko (vpa1977)]

Hi

to install patched openjdk for 20.04 Focal please run following commands
------------cut-----------------
sudo add-apt-repository ppa:vpa1977/openjdk-11-april
sudo apt-get install openjdk-11-jdk=11.0.19+7~us1-0ubuntu~20.04~ppal1
-----------cut------------------

openjdk-11-jdk=11.0.19+7~us1-0ubuntu2~ppa1 is a version for Lunar (latest release).

I believe OpenJDK 13 might not have this bug present, but it is also out of support and will contain other issues including security vulnerabilities.

[Vladimir Petko (vpa1977)]

Hi,

Is the issue still present with 11.0.20+8 [1] ?

[1] https://launchpad.net/ubuntu/+source/openjdk-lts/11.0.20+8-1ubuntu1~20.04

[Vladimir Petko (vpa1977)]

Marking invalid as the issue can not be reproduced.