Running into OpenJDK 11 JIT C2 native memory leak JDK-8292260

[Zahid Bukhari]

We have many apps but I believe one app more or less does more work and it was crashing on OpenJDK 11 on Ubuntu Bionic. We are using version 11.0.16+8-0ubuntu1~18.04.

We have perf reports, Java map files, heaps from jemalloc, Java Native Memory Profiling (i.e. NMP) summary diffs, and SVGs from jemalloc (i.e. jeprof).

We've been able to reliably reproduce this isse. Once we start sending data, the app very quickly consumes all RAM. I would say a gigabyte in about 3.5 seconds. We actually increased RAM to get to the bottom of it. We have several apps not all of them run into the same issue but not all of them work with the data the same way.

NMP summary diff shows Compiler, perf shows C2Compiler, and I'm a system administrator but DevOps and Dev see it as the C2 JIT.

Since there isn't a fix in Ubuntu OpenJDK-LTS 11 we used Adoptium 11.0.16.1 and so far have not been able to replicate the issue.

https://bugs.openjdk.org/browse/JDK-8292260

[Zahid Bukhari (zbukhari-apex)]

1) The release of Ubuntu you are using, via 'lsb_release -rd' or System -> About Ubuntu

Description: Ubuntu 18.04.6 LTS
Release: 18.04

2) The version of the package you are using, via 'apt-cache policy pkgname' or by checking in Software Center

openjdk-11-jdk:
  Installed: 11.0.16+8-0ubuntu1~18.04
  Candidate: 11.0.16+8-0ubuntu1~18.04

3) What you expected to happen

No leak and for Java app to process data with minimal native memory consumption.

4) What happened instead

Native memory leak to the point OOM killer acts on it. Java heap and metaspace memory are within reasonable ranges as we also used VisualVM to analyze it. This is where we had to use jemalloc and perf.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Zahid Bukhari (zbukhari-apex)]

... I just don't want janitor to close this for no activity so ... comment :-)

[Zahid Bukhari (zbukhari-apex)]

Attaching jcmd VM.native_memory summary.diff output which shows the compiler consuming about 41 GB of RAM.

The app has a max heap of 2 GB, we thought it was native but jcmd shows it was the compiler and perf data with a perf map shows it to be the C2 Compiler.

Regardless the kernel sees it as being allocated to the Java app and so eventually the app was killed.

[Yuta Higuchi (y-higuchi)]

11.0.16.1 was skipped, but it looks like 11.0.17, which in theory should have the fix for this C2 issue, is coming to Ubuntu packaged openjdk.

https://code.launchpad.net/ubuntu/+source/openjdk-lts

[Zahid Bukhari (zbukhari-apex)]

Thanks Yuta! That's good to know as we are temporarily leaning on Adoptium OpenJDK but we keep our OS more up to date so I'd rather bring everyone back to Ubuntu OpenJDK.

[Vladimir Petko (vpa1977)]

Closing as Fix Released as 11.0.17 was released.