jmap -heap broken since update to 7u95-2.6.4-0ubuntu0.14.04.1

[Stefan Huehner]

The jmap -heap <pid> command is broken since the update to 7u95-2.6.4-0ubuntu0.14.04.1

Given a tomcat process running with openjdk-7 under a non-root user.

Before that update sudo jmap -heap <pid> gave output about heap configured on top + also the heap usage inside each reason of the java heap (eden, old gen, permgen) etc...
Note: running as root is and was required even before to get the '-heap' option to work. Where as running 'jmap -dump:file' only works when running as user the jvm runs under and not root. That just as context as it is strange behavior of jmap since a long time.

After the update to the specified version only the intro is shown and instead of the usage the following error:

Attaching to process ID 1571, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.95-b01

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 0
   MaxHeapFreeRatio = 100
   MaxHeapSize = 1363148800 (1300.0MB)
   NewSize = 1310720 (1.25MB)
   MaxNewSize = 17592186044415 MB
   OldSize = 5439488 (5.1875MB)
   NewRatio = 2
   SurvivorRatio = 8
   PermSize = 21757952 (20.75MB)
   MaxPermSize = 268435456 (256.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.tools.jmap.JMap.runTool(JMap.java:197)
        at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap
        at sun.jvm.hotspot.tools.HeapSummary.run(HeapSummary.java:146)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
        at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:40)
        ... 6 more

The last version before we noticed that bug was:
7u91-2.6.3-0ubuntu0.14.04.1

Not 100% sure that this exact version still worked but quite sure.

We could reproduce the issue in 2 different servers both running 14.04 LTS 64bit

[Stefan Huehner (stefan-huehner)]

Only public comment i found looking maybe related is old bug here:
https://bugzilla.redhat.com/show_bug.cgi?id=1010786

Talking about missing debug symbols.

I tried blindly installing openjdk-7-dbg package as a quick test, but no change still failed.

[Stefan Huehner (stefan-huehner)]

Confirmed that jmap -heap was still working fine in version:
7u91-2.6.3-0ubuntu0.14.04.1

[Stefan Huehner (stefan-huehner)]

Adding regression-update tag as that regression was introduced to trusty via "updates, security (main)"

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Doussot (philbuntu)]

Confirm missing symbols :

use :
#export LIBSAPROC_DEBUG=1
#jmap -heap 15572
Attaching to process ID 15572, please wait...
....
...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.95-b01
libsaproc DEBUG: lookup failed for symbol '_ZTV20ParallelScavengeHeap' in obj 'libjvm.so'
libsaproc DEBUG: lookup failed for symbol '_ZTV20ParallelScavengeHeap' in obj 'libjvm_g.so'
libsaproc DEBUG: lookup failed for symbol '_ZTV20ParallelScavengeHeap' in obj 'gamma_g'
libsaproc DEBUG: lookup failed for symbol '_ZTV16GenCollectedHeap' in obj 'libjvm.so'
libsaproc DEBUG: lookup failed for symbol '_ZTV16GenCollectedHeap' in obj 'libjvm_g.so'
libsaproc DEBUG: lookup failed for symbol '_ZTV16GenCollectedHeap' in obj 'gamma_g'
libsaproc DEBUG: lookup failed for symbol '_ZTV15G1CollectedHeap' in obj 'libjvm.so'
libsaproc DEBUG: lookup failed for symbol '_ZTV15G1CollectedHeap' in obj 'libjvm_g.so'
libsaproc DEBUG: lookup failed for symbol '_ZTV15G1CollectedHeap' in obj 'gamma_g'

or :

#jmap -dump:format=b,file=here /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java /var/tmp/jvm.core.15572 2>&1 | grep UseSharedSpaces
libsaproc DEBUG: lookup failed for symbol 'UseSharedSpaces' in obj '/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so'
libsaproc DEBUG: can't lookup 'UseSharedSpaces' flag

[Brunet Thibaut (t-brunet)]

Anyone found a workaround ?

Why this bug as no priority and is not assigned ? Nobody needs jmap and heap dump ?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ben Butler-Cole (benbc)]

I also see this problem with openjdk-8-jre 8u111-b14-2ubuntu0.16.04.2.

[Robert Fitzsimons (robfitz273k)]

I found this bug to be related to "Fix stripping packages (use bash instead of expr substring).",
(http://bazaar.launchpad.net/~openjdk/openjdk/openjdk7/revision/600, http://bazaar.launchpad.net/~openjdk/openjdk/openjdk8/revision/661).

On openjdk-7 rebuilding the source packages with DEB_BUILD_OPTIONS='nostrip' I was able to do heapdumps.

On openjdk-8 installing the openjdk-8-dbg package makes the required symbols available, but does require a lot more disk space.

[Andrew Haley (therealaph)]

There's nothing we can do upstream because package managers strip out all of the debug information. It requires

If installing the correct debuginfo package doesn't work, there is a problem finding the debuginfo packages. libjvm will read the .note.gnu.build-id section in libjvm.so, and then look for a corresponding file in /usr/lib/debug.

Like this:

 $ readelf -x .note.gnu.build-id "/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.131.x86_64/jre/lib/amd64/server/libjvm.so"

Hex dump of section '.note.gnu.build-id':
  0x00000190 04000000 14000000 03000000 474e5500 ............GNU.
  0x000001a0 14501d56 be269cfd 9234342b 58407e0f .P.V.&...44+X@~.
  0x000001b0 bbe4f5c8 ....

And the debuginfo should be found here:

/usr/lib/debug/.build-id/14/501d56be269cfd9234342b58407e0fbbe4f5c8.debug

Note that the filename of the debuginfo is the hex representation of the hex data in the .note.gnu.build-id section of libjvm.so.

It would be possible for the distros to avoid this by not stripping out the symbols we need, but most distro packaging has an "all or nothing" approach to debuginfo.

[Matthias Klose (doko)]

Andrew, can jmap work with the detached debug info? Even looking at the redhat report it looks like the issue is not solved this way.

Could you clarify "the symbols we need"? I'm currently stripping with --remove-section=.comment --remove-section=.note --strip-debug, so this might be too much.

[Launchpad Janitor (janitor)]

This bug was fixed in the package openjdk-8 - 8u144-b01-1

---------------
openjdk-8 (8u144-b01-1) unstable; urgency=medium

  * Update to 8u144-b01.
    - fix regression introduced by security fix S8169392. LP: #1707082.

  [ Matthias Klose ]
  * Fix libjvm.so's .debug file names. LP: #1548434.
  * Remove dependency on multiarch-support. Closes: #870520.

  [ Tiago Stürmer Daitx ]
  * debian/apport-hook.py:
    - truncate hs_err if bigger than 100 KiB instead of ignoring it.
    - add message if hs_err file is not found at expected location.
    - report file size in human readble SI units.
  * debian/control.in:
    - move 'Breaks:' from openjdk-8-jdk-headless to openjdk-8-jre-headless.
    - remove jamvm references.
  * debian/control.jamvm-jre: removed.
  * debian/control.jamvm-trans: transactional package for jamvm.
  * debian/rules:
    - add aarch32 hotspot support.
    - build aarch32 using client jvm-variant (no server in aarch32 port).
    - use DEB_HOST_ARCH instead of DEB_HOST_ARCH_CPU as armel and armhf
      are both reported as arm.
    - explicitly add kfreebsd-i386, kfreebsd-amd64, hurd-i386 to arch_map
      and archdir_map due to usage of DEB_HOST_ARCH.
    - avoid building zero as an alternative vm for aarch32.
    - disable precompiled headers on Trusty to minimize g++-4.8 segfaults.
    - don't build zero alternate vm on Trusty, avoid g++-4.8 segfaults.
    - add a 'Breaks:' entry to ca-certificates-java for all releases
      except Trusty. LP: #1706567.
    - remove jamvm.
  * debian/patches/aarch64.diff: remove unnecessary chunks as aarch64 is
    now upstream.
  * debian/patches/aarch32.diff: add required changes to root and jdk to
    build aarch32.
  * debian/patches/hotspot-libpath-aarch32.diff: copied from
    hotspot-libpath-default.diff.
  * debian/patches/ppc64le-8036767.diff: updated.
  * debian/patches/jdk-ppc64el-S8170153.patch: updated to include aarch64.
  * debian/patches/jdk-java-nio-bits-unligned-aarch64.diff: Check for
    "aarch64" along with other unaligned access supporting architectures.

 -- Matthias Klose <email address hidden> Wed, 23 Aug 2017 21:41:07 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package openjdk-7 - 7u151-2.6.11-2ubuntu0.14.04.1

---------------
openjdk-7 (7u151-2.6.11-2ubuntu0.14.04.1) trusty-security; urgency=medium

  * Backport to 14.04.
  * debian/patches/hotspot-aarch64-S8145438-fix-field-too-big-for-insn.patch:
    the S8144028 fix was incomplete and followed up by S8145438; without it
    aarch64 JVM can fail with "Internal Error, failed: Field too big for
    insn".

 -- Tiago Stürmer Daitx <email address hidden> Tue, 21 Nov 2017 02:10:21 +0000