libjawt.so inconsistency lets JVM crash

[Reguel Wermelinger]

When running an SWT ui with an embedded AWT/Swing component the library loading process fails as soon as the user clicks on a JComboBox.

The error visible is: Inconsistency detected by ld.so: dl-lookup.c: 111: check_match: Assertion `version->filename == NULL || ! _dl_name_match_p (version->filename, map)' failed!

############
java --version
openjdk 11.0.4 2019-07-16
OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
OpenJDK 64-Bit Server VM (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3, mixed mode, sharing)

############
running the JVM with LD_DEBUG=all exposes the /lib/libjawt.so as the source of the problem.

      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/bin/java [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/libjli.so [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libz.so.1 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libgcc_s.so.1 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libawt_xawt.so [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libawt.so [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libXext.so.6 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libX11.so.6 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libXrender.so.1 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libXtst.so.6 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libXi.so.6 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libjava.so [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libxcb.so.1 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libverify.so [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libXau.so.6 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libXdmcp.so.6 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libbsd.so.0 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/librt.so.1 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libXinerama.so.1 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libselinux.so.1 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libresolv.so.2 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libmount.so.1 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/x86_64-linux-gnu/libffi.so.6 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libpcre.so.3 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libblkid.so.1 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/lib/x86_64-linux-gnu/libuuid.so.1 [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/home/rew/.swt/lib/linux/x86_64/libswt-awt-gtk-4926r21.so [0]
      3248: symbol=JAWT_GetAWT; lookup in file=/usr/lib/jvm/java-11-openjdk-amd64/lib/libjawt.so [0]
Inconsistency detected by ld.so: dl-lookup.c: 111: check_match: Assertion `version->filename == NULL || ! _dl_name_match_p (version->filename, map)' failed!

############
A quick smoke test revealed that the package from adoptOpenJDK does not have the issue. I can also use the libjawt.so from adoptOpenJDK and run my SWT/AWT ui without any issues.

[Reguel Wermelinger (ivy-rew)]

run attached file with java -jar awtBug3.jar to see the issue in action: as soon the you click on the JComboBox expansion toggle, the VM will crash.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Vlad (vlad2017)]

My application gets the same error with openjdk 11.0.3+5-1ubuntu2 and 11.0.4+11-1ubuntu2~19.04, but it works fine with AdoptOpenJDK 11.0.4+11.

[Tiago Stürmer Daitx (tdaitx)]

Also reproducible with OpenJDK 11.0.5+10-2ubuntu1 in Focal.

[Marc Streckfuß (mefisto94)]

This issue definitely seems to be certainly triggered by some of the Ubuntu patches as AdoptOpenJDK and "upstream" (OpenJDK built by Redhat) work.

The case I am affected is by being unable to load "lwjgl2" with KUbuntu 18.04 or Ubuntu 19.10:

Inconsistency detected by ld.so: dl-lookup.c: 111: check_match:
Assertion `version->filename == NULL || ! _dl_name_match_p
(version->filename, map)' failed!

symbol=Java_org_lwjgl_opengl_LinuxDisplay_nLockAWT; lookup in file=/lib/x86_64-linux-gnu/librt.so.1 [0]
usr/lib/jvm/java-11-openjdk-amd64/lib/libjawt.so: error: symbol lookup error: undefined symbol: Java_org_lwjgl_opengl_LinuxDisplay_nLockAWT (fatal)
symbol=Java_org_lwjgl_opengl_LinuxDisplay_nLockAWT; lookup in file=/home/marc/Dokumente/02_GIT/sdk/build/testuserdir/liblwjgl64.so [0]
binding file /home/marc/Dokumente/02_GIT/sdk/build/testuserdir/liblwjgl64.so [0] to /home/marc/Dokumente/02_GIT/sdk/build/testuserdir/liblwjgl64.so [0]: normal symbol `Java_org_lwjgl_opengl_LinuxDisplay_nLockAWT'

Now the openjdk-dev mailing list traces this down to this glibc/gcc bug: https://bugs.launchpad.net/ubuntu/+source/gcc-7/+bug/1764701

A note to the above output: For every native symbol, LD tries to look into a few and fails:
/usr/lib/jvm/java-11-openjdk-amd64/lib/libjawt.so: error: symbol lookup error: undefined symbol: Java_org_lwjgl_DefaultSysImplementation_setDebug (fatal)

/usr/lib/jvm/java-11-openjdk-amd64/lib/libnio.so: error: symbol lookup error: undefined symbol: Java_sun_awt_image_BytePackedRaster_initIDs (fatal)

/usr/lib/jvm/java-11-openjdk-amd64/lib/libjava.so: error: symbol lookup error: undefined symbol: Java_sun_awt_image_BytePackedRaster_initIDs (fatal)

/usr/lib/jvm/java-11-openjdk-amd64/lib/libsplashscreen.so: error: symbol lookup error: undefined symbol: Java_sun_awt_image_BytePackedRaster_initIDs (fatal)

I don't know the inner workings, but in general it seems to only produce this error on .so's from the JDK, looking up in, say, libstdc++.so.6 works. It's also confusing that libjava.so is reported as faulty for one symbol but not for another.

Actually, even this:
/usr/lib/jvm/java-11-openjdk-amd64/bin/java: error: symbol lookup error: undefined symbol: JNI_OnLoad_awt (fatal)

Is there an easy way I could do a binary diff? Maybe comparing the symbols with objdump as well?
Thanks in Advance

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Marc Streckfuß (mefisto94)]

Sorry for reposting, but this is important:
The difference seems to be with how symbols are exported.

"Broken" Ubuntu libjawt.so:
$ objdump -tT /usr/lib/jvm/java-11-openjdk-amd64/lib/libjava.so

/usr/lib/jvm/java-11-openjdk-amd64/lib/libjava.so: Dateiformat elf64-x86-64

SYMBOL TABLE:
Keine Symbole

DYNAMIC SYMBOL TABLE:
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_GetClassSignature
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_AssertionStatusDirectives
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 getenv
0000000000000000 DO *UND* 0000000000000000 GLIBC_2.2.5 __timezone
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_RegisterSignal
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.3.4 __snprintf_chk
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_ConstantPoolGetFieldAt
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 free
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_ConstantPoolGetDoubleAt
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_ConstantPoolGetLongAt
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_StartThread
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_GetInterfaceVersion
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_InitProperties
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __errno_location
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_InitStackTraceElement
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_GetClassModifiers
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strncpy
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 remove
0000000000000000 w D *UND* 0000000000000000 _ITM_deregisterTMCloneTable
0000000000000000 DO *UND* 0000000000000000 GLIBC_2.2.5 stdout
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_InitStackTraceElementArray
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 _exit
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 strcpy
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_SetClassSigners
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 mkdir
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_AddReadsModule
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_GetMethodParameters
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_Sleep
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_ConstantPoolGetMemberRefInfoAt
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 isatty
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 sigaction
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 fread
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_DoPrivileged
0000000000000000 DF *UND* 0000000000000000 SUNWprivate_1.1 JVM_ConstantPoolGetNameAndTypeRefIndexAt
0000000000000000 DO *UND* 0000000000000000 GLIBC_2.2.5 stdin
...

[Marc Streckfuß (mefisto94)]

Wildly guessing based on the date of the report and the fact that some claim downgrading to 11.0.4 works fine, I conclude that https://git.launchpad.net/ubuntu/+source/openjdk-lts/commit/?id=cc446b0311221da773095bdc25af62bb51c7ece0 broke it.

It could very well be a regression from using GCC 9, which seems to be the case since that commit

[Marc Streckfuß (mefisto94)]

Just using GCC8 didn't solve the problem, but after looking at the assertion again, I've come up with what I think is the solution, I just need to look at debian/control and find out how to fix it.

So the static symbol table is not of interest and the dynamic symbol table is intact, but: the version symbol tables are stripped just like the static symbol table is.
I guess this is some kind of compression/shrinking optimization, but LWJGL2 has some verneed entries:

Version needs section '.gnu.version_r' contains 3 entries:
 Adr: 0x000000000003b7a0 Offset: 0x03b7a0 Link: 5 (.dynstr)
  000000: Version: 1 Datei: libjawt.so Zähler: 1
  0x0010: Name: SUNWprivate_1.1 Flags: nichts Version: 5
  0x0020: Version: 1 Datei: libm.so.6 Zähler: 1
  0x0030: Name: GLIBC_2.2.5 Flags: nichts Version: 3
  0x0040: Version: 1 Datei: libc.so.6 Zähler: 3
  0x0050: Name: GLIBC_2.4 Flags: nichts Version: 6
  0x0060: Name: GLIBC_2.3.4 Flags: nichts Version: 4
  0x0070: Name: GLIBC_2.2.5 Flags: nichts Version: 2

Here it depends on libjawt.so which does not provide a versym table and thus LD fails at resolving the method.

Also the "working" libjawt.so from AdoptOpenJDK has 26 tables, where this one has 22 tables, specifically missing .gnu.version.

This is only half the puzzle, though, since neither AdoptOpenJDK nor RedHat's build expose the SUNWprivate_1.1 like libjawt.so from openjdk8 did:

Version symbols section '.gnu.version' contains 12 entries:
 Adr: 00000000000004b6 Offset: 0x0004b6 Link: 4 (.dynsym)
  000: 0 (*lokal*) 3 (SUNWprivate_1.1) 0 (*lokal*) 3 (SUNWprivate_1.1)
  004: 3 (SUNWprivate_1.1) 0 (*lokal*) 3 (SUNWprivate_1.1) 0 (*lokal*)
  008: 3 (SUNWprivate_1.1) 0 (*lokal*) 2 (SUNWprivate_1.1) 2 (SUNWprivate_1.1)

Version definition section '.gnu.version_d' contains 2 entries:
  Adr: 0x00000000000004d0 Offset: 0x0004d0 Link: 5 (.dynstr)
  000000: Rev: 1 Flags: BASE Index: 1 Zähler: 1 Name: libjawt.so
  0x001c: Rev: 1 Flags: nichts Index: 2 Zähler: 1 Name: SUNWprivate_1.1

.dynsym:
    10: 0000000000000760 110 FUNC GLOBAL DEFAULT 13 JAWT_GetAWT@@SUNWprivate_1.1

So I guess the versioning feature has been removed, but the deletion of the versioning tables is something that ld cannot handle gracefully. I don't know if this is a bug in ld/glibc or just a faulty behavior from Ubuntu's Packaging.

[Marc Streckfuß (mefisto94)]

So I managed to pinpoint this issue even further:
When using pbuilder-dist to build for disco and buster (debian usually natively builds a bit different), I can reproduce the issue only on disco.

This means, the build environment must be the problem. A shot in the dark is the custom gcc fork of ubuntu, as both are on version 8.3.0-6.

Below I've attached the diff between the .buildinfo files. Note that the first/left is buster and the right is ubuntu.
Also note that the .dsc file differs, as I needed to s/g++-9/g++-8 so that pbuilder doesn't attempt to download g++9 on buster (which it wouldn't use anyway, as rules filters for buster and uses g++8)

[Antoine Le Gonidec (vv222)]

A similar error has been investigated in the context of ./play.it development: https://forge.dotslashplay.it/play.it/games/-/issues/880

A possible workaround would be to build OpenJDK with the --no-as-needed ld flag.

[Vladimir Petko (vpa1977)]

Fixed in Debian and fix released as a part of July Security update.

[Vladimir Petko (vpa1977)]

Marking bionic Invalid as it is EOL