QEMU

Openjdk11+ fails to install on s390x

Bug #1920913 reported by Namrata Bhave on 2021-03-23

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	QEMU	Expired	Undecided	Unassigned

Bug Description

While installing openjdk11 or higher from repo, it crashes while configuring ca-certificates-java.
Although `java -version` passes, `jar -version` crashes. Detailed logs attached to this issue.

```
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x00000040126f9980, pid=8425, tid=8430
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 4 c1 java.lang.StringLatin1.hashCode([B)I java.base@11.0.10 (42 bytes) @ 0x00000040126f9980 [0x00000040126f9980+0x0000000000000000]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.8425)
#
# An error report file with more information is saved as:
# //hs_err_pid8425.log
sed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /root/core.10740)
#
# An error report file with more information is saved as:
# /root/hs_err_pid10740.log
```

Observed this on s390x/ubuntu as well as s390x/alpine when run on amd64 host.
Please note, on native s390x, the installation is successful. Also this crash is not observed while installing openjdk-8-jdk.

Qemu version: 5.2.0

Please let me know if any more details are needed.

See original description

Tags:

Revision history for this message

Namrata Bhave (nam121) wrote on 2021-03-23:

openjdk_crash_logs.log Edit (13.0 KiB, text/plain)

description:

updated

Revision history for this message

Peter Maydell (pmaydell) wrote on 2021-03-23:

You don't say how you're invoking QEMU (system emulation? usermode? what command line?) Please give the full commandline, repro steps, and any files/images we would need to reproduce the failure.

Revision history for this message

Namrata Bhave (nam121) wrote on 2021-03-23:

Please find below steps to reproduce the issue(Running on amd64 VM):

```
apt-get install -y qemu qemu-user-static
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker run -it s390x/ubuntu:20.04 bash
--> apt-get update && apt-get install -y openjdk-11-jdk
jar --version
```

Revision history for this message

David Hildenbrand (davidhildenbrand) wrote on 2021-03-30:

Same BUG as https://bugs.launchpad.net/qemu/+bug/1862874

Revision history for this message

Namrata Bhave (nam121) wrote on 2021-03-30:

Tried building jdk 11 from source, the generated executable still crashes(fastdebug as well as release mode):

```
root@24d396a17e00:~/jdk# build/linux-s390x-normal-server-release/jdk/bin/java -version
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x000000400b234440, pid=18175, tid=18178
#
# JRE version: OpenJDK Runtime Environment (11.0) (build 11-internal+0-adhoc..jdk)
# Java VM: OpenJDK 64-Bit Server VM (11-internal+0-adhoc..jdk, mixed mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 78 c1 java.util.HashMap.afterNodeInsertion(Z)V java.base (1 bytes) @ 0x000000400b234440 [0x000000400b234400+0x0000000000000040]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /root/jdk/core.18175)
#
# An error report file with more information is saved as:
# /root/jdk/hs_err_pid18175.log
Compiled method (c1) 1795 78 3 java.util.HashMap::afterNodeInsertion (1 bytes)
total in heap [0x000000400b234210,0x000000400b2345b0] = 928
relocation [0x000000400b234378,0x000000400b2343a0] = 40
constants [0x000000400b2343c0,0x000000400b234400] = 64
main code [0x000000400b234400,0x000000400b234500] = 256
stub code [0x000000400b234500,0x000000400b234558] = 88
metadata [0x000000400b234558,0x000000400b234568] = 16
scopes data [0x000000400b234568,0x000000400b234578] = 16
scopes pcs [0x000000400b234578,0x000000400b2345a8] = 48
dependencies [0x000000400b2345a8,0x000000400b2345b0] = 8
Compiled method (c1) 1806 74 3 java.util.HashMap::putVal (300 bytes)
total in heap [0x000000400b230210,0x000000400b231f20] = 7440
relocation [0x000000400b230378,0x000000400b230690] = 792
constants [0x000000400b2306c0,0x000000400b230a00] = 832
main code [0x000000400b230a00,0x000000400b231980] = 3968
stub code [0x000000400b231980,0x000000400b231a68] = 232
metadata [0x000000400b231a68,0x000000400b231ad0] = 104
scopes data [0x000000400b231ad0,0x000000400b231ce8] = 536
scopes pcs [0x000000400b231ce8,0x000000400b231eb8] = 464
dependencies [0x000000400b231eb8,0x000000400b231ec0] = 8
nul chk table [0x000000400b231ec0,0x000000400b231f20] = 96
Could not load hsdis-s390x.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)
root@24d396a17e00:~/jdk#
```

Tried building jdk 11 from source, the generated executable still crashes(fastdebug as well as release mode):

```
root@24d396a17e00:~/jdk# build/linux-s390x-normal-server-release/jdk/bin/java -version
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x000000400b234440, pid=18175, tid=18178
#
# JRE version: OpenJDK Runtime Environment (11.0) (build 11-internal+0-adhoc..jdk)
# Java VM: OpenJDK 64-Bit Server VM (11-internal+0-adhoc..jdk, mixed mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 78 c1 java.util.HashMap.afterNodeInsertion(Z)V java.base (1 bytes) @ 0x000000400b234440 [0x000000400b234400+0x0000000000000040]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /root/jdk/core.18175)
#
# An error report file with more information is saved as:
# /root/jdk/hs_err_pid18175.log
Compiled method (c1)    1795   78       3       java.util.HashMap::afterNodeInsertion (1 bytes)
 total in heap  [0x000000400b234210,0x000000400b2345b0] = 928
 relocation     [0x000000400b234378,0x000000400b2343a0] = 40
 constants      [0x000000400b2343c0,0x000000400b234400] = 64
 main code      [0x000000400b234400,0x000000400b234500] = 256
 stub code      [0x000000400b234500,0x000000400b234558] = 88
 metadata       [0x000000400b234558,0x000000400b234568] = 16
 scopes data    [0x000000400b234568,0x000000400b234578] = 16
 scopes pcs     [0x000000400b234578,0x000000400b2345a8] = 48
 dependencies   [0x000000400b2345a8,0x000000400b2345b0] = 8
Compiled method (c1)    1806   74       3       java.util.HashMap::putVal (300 bytes)
 total in heap  [0x000000400b230210,0x000000400b231f20] = 7440
 relocation     [0x000000400b230378,0x000000400b230690] = 792
 constants      [0x000000400b2306c0,0x000000400b230a00] = 832
 main code      [0x000000400b230a00,0x000000400b231980] = 3968
 stub code      [0x000000400b231980,0x000000400b231a68] = 232
 metadata       [0x000000400b231a68,0x000000400b231ad0] = 104
 scopes data    [0x000000400b231ad0,0x000000400b231ce8] = 536
 scopes pcs     [0x000000400b231ce8,0x000000400b231eb8] = 464
 dependencies   [0x000000400b231eb8,0x000000400b231ec0] = 8
 nul chk table  [0x000000400b231ec0,0x000000400b231f20] = 96
Could not load hsdis-s390x.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Aborted (core dumped)
root@24d396a17e00:~/jdk#
```

Revision history for this message

Namrata Bhave (nam121) wrote on 2021-04-08:

Download full text (3.5 KiB)

@davidhildenbrand The other issue which you have mentioned as duplicate shows java getting stuck for long, whereas for me it crashes right away. Do you think these 2 are related?

Also observed another behaviour :
java -version randomly passes, sometimes.

I can also confirm that it is observed under s390x chroot as well(logs below):
```
root@XX:/# ulimit -c unlimited
root@XX:/# java -version
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode)
root@XX:/# java -version
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x0000004012705b40, pid=156601, tid=156604
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 5 c1 java.lang.Object.<init>()V java.base@11.0.10 (1 bytes) @ 0x0000004012705b40 [0x0000004012705b00+0x0000000000000040]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.156601)
#
# An error report file with more information is saved as:
# //hs_err_pid156601.log
Compiled method (c1) 956 5 3 java.lang.Object::<init> (1 bytes)
total in heap [0x0000004012705910,0x0000004012705cb8] = 936
relocation [0x0000004012705a70,0x0000004012705aa0] = 48
constants [0x0000004012705ac0,0x0000004012705b00] = 64
main code [0x0000004012705b00,0x0000004012705c00] = 256
stub code [0x0000004012705c00,0x0000004012705c58] = 88
metadata [0x0000004012705c58,0x0000004012705c70] = 24
scopes data [0x0000004012705c70,0x0000004012705c80] = 16
scopes pcs [0x0000004012705c80,0x0000004012705cb0] = 48
dependencies [0x0000004012705cb0,0x0000004012705cb8] = 8
Compiled method (c1) 960 5 3 java.lang.Object::<init> (1 bytes)
total in heap [0x0000004012705910,0x0000004012705cb8] = 936
relocation [0x0000004012705a70,0x0000004012705aa0] = 48
constants [0x0000004012705ac0,0x0000004012705b00] = 64
main code [0x0000004012705b00,0x0000004012705c00] = 256
stub code [0x0000004012705c00,0x0000004012705c58] = 88
metadata [0x0000004012705c58,0x0000004012705c70] = 24
scopes data [0x0000004012705c70,0x0000004012705c80] = 16
scopes pcs [0x0000004012705c80,0x0000004012705cb0] = 48
dependencies [0x0000004012705cb0,0x0000004012705cb8] = 8
Could not load hsdis-s390x.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
#
Aborted (core dumped)
root@XX:/# ulimit -c unlimited
root@XX:/# java -version
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x0000004012706a40, pid=156619, tid=156622
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mi...

@davidhildenbrand The other issue which you have mentioned as duplicate shows java getting stuck for long, whereas for me it crashes right away. Do you think these 2 are related?

Also observed another behaviour :
java -version randomly passes, sometimes.

I can also confirm that it is observed under s390x chroot as well(logs below):
``` 
root@XX:/# ulimit -c unlimited
root@XX:/# java -version
openjdk version "11.0.10" 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode)
root@XX:/# java -version
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x0000004012705b40, pid=156601, tid=156604
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 5 c1 java.lang.Object.<init>()V java.base@11.0.10 (1 bytes) @ 0x0000004012705b40 [0x0000004012705b00+0x0000000000000040]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.156601)
#
# An error report file with more information is saved as:
# //hs_err_pid156601.log
Compiled method (c1)     956    5       3       java.lang.Object::<init> (1 bytes)
 total in heap  [0x0000004012705910,0x0000004012705cb8] = 936
 relocation     [0x0000004012705a70,0x0000004012705aa0] = 48
 constants      [0x0000004012705ac0,0x0000004012705b00] = 64
 main code      [0x0000004012705b00,0x0000004012705c00] = 256
 stub code      [0x0000004012705c00,0x0000004012705c58] = 88
 metadata       [0x0000004012705c58,0x0000004012705c70] = 24
 scopes data    [0x0000004012705c70,0x0000004012705c80] = 16
 scopes pcs     [0x0000004012705c80,0x0000004012705cb0] = 48
 dependencies   [0x0000004012705cb0,0x0000004012705cb8] = 8
Compiled method (c1)     960    5       3       java.lang.Object::<init> (1 bytes)
 total in heap  [0x0000004012705910,0x0000004012705cb8] = 936
 relocation     [0x0000004012705a70,0x0000004012705aa0] = 48
 constants      [0x0000004012705ac0,0x0000004012705b00] = 64
 main code      [0x0000004012705b00,0x0000004012705c00] = 256
 stub code      [0x0000004012705c00,0x0000004012705c58] = 88
 metadata       [0x0000004012705c58,0x0000004012705c70] = 24
 scopes data    [0x0000004012705c70,0x0000004012705c80] = 16
 scopes pcs     [0x0000004012705c80,0x0000004012705cb0] = 48
 dependencies   [0x0000004012705cb0,0x0000004012705cb8] = 8
Could not load hsdis-s390x.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
#   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
#
Aborted (core dumped)
root@XX:/# ulimit -c unlimited
root@XX:/# java -version
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x0000004012706a40, pid=156619, tid=156622
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 4 c1 java.lang.Object.<init>()V java.base@11.0.10 (1 bytes) @ 0x0000004012706a40 [0x0000004012706a00+0x0000000000000040]
#
.
(truncating logs)

Aborted (core dumped)
root@XX:/#
```

Increasing core limit worked once, but it fails eventually.

Could you please share your thoughts and provide some pointers on debugging further?

Revision history for this message

Namrata Bhave (nam121) wrote on 2021-04-14:

As java -version passes few times, further also checked behaviour of Maven. Observed that mvn -v crashes in a similar fashion, however after setting below:
export MAVEN_OPTS="-XX:-TieredCompilation -XX:+UseG1GC -Dcount=1000000"

mvn -v always passes.

root@XX:/# mvn -v
OpenJDK 64-Bit Server VM warning: You have loaded library /apache-maven-3.6.3/lib/jansi-native/linux64/libjansi.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /apache-maven-3.6.3
Java version: 11.0.7, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-s390x
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "5.4.0-70-generic", arch: "s390x", family: "unix"

However what I am really interested in, is mvn clean install command which never passes with above settings.

@davidhildenbrand, any help would be appreciated.

Revision history for this message

Jonathan Albrecht (jonalbrecht) wrote on 2021-04-27:

Download full text (4.9 KiB)

Hi @davidhildenbrand, I'm on the same team as @nam121 and I've been looking at this issue as well.

I think this is the same issue as: https://github.com/multiarch/qemu-user-static/issues/129

I've been running an s390x docker image on a master build (with latest s390x commit from Apr 23) of user mode qemu-s390x-static with some debug logging on:

$ sudo docker run -e QEMU_CPU="qemu" -e QEMU_LOG="unimp,guest_errors" -e QEMU_LOG_FILENAME="/s390x/qemu_s390x.log"

I ran a simple java program with:

$ java -Xcomp -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:PrintAssemblyOptions=hsdis-print-bytes -XX:+LogCompilation -XX:LogFile=java_compilation_log.log Main > java_out.txt

and the qemu log contained just one line:

unimplemented opcode 0x0000

Note that if the JIT is turned off with 'java -Xint', then all programs I've tried run without problem.

The hs_err file reports a SIGILL in the same spot as in the other comments:

--- SNIP
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x00000040126d7680, pid=208, tid=211
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, compiled mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 9 c1 java.lang.String.hashCode()I java.base (49 bytes) @ 0x00000040126d7680 [0x00000040126d7640+0x0000000000000040]
--- SNIP
--- SNIP
Instructions: (pc=0x00000040126d7680)
0x00000040126d7580: 00000040 5f5f4140 00000040 5f5f4140
0x00000040126d7590: 00000040 5f5f4140 00000040 5f5f4140
0x00000040126d75a0: 00000040 5f5f4358 00000040 5f5f4358
0x00000040126d75b0: 00000040 5f5f4358 00000040 5f5f4358
0x00000040126d75c0: 00000040 5f5f4140 00000040 5f5f4140
0x00000040126d75d0: 00000000 00000000 ffffffff ffffffff
0x00000040126d75e0: 00000040 5f5f4140 00000000 00000000
0x00000040126d75f0: ffffffff ffffffff 00000040 5f3fb9d0
0x00000040126d7600: 00000040 12238c00 00000040 12232800
0x00000040126d7610: 00000040 5f3fef18 00000040 12238c00
0x00000040126d7620: 00000040 12235000 00000000 00000000
0x00000040126d7630: 00000000 00000000 00000000 00000000
0x00000040126d7640: b9040009 cc08ffff fff85500 2008a784 # <-- String.hashCode() entry point at 0x00000040126d7640
0x00000040126d7650: 0019a51d 0040c019 12167a80 07f10700
0x00000040126d7660: 07000700 07000700 07000700 07000700
0x00000040126d7670: 07000700 07000700 07000700 07000700
0x00000040126d7680: 0000f000 ec51e3e0 f0080024 b904000f # <-- note 0x0000 at 0x00000040126d7680
0x00000040126d7690: a7fbffa0 e300f000 0024c438 ffffff73
--- SNIP

The assembly printed by java looks like:

--- SNIP
[Entry Point]
  # {method} {0x000000405f3fb9d0} 'hashCode' '()I' in 'java/lang/String'
  # [sp+0x60] (sp of caller)
  0x00000040126d7640: lgr %r0,%r9 ;...b9040009
                                                ; {no_reloc}
  0x00000040126d7644: aih %r0,-8 ;...cc08ffff fff8

0x00000040126d764a: cl %r0,8(%r2) ;...55002008

0x00000040126d764e: je 0x00000040126d7680 ;...a7840019

0x00000040126d7652: llihl %r1,64 ;...a51d0040

0x000000401...

Hi @davidhildenbrand, I'm on the same team as @nam121 and I've been looking at this issue as well.

I think this is the same issue as: https://github.com/multiarch/qemu-user-static/issues/129

I've been running an s390x docker image on a master build (with latest s390x commit from Apr 23) of user mode qemu-s390x-static with some debug logging on:

$ sudo docker run -e QEMU_CPU="qemu" -e QEMU_LOG="unimp,guest_errors" -e QEMU_LOG_FILENAME="/s390x/qemu_s390x.log"

I ran a simple java program with:

$ java -Xcomp -XX:+UnlockDiagnosticVMOptions -XX:+PrintAssembly -XX:PrintAssemblyOptions=hsdis-print-bytes -XX:+LogCompilation -XX:LogFile=java_compilation_log.log  Main > java_out.txt

and the qemu log contained just one line:

unimplemented opcode 0x0000

Note that if the JIT is turned off with 'java -Xint', then all programs I've tried run without problem.

The hs_err file reports a SIGILL in the same spot as in the other comments:

--- SNIP
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x00000040126d7680, pid=208, tid=211
#
# JRE version: OpenJDK Runtime Environment (11.0.10+9) (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.10+9-Ubuntu-0ubuntu1.20.04, compiled mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 9 c1 java.lang.String.hashCode()I java.base (49 bytes) @ 0x00000040126d7680 [0x00000040126d7640+0x0000000000000040]
--- SNIP
--- SNIP
Instructions: (pc=0x00000040126d7680)
0x00000040126d7580:   00000040 5f5f4140 00000040 5f5f4140
0x00000040126d7590:   00000040 5f5f4140 00000040 5f5f4140
0x00000040126d75a0:   00000040 5f5f4358 00000040 5f5f4358
0x00000040126d75b0:   00000040 5f5f4358 00000040 5f5f4358
0x00000040126d75c0:   00000040 5f5f4140 00000040 5f5f4140
0x00000040126d75d0:   00000000 00000000 ffffffff ffffffff
0x00000040126d75e0:   00000040 5f5f4140 00000000 00000000
0x00000040126d75f0:   ffffffff ffffffff 00000040 5f3fb9d0
0x00000040126d7600:   00000040 12238c00 00000040 12232800
0x00000040126d7610:   00000040 5f3fef18 00000040 12238c00
0x00000040126d7620:   00000040 12235000 00000000 00000000
0x00000040126d7630:   00000000 00000000 00000000 00000000
0x00000040126d7640:   b9040009 cc08ffff fff85500 2008a784  # <-- String.hashCode() entry point at 0x00000040126d7640
0x00000040126d7650:   0019a51d 0040c019 12167a80 07f10700
0x00000040126d7660:   07000700 07000700 07000700 07000700
0x00000040126d7670:   07000700 07000700 07000700 07000700
0x00000040126d7680:   0000f000 ec51e3e0 f0080024 b904000f  # <-- note 0x0000 at 0x00000040126d7680
0x00000040126d7690:   a7fbffa0 e300f000 0024c438 ffffff73
--- SNIP

The assembly printed by java looks like:

--- SNIP
[Entry Point]
  # {method} {0x000000405f3fb9d0} 'hashCode' '()I' in 'java/lang/String'
  #           [sp+0x60]  (sp of caller)
  0x00000040126d7640: lgr	%r0,%r9         ;...b9040009
                                                ;   {no_reloc}
  0x00000040126d7644: aih	%r0,-8          ;...cc08ffff fff8

0x00000040126d764a: cl	%r0,8(%r2)      ;...55002008

0x00000040126d764e: je	0x00000040126d7680  ;...a7840019

0x00000040126d7652: llihl	%r1,64          ;...a51d0040

0x00000040126d7656: iilf	%r1,303463040   ;...c0191216 7a80

0x00000040126d765c: br	%r1             ;...07f1

0x00000040126d765e: nopr                      ;...0700

0x00000040126d7660: nopr                      ;...0700

0x00000040126d7662: nopr                      ;...0700

0x00000040126d7664: nopr                      ;...0700

0x00000040126d7666: nopr                      ;...0700

0x00000040126d7668: nopr                      ;...0700

0x00000040126d766a: nopr                      ;...0700

0x00000040126d766c: nopr                      ;...0700

0x00000040126d766e: nopr                      ;...0700

0x00000040126d7670: nopr                      ;...0700

0x00000040126d7672: nopr                      ;...0700

0x00000040126d7674: nopr                      ;...0700

0x00000040126d7676: nopr                      ;...0700

0x00000040126d7678: nopr                      ;...0700

0x00000040126d767a: nopr                      ;...0700

0x00000040126d767c: nopr                      ;...0700

0x00000040126d767e: nopr                      ;...0700

[Verified Entry Point]
  0x00000040126d7680: tmy	-81920(%r15),222  ;...ebdef000 ec51

0x00000040126d7686: stg	%r14,8(%r15)    ;...e3e0f008 0024

0x00000040126d768c: lgr	%r0,%r15        ;...b904000f

0x00000040126d7690: aghi	%r15,-96        ;...a7fbffa0

0x00000040126d7694: stg	%r0,0(%r15)     ;...e300f000 0024

0x00000040126d769a: lgrl	%r3,0x00000040126d7580
                                                ;...c438ffff ff73
                                                ;   {metadata(method data for {method} {0x000000405f3fb9d0} 'hashCode' '()I' in 'java/lang/String')}
--- SNIP

so IIUC java says its generating 0xebde at 0x00000040126d7680 instead of 0x0000.

Hope the above makes sense. I'm not sure where to go from here so any suggestions would be a great help.

Revision history for this message

Jonathan Albrecht (jonalbrecht) wrote on 2021-05-03:

From looking at the in_asm logs, it looks like that instruction starting with 0xebde is executed once with no problem but the second time its changed to 0x0000.

... # First Time
----------------
IN:
0x40126d6880: ebde f000 ec51 tmy -0x14000(%r15), 0xde
0x40126d6886: e3e0 f008 0024 stg %r14, 8(%r15)
0x40126d688c: b904 000f lgr %r0, %r15
0x40126d6890: a7fb ffa0 aghi %r15, -0x60
0x40126d6894: e300 f000 0024 stg %r0, 0(%r15)
0x40126d689a: c438 ffff ff73 lgrl %r3, 0x40126d6780
0x40126d68a0: 5840 30dc l %r4, 0xdc(%r3)
0x40126d68a4: c248 0000 0008 agfi %r4, 8
0x40126d68aa: 5040 30dc st %r4, 0xdc(%r3)
0x40126d68ae: c0f4 0000 00d1 jg 0x40126d6a50
PSW=mask 0000000180000000 addr 00000040126d6880 cc CC_OP_LTGT0_64
R00=0000000000000000 R01=00000040126d6880 R02=00000006296f5d20 R03=00000006296f5d20
R04=000000405f45fcd8 R05=00000006000000e8 R06=0000004012169380 R07=0000004002c410e8
R08=0000004004019000 R09=000000405f2d29d0 R10=0000004002c41048 R11=00000006296095e0
R12=000000400280ec50 R13=0000004002c411d0 R14=00000040126d5c64 R15=0000004002c40e88

... # Second Time
unimplemented opcode 0x0000
----------------
IN:
PSW=mask 0000000180000000 addr 00000040126d6880 cc CC_OP_LTUGTU_32
R00=0000000000001808 R01=00000040126d53c0 R02=00000006296f5d78 R03=00000006296f5d78
R04=000000405f45fcd8 R05=00000006000000f0 R06=0000004012114000 R07=5f9dbb3700003030
R08=0000004004019000 R09=0000000800001808 R10=0000004002c41048 R11=00000006296095e0
R12=000000400280ec50 R13=0000004002c411d0 R14=00000040126d5c64 R15=0000004002c40e88

Revision history for this message

Namrata Bhave (nam121) wrote on 2021-05-04:

#10

Download full text (3.9 KiB)

Some more analysis:
Tried to explicitely compile as well as exclude few methods during compilation such as 'java.lang.StringLatin1::hashCode', 'java.util.concurrent.ConcurrentHashMap', 'java.lang.String*' which are part of trace as logged in above comments, with the help of advanced JIT options.
However it is not good enough to draw any conclusion as `java -version` command passes on random runs. `mvn -v` which consistently fails, is seen to be passing always with any of above combination set using MAVEN_OPTS.

Also compared the assembly log as @jonalbrecht mentioned above on qemu setup vs native s390x for `mvn -v` command.
The initial few compiled methods match, however it fails for 'java.lang.String::isLatin1':

Failure in qemu :
ImmutableOopMap{Z_R2=Oop }pc offsets: 170 232 244 272 Compiled method (c1) 1077 12 2 java.lang.String::equalsIgnoreCase (45 bytes)
total in heap [0x00000040117f2210,0x00000040117f28b0] = 1696
relocation [0x00000040117f2370,0x00000040117f23c8] = 88
constants [0x00000040117f2400,0x00000040117f2440] = 64
main code [0x00000040117f2440,0x00000040117f2600] = 448
stub code [0x00000040117f2600,0x00000040117f2668] = 104
metadata [0x00000040117f2668,0x00000040117f2688] = 32
scopes data [0x00000040117f2688,0x00000040117f2738] = 176
scopes pcs [0x00000040117f2738,0x00000040117f2888] = 336
dependencies [0x00000040117f2888,0x00000040117f2890] = 8
nul chk table [0x00000040117f2890,0x00000040117f28b0] = 32

ImmutableOopMap{}pc offsets: 288
ImmutableOopMap{Z_R2=Oop Z_R5=Oop }pc offsets: 372
ImmutableOopMap{Z_R5=Oop Z_R2=Oop }pc offsets: 384 392 400 unimplemented opcode 0x0000
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x00000040117f1680, pid=11738, tid=11787
#
# JRE version: OpenJDK Runtime Environment (11.0.11+9) (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.11+9-Ubuntu-0ubuntu2.20.04, compiled mode, tiered, compressed oops, g1 gc, linux-s390x)
# Problematic frame:
# J 9 c1 java.lang.String.hashCode()I java.base (49 bytes) @ 0x00000040117f1680 [0x00000040117f1640+0x0000000000000040]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to //core.11738)
#
# An error report file with more information is saved as:
# //hs_err_pid11738.log

Native s390x log:
ImmutableOopMap{Z_R2=Oop }pc offsets: 170 232 244 272 Compiled method (c1) 34 12 2 java.lang.String::equalsIgnoreCase (45 bytes)
total in heap [0x000003ff7a097110,0x000003ff7a0977b0] = 1696
relocation [0x000003ff7a097270,0x000003ff7a0972c8] = 88
constants [0x000003ff7a097300,0x000003ff7a097340] = 64
main code [0x000003ff7a097340,0x000003ff7a097500] = 448
stub code [0x000003ff7a097500,0x000003ff7a097568] = 104
metadata [0x000003ff7a097568,0x000003ff7a097588] = 32
scopes data [0x000003ff7a097588,0x000003ff7a097638] = 176
scopes pcs [0x000003ff7a097638,0x000003ff7a097788] = 336
dependencies [0x000003ff7a097788,0x000003ff7a097790] = 8
nul chk table [0x000003ff7a097790,0x000003ff7a097...

Other bug subscribers

Bug attachments

openjdk_crash_logs.log Edit

Add attachment

Remote bug watches

auto-github-multiarch-qemu-user-static #129
[closed] Edit
auto-gitlab.com-qemu-project-qemu-- #319
[closed Launchpad accel: TCG kind::Bug target: s390x workflow::In Progress] Edit

Bug watches keep track of this bug in other bug trackers.

Changed in qemu:
status:	New → Expired