Segmentation fault in os::write_memory_serialize_page

[Zhihong Yu]

uname -a
Linux securecluster 4.9.8-moby #1 SMP Wed Feb 8 09:56:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I was troubleshooting HBASE-18541 on docker VM.

Here was stack trace provided by gdb:

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fb387315dc8 in os::write_memory_serialize_page (thread=0x2af3000) at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.hpp:419
419 /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.hpp: No such file or directory.
[Current thread is 1 (Thread 0x7fb387dbe840 (LWP 9221))]
Installing openjdk unwinder
(gdb) bt
#0 0x00007fb387315dc8 in ThreadStateTransition::transition_and_fence(JavaThread*, JavaThreadState, JavaThreadState) (thread=0x2af3000)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.hpp:419
#1 0x00007fb387315dc8 in ThreadStateTransition::transition_and_fence(JavaThread*, JavaThreadState, JavaThreadState) (thread=0x2af3000)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/interfaceSupport_linux.hpp:31
#2 0x00007fb387315dc8 in ThreadStateTransition::transition_and_fence(JavaThread*, JavaThreadState, JavaThreadState) (thread=thread@entry=0x2af3000, to=_thread_in_native, from=_thread_in_vm) at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/interfaceSupport.hpp:179
#3 0x00007fb38731719f in JVM_FillInStackTrace(JNIEnv*, jobject) (to=_thread_in_native, from=_thread_in_vm, this=<synthetic pointer>)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/interfaceSupport.hpp:232
#4 0x00007fb38731719f in JVM_FillInStackTrace(JNIEnv*, jobject) (this=<synthetic pointer>, __in_chrg=<optimized out>)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/interfaceSupport.hpp:281
#5 0x00007fb38731719f in JVM_FillInStackTrace(JNIEnv*, jobject) (env=<optimized out>, receiver=receiver@entry=0x7ffde93448a0)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/prims/jvm.cpp:516
#6 0x00007fb38395e851 in Java_java_lang_Throwable_fillInStackTrace (env=<optimized out>, throwable=0x7ffde93448a0, dummy=<optimized out>)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/jdk/src/share/native/java/lang/Throwable.c:49
#7 0x00007fb373eb9a28 in [native offset=0xa8] java.lang.Throwable.fillInStackTrace(int) () at java/lang/Throwable.java
#8 0x00007fb3743472a4 in [compiled offset=0x84] java.lang.Throwable.fillInStackTrace() () at java/lang/Throwable.java:781
#9 0x00007fb3743bc914 in [compiled offset=0x194] java.lang.Throwable.<init>() () at java/lang/Throwable.java:249
#10 0x00007fb37421a0d4 in [compiled offset=0x1b4] org.apache.log4j.helpers.PatternParser$LocationPatternConverter.convert(org.apache.log4j.spi.LoggingEvent) ()
    at org/apache/log4j/helpers/PatternParser.java:500
#11 0x00007fb37417eab4 in [compiled offset=0x114] org.apache.log4j.helpers.PatternConverter.format(java.lang.StringBuffer,org.apache.log4j.spi.LoggingEvent) ()
    at org/apache/log4j/helpers/PatternConverter.java:65
#12 0x00007fb37426315c in [inlined] java.lang.StringBuffer.setLength(int) () at java/lang/StringBuffer.java:193
0x00007fb37426315c in [compiled offset=0x71c] org.apache.log4j.PatternLayout.format(org.apache.log4j.spi.LoggingEvent) () at org/apache/log4j/PatternLayout.java:503
#13 0x00007fb37454484c in [compiled offset=0x12c] org.apache.log4j.WriterAppender.subAppend(org.apache.log4j.spi.LoggingEvent) () at org/apache/log4j/WriterAppender.java:310
#14 0x00007fb374538aac in [compiled offset=0x1ec] org.apache.log4j.WriterAppender.append(org.apache.log4j.spi.LoggingEvent) () at org/apache/log4j/WriterAppender.java:160
#15 0x00007fb37454793c in [compiled offset=0x113c] org.apache.log4j.AppenderSkeleton.doAppend(org.apache.log4j.spi.LoggingEvent) ()
    at org/apache/log4j/AppenderSkeleton.java:251
#16 0x00007fb374074204 in [compiled offset=0x4c4] org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(org.apache.log4j.spi.LoggingEvent) ()
    at org/apache/log4j/helpers/AppenderAttachableImpl.java:66
#17 0x00007fb3742b5f24 in [compiled offset=0x1e4] org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) () at org/apache/log4j/Category.java:200
#18 0x00007fb374208d5c in [inlined] org.apache.log4j.Category.forcedLog(java.lang.String,org.apache.log4j.Priority,java.lang.Object,java.lang.Throwable) ()
    at org/apache/log4j/Category.java:392
0x00007fb374208d5c in [compiled offset=0x67c] org.apache.log4j.Category.log(java.lang.String,org.apache.log4j.Priority,java.lang.Object,java.lang.Throwable) ()
    at org/apache/log4j/Category.java:858
#19 0x00007fb37454b374 in [compiled offset=0x154] org.apache.commons.logging.impl.Log4JLogger.info(java.lang.Object) () at org/apache/commons/logging/impl/Log4JLogger.java:177
#20 0x00007fb373cee042 in [interpreted: bc = 50] org.apache.hadoop.hbase.regionserver.HRegionServer.stop(java.lang.String) ()
    at org/apache/hadoop/hbase/regionserver/HRegionServer.java:1925

[Zhihong Yu (tedyu)]

In one of the crashes, I saw:

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f61d5853a9a in ?? ()
[Current thread is 1 (Thread 0x7f61e95af840 (LWP 13834))]
Installing openjdk unwinder
(gdb) bt
#0 0x00007f61d5853a9a in ()
#1 0x00007ffd17e05df0 in ()
#2 0x00007f61d584c21c in [compiled offset=0x1bc] java.io.ExpiringCache.get(java.lang.String) () at java/io/ExpiringCache.java:78
#3 0x00007f61d592609c in [compiled offset=0x1bc] java.io.UnixFileSystem.canonicalize(java.lang.String) () at java/io/UnixFileSystem.java:153
#4 0x00007f61d591b494 in [inlined] java.io.UnixFileSystem.resolve(java.io.File) () at java/io/UnixFileSystem.java:133
0x00007f61d591b494 in [compiled offset=0x5b4] java.io.File.getCanonicalPath() () at java/io/File.java:618
#5 0x00007f61d54ded80 in [interpreted: bc = 1] java.io.File.getCanonicalFile() () at java/io/File.java:644
#6 0x00007f61d54ded80 in [interpreted: bc = 59] org.apache.commons.io.FileUtils.isSymlink(java.io.File) () at org/apache/commons/io/FileUtils.java:2936
#7 0x00007f61d54de7d0 in [interpreted: bc = 9] org.apache.commons.io.FileUtils.deleteDirectory(java.io.File) () at org/apache/commons/io/FileUtils.java:1535
#8 0x00007f61d54deffd in [interpreted: bc = 8] org.apache.commons.io.FileUtils.forceDelete(java.io.File) () at org/apache/commons/io/FileUtils.java:2272
#9 0x00007f61d54deffd in [interpreted: bc = 134] org.apache.commons.io.FileUtils.cleanDirectory(java.io.File) () at org/apache/commons/io/FileUtils.java:1656
#10 0x00007f61d54deffd in [interpreted: bc = 16] org.apache.commons.io.FileUtils.deleteDirectory(java.io.File) () at org/apache/commons/io/FileUtils.java:1538
#11 0x00007f61d54deffd in [interpreted: bc = 8] org.apache.commons.io.FileUtils.forceDelete(java.io.File) () at org/apache/commons/io/FileUtils.java:2272
#12 0x00007f61d54deffd in [interpreted: bc = 134] org.apache.commons.io.FileUtils.cleanDirectory(java.io.File) () at org/apache/commons/io/FileUtils.java:1656
#13 0x00007f61d54deffd in [interpreted: bc = 16] org.apache.commons.io.FileUtils.deleteDirectory(java.io.File) () at org/apache/commons/io/FileUtils.java:1538
#14 0x00007f61d54deffd in [interpreted: bc = 8] org.apache.commons.io.FileUtils.forceDelete(java.io.File) () at org/apache/commons/io/FileUtils.java:2272
#15 0x00007f61d54deffd in [interpreted: bc = 134] org.apache.commons.io.FileUtils.cleanDirectory(java.io.File) () at org/apache/commons/io/FileUtils.java:1656
#16 0x00007f61d54deffd in [interpreted: bc = 16] org.apache.commons.io.FileUtils.deleteDirectory(java.io.File) () at org/apache/commons/io/FileUtils.java:1538
#17 0x00007f61d54deffd in [interpreted: bc = 8] org.apache.commons.io.FileUtils.forceDelete(java.io.File) () at org/apache/commons/io/FileUtils.java:2272
#18 0x00007f61d54deffd in [interpreted: bc = 134] org.apache.commons.io.FileUtils.cleanDirectory(java.io.File) () at org/apache/commons/io/FileUtils.java:1656
#19 0x00007f61d54deffd in [interpreted: bc = 16] org.apache.commons.io.FileUtils.deleteDirectory(java.io.File) () at org/apache/commons/io/FileUtils.java:1538
#20 0x00007f61d54deffd in [interpreted: bc = 8] org.apache.commons.io.FileUtils.forceDelete(java.io.File) () at org/ap...

[Tiago Stürmer Daitx (tdaitx)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Your report is missing the following information:
- OpenJDK 8 version
- Ubuntu Version/Release

In order to collect this information through apport please execute the following command only once, as it will automatically gather debugging information, in a terminal:

apport-collect 1710674

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

Alternatively you can fetch the same information by issuing the following 2 commands:

apt-cache policy openjdk-8-jre
cat /etc/os-release

Just paste the output in a comment or add them to the bug description.

[Zhihong Yu (tedyu)]

# apt-cache policy openjdk-8-jre
openjdk-8-jre:
  Installed: 8u131-b11-2ubuntu1.16.04.3
  Candidate: 8u131-b11-2ubuntu1.16.04.3
  Version table:
 *** 8u131-b11-2ubuntu1.16.04.3 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     8u77-b03-3ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial

[Zhihong Yu (tedyu)]

Any more information that I should provide ?

[Tiago Stürmer Daitx (tdaitx)]

Thanks for the information!

Additionally to that:
1. Were you able to isolate and find reproducible steps for this problem?
2. Do you happen to have the hs_err log files?

I'm setting it as new, will do some investigation on this tomorrow.

[Zhihong Yu (tedyu)]

I haven't found reproducible steps for segfault.

I didn't pay attention to hs_err log file. Would it be generated under PWD ?

Here was another stack trace collected by my coworker on his machine:

#0 0x00007fc70d8a3d18 in ThreadStateTransition::transition_from_native(JavaThread*, JavaThreadState) (thread=0x17e7000)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/os.hpp:419
#1 0x00007fc70d8a3d18 in ThreadStateTransition::transition_from_native(JavaThread*, JavaThreadState) (thread=0x17e7000)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/os/linux/vm/interfaceSupport_linux.hpp:31
#2 0x00007fc70d8a3d18 in ThreadStateTransition::transition_from_native(JavaThread*, JavaThreadState) (thread=thread@entry=0x17e7000, to=_thread_in_vm)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/interfaceSupport.hpp:212
#3 0x00007fc70d8a5346 in JVM_IHashCode(JNIEnv*, jobject) (to=_thread_in_vm, this=<synthetic pointer>)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/interfaceSupport.hpp:231
#4 0x00007fc70d8a5346 in JVM_IHashCode(JNIEnv*, jobject) (thread=<optimized out>, this=<synthetic pointer>)
    at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/runtime/interfaceSupport.hpp:278
#5 0x00007fc70d8a5346 in JVM_IHashCode(JNIEnv*, jobject) (env=<optimized out>, handle=0x7ffd1ded8970) at /build/openjdk-8-pZyJp3/openjdk-8-8u131-b11/src/hotspot/src/share/vm/prims/jvm.cpp:542
#6 0x00007fc6fa61ebbf in [native offset=0xbf] java.lang.System.identityHashCode(java.lang.Object) () at java/lang/System.java
#7 0x00007fc6fa27ba40 in [interpreted: bc = 4] org.apache.hadoop.security.UserGroupInformation.hashCode() () at org/apache/hadoop/security/UserGroupInformation.java:1616
#8 0x00007fc6fa27ba40 in [interpreted: bc = 4] org.apache.hadoop.hbase.security.User.hashCode() () at org/apache/hadoop/hbase/security/User.java:152
#9 0x00007fc6fa27ba40 in [interpreted: bc = 22] org.apache.hadoop.hbase.ipc.ConnectionId.hashCode(org.apache.hadoop.hbase.security.User,java.lang.String,java.net.InetSocketAddress) ()
    at org/apache/hadoop/hbase/ipc/ConnectionId.java:79
#10 0x00007fc6fa27ba40 in [interpreted: bc = 84] org.apache.hadoop.hbase.ipc.AsyncRpcClient.createRpcChannel(java.lang.String,java.net.InetSocketAddress,org.apache.hadoop.hbase.security.User) ()
    at org/apache/hadoop/hbase/ipc/AsyncRpcClient.java:413

[Zhihong Yu (tedyu)]

Is there any more information I should provide ?

[Zhihong Yu (tedyu)]

Gentle ping.

[Tiago Stürmer Daitx (tdaitx)]

I took a look at the core file and the stacktraces, but so far I couldn't pinpoint what might be causing that, much less reproduce it.

The hs_err log files would help, they are usually saved on the current dir of the process. The location might be modified by the '-XX:ErrorFile=' java option.

One of the stacktraces led me to the indication that -XX:+UseMembar might help depending on what is going on [1].

A few additional questions:
1. Could you please confirm what was the previous version of OpenJDK where everything worked fine?
2. Are all the errors from the same environment? By environment I mean either hardware, same OS, or same image - uname indicates this is a docker container, not a default Ubuntu installation.

Thanks!

[1] http://www.jwrapper.com/blog/high-privileged-time-and-exceptions-important-java-jvm-options-you-should-consider-if-running-a-server

[Zhihong Yu (tedyu)]

I tried the following passing the flag to JNI_CreateJavaVM() call:

  auto options = std::string{"-XX:+UseMembar -Djava.class.path="} + clspath;
...
  rv = JNI_CreateJavaVM(jvm, reinterpret_cast<void **>(&env), &args);

rv was -6 from the call.

Our unit test, written in C++, the intermittent crash has been happening quite early in our development. So I don't know which JDK version worked.

openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~16.04.1-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

Development is done inside docker VM.
If you clone HBASE-14850 branch of hbase git repo, you would be able to reproduce by running the following commands:

hbase-native-client/bin/start-docker.sh
for i in `seq 1 10`; do buck test --no-results-cache core:multi-retry-test || break 1; done

Thanks

[Zhihong Yu (tedyu)]

I managed to apply -XX:+UseMembar to the JVM created by unit test.

However, the crash still happens

[Zhihong Yu (tedyu)]

Gentle ping.