Segmentation fault

[Ubuntu Report]

DISTRIB_RELEASE=16.04
snoopy 2.3.1-2 amd64

/lib/libsnoopy.so.0.0.0 -> Segmentation fault (core dumped)
/lib/snoopy.so -> Segmentation fault (core dumped)

[Ubuntu Report (ubuntureport)]

We found the bug. The problem was the syslog-ng configuration file.

/etc/syslog-ng/syslog-ng.conf
14.04:
unix-stream("/dev/log");
16.04:
unix-dgram("/dev/log");

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Thayne (thayne-u)]

I'm not sure if this is the same issue. But while running BitBucket we got a segfault about once a day (and had to restart bitbucket), until we figured out it was coming from snoopy code and disabled snoopy on the host.

The relevant part of the backtrace:

#1236 0x00007fa0e54ac746 in strlen () at ../sysdeps/x86_64/strlen.S:106
#1237 0x00007fa0e59fd0fd in snoopy_datasource_cmdline () at /lib/snoopy.so
#1238 0x00007fa0e59fcdc4 in snoopy_message_generateFromFormat () at /lib/snoopy.so
#1239 0x00007fa0e59fcc6b in snoopy_log_syscall () at /lib/snoopy.so
#1240 0x00007fa0e59fca45 in execve () at /lib/snoopy.so
#1241 0x00007fa0e2fd8284 in execve_with_shell_fallback (mode=<optimized out>, file=0x7fa081e75d60 "/usr/bin/git", argv=0x7fa080bfd5e0, envp=0x7fa081316ad0)
    at /build/openjdk-8-W2Qe27/openjdk-8-8u151-b12/src/jdk/src/solaris/native/java/lang/childproc.c:213
#1242 0x00007fa0e2fd86a9 in childProcess (arg=arg@entry=0x7fa0808e4ee0) at /build/openjdk-8-W2Qe27/openjdk-8-8u151-b12/src/jdk/src/solaris/native/java/lang/childproc.c:365
#1243 0x00007fa0e2fd5114 in vforkChild (c=c@entry=0x7fa0808e4ee0) at /build/openjdk-8-W2Qe27/openjdk-8-8u151-b12/src/jdk/src/solaris/native/java/lang/UNIXProcess_md.c:434
#1244 0x00007fa0e2fd580d in Java_java_lang_UNIXProcess_forkAndExec (env=0x7fa09c73f9e0, process=<optimized out>, helperpath=<optimized out>, c=0x7fa0808e4ee0)
    at /build/openjdk-8-W2Qe27/openjdk-8-8u151-b12/src/jdk/src/solaris/native/java/lang/UNIXProcess_md.c:552
#1245 0x00007fa0e2fd580d in Java_java_lang_UNIXProcess_forkAndExec (env=0x7fa09c73f9e0, process=<optimized out>, mode=3, helperpath=0x7fa03276cef0, prog=0x7fa03276cef8, argBlock=0x7fa03276cf00, argc=2, envBlock=0x7fa03276cf30, envc=21, dir=0x7fa03276cf40, std_fds=0x7fa03276cf48, redirectErrorStream=0 '\000') at /build/openjdk-8-W2Qe27/openjdk-8-8u151-b12/src/jdk/src/solaris/native/java/lang/UNIXProcess_md.c:645
#1246 0x00007fa0cf948ac9 in [native offset=0x149] java.lang.UNIXProcess.forkAndExec(int,byte,byte,byte,int,byte,int,byte,int,boolean) () at java/lang/UNIXProcess.java
#1247 0x00007fa0d43e3394 in [inlined] java.lang.StringCoding.encode(char,int,int) () at java/lang/StringCoding.java:388
0x00007fa0d43e3394 in [inlined] java.lang.String.getBytes() () at java/lang/String.java:958
0x00007fa0d43e3394 in [inlined] java.lang.ProcessImpl.toCString(java.lang.String) () at java/lang/ProcessImpl.java:51
0x00007fa0d43e3394 in [compiled offset=0x834] java.lang.ProcessImpl.start(java.lang.String,java.util.Map,java.lang.String,java.lang.ProcessBuilder$Redirect,boolean) () at java/lang/ProcessImpl.java:134
#1248 0x00007fa0d3745548 in [inlined] com.atlassian.utils.process.ExternalProcessImpl.createDefaultProcess(java.util.List,java.util.Map,java.io.File) () at com/atlassian/utils/process/ExternalProcessImpl.java:376
0x00007fa0d3745548 in [compiled offset=0x7c8] com.atlassian.utils.process.ExternalProcessImpl.createProcess(java.util.List,java.util.Map,java.io.File) () at com/atlassian/utils/process/ExternalProcessImpl.java:387
#1249 0x00007fa0d52478cc in [inlined] com.atlassian.utils.process.ExternalProcessBuilder.build() () at com/atlassian/utils/process/ExternalProcessBuilder.java:55

May be relat...

[Bostjan Skufca Jese (bostjanskufcajese)]

@Thayne, can this one be reproduced with the latest Snoopy versions (2.4.9)? If so, can you provide either a patch that fixes it, or a method that I can use to reproduce the issue locally?

[Thayne (thayne-u)]

I can look at seeing if it can be reproduced with a newer version. 2.4.6 would be easier to test than 2.4.9, since it is in the apt repos for 18.04 and 20.04, but I might be able to compile 2.4.9 as well.

As far as reproducing it, I don't have great reproduction steps. I've only seen this happen with bitbucket server (which is proprietary), and even then it was happening somewhat randomly, but about once a day.

[Bostjan Skufca Jese (bostjanskufcajese)]

@Thanye, I've checked the stack trace above again, and then inspected the git history since 2.4.6 has been released in 2016, and only the following commits have touched the src/datasource/cmdline.c file:
- 4b4ab04 Sun Oct 4 2020 +0200 GH #157: Fix (potentially*) incorrect malloc size in cmdline
- ede16a4 Mon Nov 25 2019 +0000 correct pointer vs byte compare in cmdline.c
- e989f26 Thu Jul 27 2017 +0200 Make snoopy compile on 64-bit Arch Linux with GCC 7.1.1

None of these commits seems to be resolving an issue that could potentially result in a segmentation fault.

However, "cmdline" datasource is one of the two Snoopy datasources that are accessing Snoopy's "global" (in the context of a single process) data storage facility. If your application is heavily threaded, this could potentially result in a segmentation fault, if the conditions are just right.

Therefore, only upgrading to 2.4.9 should not resolve this particular issue as far as I can see now. But using the --enable-thread-safety ./configure flag might.

If you're willing to build Snoopy yourself with the --enable-thread-safety flag and test this hypothesis out, let me know the outcome. I am looking at switching thread safety build flag from default off to default on in the upcoming 2.5.0 release.

[Thayne (thayne-u)]

@Bostjan

I'm really sorry it took so long to get back to you. I just kept putting it off because I was busy, and never got around to testing it.

But we recently ran into this issue again, and tested with building with the --enable-thread-safety configure flag, and that did in fact fix the issue.

[Bostjan Skufca Jese (bostjanskufcajese)]

@Thayne,

No worries, and thanks for getting back to (not just) me. In fact, this is not the first Java application with such Snoopy behaviour that I've heard about.

https://github.com/a2o/snoopy/issues/215 will make sure thread safety gets enabled by default starting with Snoopy version 2.5.0.