Java 1.8.0_66 is extremely deadlock prone regarding AWT/GTK interaction

[md_5]

Since upgrading to Ubuntu 15.10 from Ubuntu 15.04, the various GUI applications that I run, including Netbeans and Minecraft have been suffering from dead/waitlocks which prevent them from operating properly. Since I have reproduced this issue these two completely different codebases, I am moving the bug further upstream.

The original Netbeans bug report can be viewed here: https://netbeans.org/bugzilla/show_bug.cgi?id=256124
Essentially, during routine interaction and calling of various AWT methods, somewhere between 8u45 (15.04) and 8u66 (15.10), something broke causing the application to hang intermittently when these methods are called.

Some sample jstack traces are attached, the issue should be fairly obvious to spot.

[md_5 (md-5)]

Issue lies somewhere in the (mis)use of AWT_LOCK.

[Papadakos Panagiotis (papadako)]

I also used to have this bug. Now netbeans just crashes after a while. Unfortunately there is not any useful message in the logs.

[Papadakos Panagiotis (papadako)]

Even running the netbeans installer gets killed.
Configuring the installer...
Searching for JVM on the system...
Extracting installation data...
Running the installer wizard...
Killed

How am I supposed to debug this?

[Papadakos Panagiotis (papadako)]

Well, I think it is related to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798131
Commenting out the line in /etc/java-7-openjdk/accessibility.properties or /etc/java-8-openjdk/accessibility.properties respectively,
does not help

[Papadakos Panagiotis (papadako)]

I also tried the patched openjdk 7 packages described in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797595 , in post #70 but again no luck. This is a rather important bug, since it affects my work (i.e. netbeans do not work).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[md_5 (md-5)]

@papadako
Your bug is unrelated.

[Papadakos Panagiotis (papadako)]

@md_5
A few days ago I had exactly the same bug, i.e. netbeans locking randomly in loading modules. From yesterday, that I updated the packages of my 15.10 system, netbeans, netbeans installer and freemind get killed randomly. From the related bugs that I have reported it seems that XInitThreads is not called at all, which could possible explain (the locking and crashing problems)
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797595#60

By the way, have you installed the latest updates? Do you experience the same bug if you downgrade to openjdk-7?

[md_5 (md-5)]

@papadako if you will look closely at the stack traces present you will see that the issue you are linking is not related to this one. Your netbeans locking up probably is, use jstack to confirm this.
The issue is not present in OpenJDK 1.7.0_85, and I have not yet reproduced it in Oracle JDK 8_66

[Papadakos Panagiotis (papadako)]

@md_5
Ok. One more question though. Have you updated your system with the latest updates?

[md_5 (md-5)]

Please read:
> Since upgrading to Ubuntu 15.10 from Ubuntu 15.04, the various GUI

[Papadakos Panagiotis (papadako)]

Dear md_5, I mean if you have the latest updates (i.e. apt-get update, apt-get upgrade). I am using 15.10 from its beta version, upgrading the packages every few days. I used to have the locking problem, but after installing the latest updates now netbeans just crashes. If you don't have a problem taking the risk that your netbeans might be unusable, I would like to see if you will experience the same problem. For example what "cat /var/log/dpkg.log | grep "\ install\ " |tail -n 10" report?

[Papadakos Panagiotis (papadako)]

Ok, I just found my problem, and it is irrelevant to what is described here. A server I recently installed was killing every 1 minute
each java applications I had running.

By the way, have you tried what I suggested previously, i.e. editing
/etc/java-8-openjdk/accessibility.properties and commenting out the line assistive_technologies=org.GNOME.Accessibility.AtkWrapper?

[md_5 (md-5)]

Tagging Matthias onto this bug.

Haven't see any issues with removing AtkWrapper, but too early to tell.

[Pauli (paniemin)]

I noticed the hang in load_gtk when trying to run netbeans with openjdk-7. For me disabling accessible like in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798131. The hand seems to be X11 threading related based on other similar hangs reported in web.

My java backtrace:
"AWT-EventQueue-1" prio=10 tid=0x00007fc0dc163000 nid=0xb4e runnable [0x00007fc0d2b6f000]
   java.lang.Thread.State: RUNNABLE
 at sun.awt.UNIXToolkit.load_gtk(Native Method)
 at sun.awt.UNIXToolkit.loadGTK(UNIXToolkit.java:108)
 - locked <0x00000000d12681f0> (a java.lang.Object)
 at com.sun.java.swing.plaf.gtk.GTKLookAndFeel.initialize(GTKLookAndFeel.java:1452)
 at javax.swing.UIManager.setLookAndFeel(UIManager.java:535)
 at org.netbeans.swing.plaf.Startup.initialize(Unknown Source)
 at org.netbeans.swing.plaf.Startup.<init>(Unknown Source)
 at org.netbeans.swing.plaf.Startup.run(Unknown Source)
 at org.netbeans.core.CoreBridgeImpl$1$1.run(Unknown Source)
 at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
 at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:745)
 at java.awt.EventQueue.access$300(EventQueue.java:103)
 at java.awt.EventQueue$3.run(EventQueue.java:706)
 at java.awt.EventQueue$3.run(EventQueue.java:704)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:77)
 at java.awt.EventQueue.dispatchEvent(EventQueue.java:715)
 at org.GNOME.Accessibility.AtkWrapper$5.dispatchEvent(AtkWrapper.java:697)
 at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
 at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
 at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
 at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)

My native backtrace for same thread:
Thread 8 (Thread 0x7fc0d2b71700 (LWP 2894)):
#0 0x00007fc1399d98dd in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fc1231dbbd2 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#2 0x00007fc1231dd4df in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3 0x00007fc1231dd5f1 in xcb_wait_for_reply () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#4 0x00007fc123d03a47 in _XReply () from /usr/lib/x86_64-linux-gnu/libX11.so.6
#5 0x00007fc123cf99e6 in XQueryExtension () from /usr/lib/x86_64-linux-gnu/libX11.so.6
#6 0x00007fc123ced4f2 in XInitExtension () from /usr/lib/x86_64-linux-gnu/libX11.so.6
#7 0x00007fc128269fdf in XextAddDisplay () from /usr/lib/x86_64-linux-gnu/libXext.so.6
#8 0x00007fc1238b6f63 in ?? () from /usr/lib/x86_64-linux-gnu/libXi.so.6
#9 0x00007fc1238b59a9 in XListInputDevices () from /usr/lib/x86_64-linux-gnu/libXi.so.6
#10 0x00007fc119ddbf99 in ?? () from /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0
#11 0x00007fc119db76a0 in gdk_display_open () from /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0
#12 0x00007fc119d8704f in gdk_display_open_default_libgtk_only ()
   from /usr/lib/x86_64-linux-gnu/libgdk...

[Pauli (paniemin)]

I managed to find a very similar sounding bug that is from year 2012 but it was iced-tea specific: http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=857

Solution there was to make sure that setLookAndFeel call happens before any other call to swing. But I have no idea how that could be related to this bug. Only similarity is that atk-wrapper seems to trigger bugs in java UI code. Possible completely different issue.

Redhat bug report without any solution from year 2014: https://bugzilla.redhat.com/show_bug.cgi?id=1073984

Debian bug without solution from year 2013: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705511

[md_5 (md-5)]

atk-wrapper went through some quite big updates between 15.04 and 15.10, so I'd be betting on that being the issue (or it somehow triggering a Java edge case)

[Alessandro Polverini (polve)]

I confirm the problems with openJDK 8 (at least with current version 8u66-b17-1), I'm unable to even run the netbeans installer, while using Oracle JDK it works fine.

If I install netbeans with Oracle JDK and then run it with OpenJDK 8 the experience is VERY bad, with very long pauses here and there, making it almost unusable.

It used to work fine in Ubuntu 15.04

[Pauli (paniemin)]

"atk-wrapper went through some quite big updates between 15.04 and 15.10, so I'd be betting on that being the issue (or it somehow triggering a Java edge case)"

I don't think that is the primary bug. Bugs in Debian and Redhat are much older and look like perfect match to this bug. It feels like a race condition somewhere but I don't know how to debug jvm. Maybe some kind of trace from xcb telling which threads are using which connection would help.

[Pauli (paniemin)]

src/solaris/native/sun/awt/awt_UNIXToolkit.c:
    AWT_LOCK();
    XSync(awt_display, False);
    AWT_UNLOCK();

Looks like X11 is supposed to be protected by AWT_LOCK.

src/solaris/classes/sun/awt/UNIXToolkit.java:
    /** All calls into GTK should be synchronized on this lock */
    public static final Object GTK_LOCK = new Object();

It looks like also gtk is protected with a java lock.

Mean while looking at java-atk-wrapper code it is using gtk/atk/gdk without unix toolkit lock.

But I simple can't remember any specific reason to use locks if opening two separate X display for two threads.

Also I can't understand how gdk_open_display can hit a race condition ...

[Franck (alci)]

I confirm the following workaround is working: comment out the line "assistive_technologies=org.GNOME.Accessibility.AtkWrapper" in /etc/java-7-openjdk/accessibility.properties

I had Freeplane systematically hanging when browsing or even saving a file, and the works around the problem perfectly. But this should definitely be fixed as many Java apps seems to be affected....

[Daniel (daniel-nuest)]

I can confirm the workaround described in #24 (https://bugs.launchpad.net/ubuntu/+source/openjdk-8/+bug/1510009/comments/24) as well for Ubuntu 15.10. and Netbeans 8.1 with OpenJDK Runtime Environment (build 1.8.0_66-internal-b17).

[Don-vip (vincent-privat)]

GNOME Atk wrapper is unmaintained, it must be disabled in Debian openjdk packaging asap.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813143

[Joachim Durchholz (jo-durchholz)]

For me, I could reproduce the hang by doing this in a fresh-ish install:
1) Start Netbeans
2) Tools - Options - Fonts & Colors
3) With Profile at NetBeans and the Syntax tab selected, click on the ellipsis button to the right of the F_ont field.
4) Netbeans hangs.

I can confirm that commenting out AtkWrapper in /etc/java-7-openjdk/accessibility.properties fixed the hang.

[Matthias Klose (doko)]

please recheck with 16.04

[Alan Ferguson (alan-ferguson)]

Joachim Durchholz (jo-durchholz) offered a fix to the locking problem (entry #27). I had the same problem. It took several tries to get netBeans 8.1 started (using java-7-openjdk), and then it would lock occasionally when it was started (at the ...Starting Modules). It was finally working reasonably well when I tried to get the red, vertical line removed. After that, it locked up before the column for the red line could be changed, and did not load after that at all, after 4 or 5 tries.

I commented out the line in the accessibility.properties file, and NetBeans 8.1 has loaded every time since that fix (only 3 or 4 times so far). A benefit was that I was also able to change te excessive column to line 200, and the text box in which the column number was contained was now wide enough to see all 3 digits (200).

I am just starting to try NetBeans now, so I don't consider myself an expert on this development system.

[liamdawe (liamdawe)]

I believe this is the issue I am also coming across trying to play games that use Java on Ubuntu 16.04, I consider it a major issue since gaming is driving us a lot right now.

Here's an example when trying to launch Retro-Pixel Castles:
Exception in thread "main" java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.UnsatisfiedLinkError: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/libatk-wrapper.so: /usr/lib/x86_64-linux-gnu/libatspi.so.0: undefined symbol: g_type_class_adjust_private_offset
 at java.lang.ClassLoader$NativeLibrary.load(Native Method)
 at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1938)
 at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1834)
 at java.lang.Runtime.loadLibrary0(Runtime.java:870)
 at java.lang.System.loadLibrary(System.java:1122)
 at org.GNOME.Accessibility.AtkWrapper.<clinit>(AtkWrapper.java:39)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at java.lang.Class.newInstance(Class.java:442)
 at java.awt.Toolkit.loadAssistiveTechnologies(Toolkit.java:805)
 at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:886)
 at org.lwjgl.LinuxSysImplementation.<clinit>(LinuxSysImplementation.java:50)
 at org.lwjgl.Sys.createImplementation(Sys.java:131)
 at org.lwjgl.Sys.<clinit>(Sys.java:116)
 at org.lwjgl.opengl.Display.<clinit>(Display.java:135)
 at rpc.system.ScaleControl.setWorldScale(ScaleControl.java:56)
 at rpc.system.ScaleControl.initDisplay(ScaleControl.java:17)
 at rpc.system.Game.launchGame(Game.java:73)
 at rpc.system.Launcher.main(Launcher.java:47)
 ... 5 more
Game removed: AppID 328080 "Retro-Pixel Castles", ProcID 6188

[liamdawe (liamdawe)]

About my last comment, it's only an issue in OpenJDK and not Oracle Java.

[Matthias Klose (doko)]

now turned off by default in 16.04