Java 1.8.0_66 is extremely deadlock prone regarding AWT/GTK interaction

Bug #1510009 reported by md_5
80
This bug affects 17 people
Affects Status Importance Assigned to Milestone
JOSM
Unknown
Critical
java-atk-wrapper (Debian)
Fix Released
Unknown
java-atk-wrapper (Ubuntu)
High
Unassigned

Bug Description

Since upgrading to Ubuntu 15.10 from Ubuntu 15.04, the various GUI applications that I run, including Netbeans and Minecraft have been suffering from dead/waitlocks which prevent them from operating properly. Since I have reproduced this issue these two completely different codebases, I am moving the bug further upstream.

The original Netbeans bug report can be viewed here: https://netbeans.org/bugzilla/show_bug.cgi?id=256124
Essentially, during routine interaction and calling of various AWT methods, somewhere between 8u45 (15.04) and 8u66 (15.10), something broke causing the application to hang intermittently when these methods are called.

Some sample jstack traces are attached, the issue should be fairly obvious to spot.

Revision history for this message
md_5 (md-5) wrote :
Revision history for this message
md_5 (md-5) wrote :
Revision history for this message
md_5 (md-5) wrote :
Revision history for this message
md_5 (md-5) wrote :

Issue lies somewhere in the (mis)use of AWT_LOCK.

Revision history for this message
Papadakos Panagiotis (papadako) wrote :

I also used to have this bug. Now netbeans just crashes after a while. Unfortunately there is not any useful message in the logs.

Revision history for this message
Papadakos Panagiotis (papadako) wrote :

Even running the netbeans installer gets killed.
Configuring the installer...
Searching for JVM on the system...
Extracting installation data...
Running the installer wizard...
Killed

How am I supposed to debug this?

Revision history for this message
Papadakos Panagiotis (papadako) wrote :

Well, I think it is related to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798131
Commenting out the line in /etc/java-7-openjdk/accessibility.properties or /etc/java-8-openjdk/accessibility.properties respectively,
does not help

Revision history for this message
Papadakos Panagiotis (papadako) wrote :

I also tried the patched openjdk 7 packages described in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797595 , in post #70 but again no luck. This is a rather important bug, since it affects my work (i.e. netbeans do not work).

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openjdk-8 (Ubuntu):
status: New → Confirmed
Revision history for this message
md_5 (md-5) wrote :

@papadako
Your bug is unrelated.

Revision history for this message
Papadakos Panagiotis (papadako) wrote :

@md_5
A few days ago I had exactly the same bug, i.e. netbeans locking randomly in loading modules. From yesterday, that I updated the packages of my 15.10 system, netbeans, netbeans installer and freemind get killed randomly. From the related bugs that I have reported it seems that XInitThreads is not called at all, which could possible explain (the locking and crashing problems)
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797595#60

By the way, have you installed the latest updates? Do you experience the same bug if you downgrade to openjdk-7?

Revision history for this message
md_5 (md-5) wrote :

@papadako if you will look closely at the stack traces present you will see that the issue you are linking is not related to this one. Your netbeans locking up probably is, use jstack to confirm this.
The issue is not present in OpenJDK 1.7.0_85, and I have not yet reproduced it in Oracle JDK 8_66

Revision history for this message
Papadakos Panagiotis (papadako) wrote :

@md_5
Ok. One more question though. Have you updated your system with the latest updates?

Revision history for this message
md_5 (md-5) wrote :

Please read:
> Since upgrading to Ubuntu 15.10 from Ubuntu 15.04, the various GUI

Revision history for this message
Papadakos Panagiotis (papadako) wrote :

Dear md_5, I mean if you have the latest updates (i.e. apt-get update, apt-get upgrade). I am using 15.10 from its beta version, upgrading the packages every few days. I used to have the locking problem, but after installing the latest updates now netbeans just crashes. If you don't have a problem taking the risk that your netbeans might be unusable, I would like to see if you will experience the same problem. For example what "cat /var/log/dpkg.log | grep "\ install\ " |tail -n 10" report?

Revision history for this message
Papadakos Panagiotis (papadako) wrote :

Ok, I just found my problem, and it is irrelevant to what is described here. A server I recently installed was killing every 1 minute
each java applications I had running.

By the way, have you tried what I suggested previously, i.e. editing
/etc/java-8-openjdk/accessibility.properties and commenting out the line assistive_technologies=org.GNOME.Accessibility.AtkWrapper?

Revision history for this message
md_5 (md-5) wrote :

Tagging Matthias onto this bug.

Haven't see any issues with removing AtkWrapper, but too early to tell.

Revision history for this message
Pauli (paniemin) wrote :
Download full text (5.6 KiB)

I noticed the hang in load_gtk when trying to run netbeans with openjdk-7. For me disabling accessible like in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798131. The hand seems to be X11 threading related based on other similar hangs reported in web.

My java backtrace:
"AWT-EventQueue-1" prio=10 tid=0x00007fc0dc163000 nid=0xb4e runnable [0x00007fc0d2b6f000]
   java.lang.Thread.State: RUNNABLE
 at sun.awt.UNIXToolkit.load_gtk(Native Method)
 at sun.awt.UNIXToolkit.loadGTK(UNIXToolkit.java:108)
 - locked <0x00000000d12681f0> (a java.lang.Object)
 at com.sun.java.swing.plaf.gtk.GTKLookAndFeel.initialize(GTKLookAndFeel.java:1452)
 at javax.swing.UIManager.setLookAndFeel(UIManager.java:535)
 at org.netbeans.swing.plaf.Startup.initialize(Unknown Source)
 at org.netbeans.swing.plaf.Startup.<init>(Unknown Source)
 at org.netbeans.swing.plaf.Startup.run(Unknown Source)
 at org.netbeans.core.CoreBridgeImpl$1$1.run(Unknown Source)
 at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:312)
 at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:745)
 at java.awt.EventQueue.access$300(EventQueue.java:103)
 at java.awt.EventQueue$3.run(EventQueue.java:706)
 at java.awt.EventQueue$3.run(EventQueue.java:704)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:77)
 at java.awt.EventQueue.dispatchEvent(EventQueue.java:715)
 at org.GNOME.Accessibility.AtkWrapper$5.dispatchEvent(AtkWrapper.java:697)
 at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
 at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
 at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
 at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
 at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)

My native backtrace for same thread:
Thread 8 (Thread 0x7fc0d2b71700 (LWP 2894)):
#0 0x00007fc1399d98dd in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007fc1231dbbd2 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#2 0x00007fc1231dd4df in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#3 0x00007fc1231dd5f1 in xcb_wait_for_reply () from /usr/lib/x86_64-linux-gnu/libxcb.so.1
#4 0x00007fc123d03a47 in _XReply () from /usr/lib/x86_64-linux-gnu/libX11.so.6
#5 0x00007fc123cf99e6 in XQueryExtension () from /usr/lib/x86_64-linux-gnu/libX11.so.6
#6 0x00007fc123ced4f2 in XInitExtension () from /usr/lib/x86_64-linux-gnu/libX11.so.6
#7 0x00007fc128269fdf in XextAddDisplay () from /usr/lib/x86_64-linux-gnu/libXext.so.6
#8 0x00007fc1238b6f63 in ?? () from /usr/lib/x86_64-linux-gnu/libXi.so.6
#9 0x00007fc1238b59a9 in XListInputDevices () from /usr/lib/x86_64-linux-gnu/libXi.so.6
#10 0x00007fc119ddbf99 in ?? () from /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0
#11 0x00007fc119db76a0 in gdk_display_open () from /usr/lib/x86_64-linux-gnu/libgdk-x11-2.0.so.0
#12 0x00007fc119d8704f in gdk_display_open_default_libgtk_only ()
   from /usr/lib/x86_64-linux-gnu/libgdk...

Read more...

Revision history for this message
Pauli (paniemin) wrote :

I managed to find a very similar sounding bug that is from year 2012 but it was iced-tea specific: http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=857

Solution there was to make sure that setLookAndFeel call happens before any other call to swing. But I have no idea how that could be related to this bug. Only similarity is that atk-wrapper seems to trigger bugs in java UI code. Possible completely different issue.

Redhat bug report without any solution from year 2014: https://bugzilla.redhat.com/show_bug.cgi?id=1073984

Debian bug without solution from year 2013: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705511

Revision history for this message
md_5 (md-5) wrote :

atk-wrapper went through some quite big updates between 15.04 and 15.10, so I'd be betting on that being the issue (or it somehow triggering a Java edge case)

Revision history for this message
Alessandro Polverini (polve) wrote :

I confirm the problems with openJDK 8 (at least with current version 8u66-b17-1), I'm unable to even run the netbeans installer, while using Oracle JDK it works fine.

If I install netbeans with Oracle JDK and then run it with OpenJDK 8 the experience is VERY bad, with very long pauses here and there, making it almost unusable.

It used to work fine in Ubuntu 15.04

Revision history for this message
Pauli (paniemin) wrote :

"atk-wrapper went through some quite big updates between 15.04 and 15.10, so I'd be betting on that being the issue (or it somehow triggering a Java edge case)"

I don't think that is the primary bug. Bugs in Debian and Redhat are much older and look like perfect match to this bug. It feels like a race condition somewhere but I don't know how to debug jvm. Maybe some kind of trace from xcb telling which threads are using which connection would help.

Revision history for this message
Pauli (paniemin) wrote :

src/solaris/native/sun/awt/awt_UNIXToolkit.c:
    AWT_LOCK();
    XSync(awt_display, False);
    AWT_UNLOCK();

Looks like X11 is supposed to be protected by AWT_LOCK.

src/solaris/classes/sun/awt/UNIXToolkit.java:
    /** All calls into GTK should be synchronized on this lock */
    public static final Object GTK_LOCK = new Object();

It looks like also gtk is protected with a java lock.

Mean while looking at java-atk-wrapper code it is using gtk/atk/gdk without unix toolkit lock.

But I simple can't remember any specific reason to use locks if opening two separate X display for two threads.

Also I can't understand how gdk_open_display can hit a race condition ...

Revision history for this message
Franck (alci) wrote :

I confirm the following workaround is working: comment out the line "assistive_technologies=org.GNOME.Accessibility.AtkWrapper" in /etc/java-7-openjdk/accessibility.properties

I had Freeplane systematically hanging when browsing or even saving a file, and the works around the problem perfectly. But this should definitely be fixed as many Java apps seems to be affected....

Revision history for this message
Daniel (daniel-nuest) wrote :

I can confirm the workaround described in #24 (https://bugs.launchpad.net/ubuntu/+source/openjdk-8/+bug/1510009/comments/24) as well for Ubuntu 15.10. and Netbeans 8.1 with OpenJDK Runtime Environment (build 1.8.0_66-internal-b17).

Revision history for this message
Don-vip (vincent-privat) wrote :

GNOME Atk wrapper is unmaintained, it must be disabled in Debian openjdk packaging asap.
See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813143

Changed in josm:
importance: Unknown → Critical
Changed in openjdk-8 (Debian):
status: Unknown → New
Revision history for this message
Joachim Durchholz (jo-durchholz) wrote :

For me, I could reproduce the hang by doing this in a fresh-ish install:
1) Start Netbeans
2) Tools - Options - Fonts & Colors
3) With Profile at NetBeans and the Syntax tab selected, click on the ellipsis button to the right of the F_ont field.
4) Netbeans hangs.

I can confirm that commenting out AtkWrapper in /etc/java-7-openjdk/accessibility.properties fixed the hang.

Changed in openjdk-8 (Debian):
status: New → Fix Released
Revision history for this message
Matthias Klose (doko) wrote :

please recheck with 16.04

Changed in openjdk-8 (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Alan Ferguson (alan-ferguson) wrote :

Joachim Durchholz (jo-durchholz) offered a fix to the locking problem (entry #27). I had the same problem. It took several tries to get netBeans 8.1 started (using java-7-openjdk), and then it would lock occasionally when it was started (at the ...Starting Modules). It was finally working reasonably well when I tried to get the red, vertical line removed. After that, it locked up before the column for the red line could be changed, and did not load after that at all, after 4 or 5 tries.

I commented out the line in the accessibility.properties file, and NetBeans 8.1 has loaded every time since that fix (only 3 or 4 times so far). A benefit was that I was also able to change te excessive column to line 200, and the text box in which the column number was contained was now wide enough to see all 3 digits (200).

I am just starting to try NetBeans now, so I don't consider myself an expert on this development system.

Revision history for this message
liamdawe (liamdawe) wrote :

I believe this is the issue I am also coming across trying to play games that use Java on Ubuntu 16.04, I consider it a major issue since gaming is driving us a lot right now.

Here's an example when trying to launch Retro-Pixel Castles:
Exception in thread "main" java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.UnsatisfiedLinkError: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext/libatk-wrapper.so: /usr/lib/x86_64-linux-gnu/libatspi.so.0: undefined symbol: g_type_class_adjust_private_offset
 at java.lang.ClassLoader$NativeLibrary.load(Native Method)
 at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1938)
 at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1834)
 at java.lang.Runtime.loadLibrary0(Runtime.java:870)
 at java.lang.System.loadLibrary(System.java:1122)
 at org.GNOME.Accessibility.AtkWrapper.<clinit>(AtkWrapper.java:39)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at java.lang.Class.newInstance(Class.java:442)
 at java.awt.Toolkit.loadAssistiveTechnologies(Toolkit.java:805)
 at java.awt.Toolkit.getDefaultToolkit(Toolkit.java:886)
 at org.lwjgl.LinuxSysImplementation.<clinit>(LinuxSysImplementation.java:50)
 at org.lwjgl.Sys.createImplementation(Sys.java:131)
 at org.lwjgl.Sys.<clinit>(Sys.java:116)
 at org.lwjgl.opengl.Display.<clinit>(Display.java:135)
 at rpc.system.ScaleControl.setWorldScale(ScaleControl.java:56)
 at rpc.system.ScaleControl.initDisplay(ScaleControl.java:17)
 at rpc.system.Game.launchGame(Game.java:73)
 at rpc.system.Launcher.main(Launcher.java:47)
 ... 5 more
Game removed: AppID 328080 "Retro-Pixel Castles", ProcID 6188

Revision history for this message
liamdawe (liamdawe) wrote :

About my last comment, it's only an issue in OpenJDK and not Oracle Java.

Revision history for this message
Matthias Klose (doko) wrote :

now turned off by default in 16.04

Changed in openjdk-8 (Ubuntu):
status: Incomplete → Fix Released
Mathew Hodson (mhodson)
affects: openjdk-8 (Debian) → java-atk-wrapper (Debian)
affects: openjdk-8 (Ubuntu) → java-atk-wrapper (Ubuntu)
Changed in java-atk-wrapper (Ubuntu):
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.