java applet crashes x server

[Matthias]

Binary package hint: xorg

The attached (unsigned) java applet, if started in an xnest X server, reliably crashes that xnest server, at least for me. When started on the "real" X server, it caused severe display corruption and X basically froze. The system stayed somewhat responsive (the mouse freezed for a couple of seconds, but sporadically it moved again). Switching consoles worked, but only during the times when the mouse cursor was responsive. Restarting X fixed the problem the first time I tried, but the second time a full reboot (complete with unplugging the power cable) was required to get the system back. X apparently didn't produce any core dump, probably because it stayed active enough to be terminated normally.

How to reproduce: Run the applet. Firefox worked for me, and so did Sun's applet viewer. The applet was generated by Processing, and running the sketch in Processing also worked. Sometimes it's necessary to create a new window (an xterm will do, or any right mouse button popup, including the window manager's popups). If you opened it in an Xnest instance, that Xnest should close and spit out this error:

X Error of failed request: BadDrawable (invalid Pixmap or Window parameter)
  Major opcode of failed request: 70 (X_PolyFillRectangle)
  Resource id in failed request: 0x0
  Serial number of failed request: 3646
  Current serial number in output stream: 3647

If you ran the applet on the real X server, you should see display corruption. The Xorg.0.log.old that ubuntu-bug hopefully attached was generated by the "crashing" server.

This looks a lot like memory corruption (note that the applet renders the "foobar" string at position (0,0), that is, above the top border of its window because the coordinates specify the bottom-left corner of the text), and a java applet shouldn't be able to crash the X server anyway, so I'm tagging this as a security problem.

My system has compiz running, and some of the corruption was clearly compiz-related (corrupted shadows and such), but Xnest crashes with nothing but the applet viewer running in it, no window manager or anything. I'll probably try this on different hardware and see how reliably it works (a friend also had display corruption resulting from Processing).

ProblemType: Bug
Architecture: i386
DistroRelease: Ubuntu 9.04
NonfreeKernelModules: nvidia
Package: xorg 1:7.4~5ubuntu18
ProcEnviron:
 PATH=(custom, user)
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
ProcVersion: Linux version 2.6.28-13-generic (buildd@vernadsky) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #45-Ubuntu SMP Tue Jun 30 19:49:51 UTC 2009
SourcePackage: xorg
Uname: Linux 2.6.28-13-generic i686

[lspci]
00:00.0 Host bridge [0600]: Intel Corporation Mobile 4 Series Chipset Memory Controller Hub [8086:2a40] (rev 07)
     Subsystem: Samsung Electronics Co Ltd Device [144d:c042]
01:00.0 VGA compatible controller [0300]: nVidia Corporation GeForce 9200M GS [10de:06e8] (rev a1)
     Subsystem: Samsung Electronics Co Ltd Device [144d:c042]

[Kees Cook (kees)]

Can you attach the crash report for the Xnest process? That would give a better sense of what is causing the crash. Also, is it the font that is causing the problem? Which line, if changed/removed, from the applet causes the crashing to stop?

Thanks for the report!

[Matthias (terrormafia)]

I did some further testing with the applet on a couple of different systems:
- Nothing happens without compiz / desktop effects enabled. Even with compiz enabled, it doesn't crash on every system, but on systems where it does crash, it crashes reliably.
- On ASUS eeePC (Intel onboard graphics), X crashes immediatelly. No screen corruption, the server just terminates with signal 11. I'll probably create a proper backtrace from this tomorrow.

Looks like it's a compiz bug, though it's the X server which crashes, not compiz.

[Matthias (terrormafia)]

Ok, the minimal script that I got to crash reliably is:

PFont font;

void setup() {
 font = loadFont("arial.vlw");
}

void draw() {}

It's possible to get it to crash Xnest without a draw() method, though then it only crashes if Xnest ist minimized when the appletviewer starts. Adding an empty draw method makes it crash if any new window is created. The font doesn't seem to make a difference, but some font must be loaded and assigned to "font" (probably so the loading doesn't get optimized away, as there's no need to actually use the font). It can be any size, aliased or not, all characters included or not.

In order to crash the eee, the following line had to be added before the loadFont line:
size(1000, 1000);

Xnest doesn't really crash, it calls exit() after detecting an X error, so apport does not react, and what ubuntu-bug creates only contains generic information about the package. I'll attach a backtrace generated by setting a breakpoint on exit(). The gdb command was:
gdb --args Xnest -sync :2 2>&1 | tee gdb-xnest.txt

[Matthias (terrormafia)]

I also generated a backtrace for the crash on the eeePC, where the real X server gets a SIGSEGV immediatelly after the applet is fully loaded and starts drawing.

By the way, both backtraces were generated from the script with the size() line included, that is, by:
PFont font;

void setup() {
        size(1000, 1000);
 font = loadFont("wasy.vlw");
}

void draw() {
}

[Kees Cook (kees)]

In the gdb crash, can you also include the output of:
 x/16i $pc
 info registers

(I've added the former to https://wiki.ubuntu.com/Backtrace now)

[Matthias (terrormafia)]

Here you are.

[Kees Cook (kees)]

Thanks for getting that caught. This looks like a directly shut-down, rather than an overflow or other uncontrolled failure. As such, I'm going to unmark it as security+private so that more developers will have a chance to look at this issue. Thanks!

[Launchpad Janitor (janitor)]

[Expired for nvidia-graphics-drivers-180 (Ubuntu) because there has been no activity for 60 days.]