More comment: In another computer with ATI+Xgl, Matlab 2007b works fine with compiz-fusion after I put "export AWT_TOOLKIT=MToolkit" in the beginning of matlab shell script. But the problem with Mathematica 6.0.1 remains. Both Mathematica 6.0.1 and Matlab 2007b use Java.
More comment: In another computer with ATI+Xgl, Matlab 2007b works fine with compiz-fusion after I put "export AWT_TOOLKIT= MToolkit" in the beginning of matlab shell script. But the problem with Mathematica 6.0.1 remains. Both Mathematica 6.0.1 and Matlab 2007b use Java.