Using an up to date (as of friday) Kubuntu Dapper and the latest JDK from Sun, I get a large memory leak with Java applications that causes Xorg to consume more and more memory until processes start to crash. xrestop confirms that the Java app's window is the one consuming ever-increasing amounts of memory. The application that caused me to notice this is proprietary and cannot be shared, however I have noticed that it is also easy to see this happening in Azureus, which consumes approximately 500k of additional memory per second.
While the official JDK is not supported but Kubuntu, I don't think this is a bug in either the JDK or the Java apps themselves, due to the fact that Linux users can run both my proprietary Java app and other apps like Azureus for more than a few hours on other distros.
I've delt with Java memory leaks before and I have not seen them spill into the underlying OS, however it still may be possible. The JVM is usually confined to a specific heapsize and cannot grow past it. It sounds like your propriatary program has a memory leak in it. Using a tool like JProfiler should help you find it. As for Azureus as it may seem like the memory of it is going up and down I bet it stays within its heap. Anything is possible though.
Can you give detailed specs of JVM and steps to get Azureus to crash/consume large amounts of memory on Dapper? Or perhaps some sample java program that demonstrates this? Are you getting exceptions?