allocates heap memory beyond RLIMIT_AS

[Kees Cook]

During default startup, openjdk-6 appears to allocate heap memory based on something different than getrlimit(RLIMIT_AS):

$ free
             total used free shared buffers cached
Mem: 8000320 7907296 93024 0 574804 4336192
-/+ buffers/cache: 2996300 5004020
Swap: 3903672 301564 3602108

$ bootstrap/jdk1.6.0/bin/java -version
java version "1.6.0_0"
OpenJDK 1.4 (6b14-0ubuntu15~kees1.fortify) Runtime Environment (build 1.6.0_0-b14)
OpenJDK 64-Bit Server VM (build 14.0-b08, mixed mode)

$ ulimit -v 1024000

$ bootstrap/jdk1.6.0/bin/java -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

$ strace -f -e trace=mmap bootstrap/jdk1.6.0/bin/java -version
...
[pid 1187] mmap(NULL, 2225078272, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
...

$ bootstrap/jdk1.6.0/bin/java -Xmx128m -version
java version "1.6.0_0"
OpenJDK 1.4 (6b14-0ubuntu15~kees1.fortify) Runtime Environment (build 1.6.0_0-b14)
OpenJDK 64-Bit Server VM (build 14.0-b08, mixed mode)