If you want to limit memory for jvm (not the heap size ) ulimit -v
To get an idea of the difference between jvm and heap memory , take a look at this excellent article http://blogs.vmware.com/apps/2011/06/taking-a-closer-look-at-sizing-the-java-process.html