Yes, there is a maximum, but it's system dependent. Try it and see, doubling until you hit a limit then searching down. At least with Sun JRE 1.6 on linux you get interesting if not always informative error messages (peregrino is netbook running 32 bit ubuntu with 2G RAM and no swap):
peregrino:$ java -Xmx4096M -cp bin WheelPrimes
Invalid maximum heap size: -Xmx4096M
The specified size exceeds the maximum representable size.
Could not create the Java virtual machine.
peregrino:$ java -Xmx4095M -cp bin WheelPrimes
Error occurred during initialization of VM
Incompatible minimum and maximum heap sizes specified
peregrino:$ java -Xmx4092M -cp bin WheelPrimes
Error occurred during initialization of VM
The size of the object heap + VM data exceeds the maximum representable size
peregrino:$ java -Xmx4000M -cp bin WheelPrimes
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.
(experiment reducing from 4000M until)
peregrino:$ java -Xmx2686M -cp bin WheelPrimes
(normal execution)
Most are self explanatory, except -Xmx4095M which is rather odd (maybe a signed/unsigned comparison?), and that it claims to reserve 2686M on a 2GB machine with no swap. But it does hint that the maximum size is 4G not 2G for a 32 bit VM, if the OS allows you to address that much.