You need to look at upgrading your OS and Java. Java 5.0 is EOL but if you cannot update to Java 6, you could use the latest patch level 22!
32-bit Windows is limited to ~ 1.3 GB so you are doing well to se the maximum to 1.8. Note: this is a problem with continous memory, and as your system runs its memory space can get fragmented so it does not suprise me you have this problem.
A 64-bit OS, doesn't have this problem as it has much more virtual space, you don't even have to upgrade to a 64-bit version of java to take advantage of this.
BTW, in my experience, 32-bit Java 5.0 can be faster than 64-bit Java 5.0. It wasn't until many years later that Java 6 update 10 was faster for 64-bit.