It depends on the GC your java is using. Parallel GCs might work better on larger memory settings - I'm no expert on that though.
In general, if you have larger memory the less frequent it needs to be GC-ed - there is lots of room for garbage. However, when it comes to a GC, the GC has to work on more memory - which in turn might be slower.