The speed tradeoffs between various settings of -Xms and -Xmx depend on the application and system that you run your Java application on. It also depends on your JVM and other garbage collection parameters you use.
This question is 11 years old, and since then the effects of the JVM parameters on performance have become even harder to predict in advance. So you can try different values and see the effects on performance, or use a free tool like Optimizer Studio that will find the optimal JVM parameter values automatically.