You need to modify the jenkins.xml file. Specifically you need to change
<arguments>-Xrs -Xmx256m
-Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle
-jar "%BASE%\jenkins.war" --httpPort=8080</arguments>
to
<arguments>-Xrs -Xmx2048m -XX:MaxPermSize=512m
-Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle
-jar "%BASE%\jenkins.war" --httpPort=8080</arguments>
You can also verify the Java options that Jenkins is using by installing the Jenkins monitor plugin via Manage Jenkins / Manage Plugins and then navigating to Managing Jenkins / Monitoring of Hudson / Jenkins master to use monitoring to determine how much memory is available to Jenkins.
If you are getting an out of memory error when Jenkins calls Maven, it may be necessary to set MAVEN_OPTS via Manage Jenkins / Configure System e.g. if you are running on a version of Java prior to JDK 1.8 (the values are suggestions):
-Xmx2048m -XX:MaxPermSize=512m
If you are using JDK 1.8:
-Xmx2048m