I found the smoothest way to achieve this was using Pageant as the SSH agent and plink.
You need to have a putty session configured for the hostname that is used in your remote.
You will also need plink.exe which can be downloaded from the same site as putty.
And you need Pageant running with your key loaded. I have a shortcut to pageant in my startup folder that loads my SSH key when I log in.
When you install git-scm you can then specify it to use tortoise/plink rather than OpenSSH.
The net effect is you can open git-bash whenever you like and push/pull without being challenged for passphrases.
Same applies with putty and WinSCP sessions when pageant has your key loaded. It makes life a hell of a lot easier (and secure).