Way better idea to add that host or ip to the .ssh/config
file like so:
Host (a space separated list of made up aliases you want to use for the host)
User git
Hostname (ip or hostname of git server)
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_rsa_(the key you want for this repo)