Above answers are nice and great, but not clear for new git users like me. So after some investigation, i offer this new answer.
When the config file does not exists, you can create one. Besides port
the config file can include other ssh config option:user
IdentityFile
and so on, the config file looks like
Host mydomain.com
User git
Port 12345
If you are running linux, take care the config file must have strict permission: read/write for the user, and not accessible by others
It's cool, the only thing we should know is that there two syntaxes for ssh url in git
ssh://[user@]host.xz[:port]/path/to/repo.git/
[user@]host.xz:path/to/repo.git/
By default Gitlab and Github will show the scp like syntax url, and we can not give the custom ssh port. So in order to change ssh port, we need use the standard syntax