It depends.
The web server on the other end will be set to a certain port, usually 80 and will only accept requests on that specific port. Something along the chain will need to be talking to port 80 to the website.
If you control the website, then you can change the port, or get it to accept requests on multiple ports.
If the website is already talking on a different port, you can just use the colon syntax to reference another port (eg: http://server.com:1234 for port 1234).
If you want to use a different port on your client end, but you want to talk to port 80 at the web server end, you'll need to route traffic from port x to port 80. A common way to get this up and running is to use Port Fowarding. ssh can do this for you, see here for a Unix/technical overview or here if you're on Windows.
Hope that helps.