Proxy with urllib2

Question

I open urls with   site   urllib2 urlopen  http   google com    And what I want to do is connect the same way with a proxy I got somewhere telling me   site   urllib2 urlopen  http   google com   proxies   http   127 0 0 1     but that didn t work either   I know urllib2 has something like a proxy handler  but I can t recall that function

User · Answer

To use the default system proxies  e g  from the http support environment variable   the following works for the current request  without installing it into urllib2 globally    url    http   www example com   proxy   urllib2 ProxyHandler   opener   urllib2 build opener proxy  in    opener open url  in  read

User · Answer

One can also use requests if we would like to access a web page using proxies  Python 3 code     gt  gt  gt  import requests  gt  gt  gt  url    http   www google com   gt  gt  gt  proxy    169 50 87 252 80   gt  gt  gt  requests get url  proxies   http  proxy    lt Response  200  gt    More than one proxies can also be added    gt  gt  gt  proxy1    169 50 87 252 80   gt  gt  gt  proxy2    89 34 97 132 8080   gt  gt  gt  requests get url  proxies   http  proxy1  http  proxy2    lt Response  200  gt

User · Answer

In Addition to the accepted answer  My scipt gave me an error   File  c  Python23 lib urllib2 py   line 580  in proxy open     if     in host  TypeError  iterable argument required   Solution was to add http    in front of the proxy string   proxy   urllib2 ProxyHandler   http    http   proxy xy z 8080    opener   urllib2 build opener proxy  urllib2 install opener opener  urllib2 urlopen  http   www google com

User · Answer

proxy   urllib2 ProxyHandler   http    127 0 0 1    opener   urllib2 build opener proxy  urllib2 install opener opener  urllib2 urlopen  http   www google com

User · Answer

You can set proxies using environment variables   import os os environ  http proxy      127 0 0 1  os environ  https proxy      127 0 0 1    urllib2 will add proxy handlers automatically this way  You need to set proxies for different protocols separately otherwise they will fail  in terms of not going through proxy   see below   For example   proxy   urllib2 ProxyHandler   http    127 0 0 1    opener   urllib2 build opener proxy  urllib2 install opener opener  urllib2 urlopen  http   www google com     next line will fail  will not go through the proxy   https  urllib2 urlopen  https   www google com     Instead   proxy   urllib2 ProxyHandler        http    127 0 0 1        https    127 0 0 1     opener   urllib2 build opener proxy  urllib2 install opener opener    this way both http and https requests go through the proxy urllib2 urlopen  http   www google com   urllib2 urlopen  https   www google com

User · Answer

In addition set the proxy for the command line session Open a command line where you might want to run your script  netsh winhttp set proxy YourProxySERVER yourProxyPORT   run your script in that terminal

User · Answer

You have to install a ProxyHandler  urllib2 install opener      urllib2 build opener          urllib2 ProxyHandler   http    127 0 0 1            urllib2 urlopen  http   www google com

[python] Proxy with urllib2

Examples related to python

Examples related to proxy

Examples related to urllib2