Using an HTTP PROXY - Python

Question

I familiar with the fact that I should set the HTTP RPOXY environment variable to the proxy address  Generally urllib works fine  the problem is dealing with urllib2   gt  gt  gt  urllib2 urlopen  quot http   www google com quot   read    returns urllib2 URLError   lt urlopen error  Errno 10061  No connection could be made because the target machine actively refused it gt   or urllib2 URLError   lt urlopen error  Errno 11004  getaddrinfo failed gt    Extra info  urllib urlopen       works fine  It is just urllib2 that is playing tricks    I tried  Fenikso answer but I m getting this error now  URLError   lt urlopen error  Errno 10060  A connection attempt failed because the  connected party did not properly respond after a period of time  or established connection failed because connected host has failed to respond gt          Any ideas

User · Answer

I recommend you just use the requests module.

It is much easier than the built in http clients: http://docs.python-requests.org/en/latest/index.html

Sample usage:

r = requests.get('http://www.thepage.com', proxies={"http":"http://myproxy:3129"})
thedata = r.content

User · Answer

You can do it even without the HTTP PROXY environment variable  Try this sample   import urllib2  proxy support   urllib2 ProxyHandler   http   http   61 233 25 166 80    opener   urllib2 build opener proxy support  urllib2 install opener opener   html   urllib2 urlopen  http   www google com   read   print html   In your case it really seems that the proxy server is refusing the connection     Something more to try   import urllib2   proxy    61 233 25 166 80  proxy    YOUR PROXY GOES HERE   proxies     http   http    s    proxy  url    http   www google com search q test  headers   User-agent     Mozilla 5 0    proxy support   urllib2 ProxyHandler proxies  opener   urllib2 build opener proxy support  urllib2 HTTPHandler debuglevel 1   urllib2 install opener opener   req   urllib2 Request url  None  headers  html   urllib2 urlopen req  read   print html     Edit 2014  This seems to be a popular question   answer  However today I would use third party requests module instead   For one request just do   import requests  r   requests get  http   www google com                     proxies   http    http   61 233 25 166 80    print r text    For multiple requests use Session object so you do not have to add proxies parameter in all your requests   import requests  s   requests Session   s proxies     http    http   61 233 25 166 80    r   s get  http   www google com   print r text

User · Answer

I encountered this on jython client  The server was only talking TLS and the client using SSL context    javax net ssl SSLContext getInstance  SSL     Once the client was to TLS  things started working

User · Answer

Python 3   import urllib request  htmlsource   urllib request FancyURLopener   http   http   127 0 0 1 8080    open url  read   decode  utf-8

User · Answer

Just wanted to mention  that you also may have to set the https proxy OS environment variable in case https URLs need to be accessed  In my case it was not obvious to me and I tried for hours to discover this   My use case  Win 7  jython-standalone-2 5 3 jar  setuptools installation via ez setup py

[python] Using an HTTP PROXY - Python

Examples related to python

Examples related to http

Examples related to proxy

Examples related to urllib2