Proxies with Python Requests module

Question

Just a short  simple one about the excellent Requests module for Python   I can t seem to find in the documentation what the variable  proxies  should contain  When I send it a dict with a standard  IP PORT  value it rejected it asking for 2 values  So  I guess  because this doesn t seem to be covered in the docs  that the first value is the ip and the second the port    The docs mention this only      proxies      optional  Dictionary mapping protocol to the URL of the proxy    So I tried this    what should I be doing    proxy     ip  port    and should I convert these to some type before putting them in the dict    r   requests get url headers headers proxies proxy

User · Answer

If you d like to persisist cookies and session data  you d best do it like this   import requests  proxies          http    http   user pass 10 10 1 0 3128        https    https   user pass 10 10 1 0 3128        Create the session and set the proxies  s   requests Session   s proxies   proxies    Make the HTTP request through the session  r   s get  http   www showmemyip com

User · Answer

I have found that urllib has some really good code to pick up the system s proxy settings and they happen to be in the correct form to use directly   You can use this like   import urllib      r   requests get  http   example org   proxies urllib request getproxies      It works really well and urllib knows about getting Mac OS X and Windows settings as well

User · Answer

It   s a bit late but here is a wrapper class that simplifies scraping proxies and then making an http POST or GET   ProxyRequests  https   github com rootVIII proxy requests

User · Answer

The accepted answer was a good start for me  but I kept getting the following error   AssertionError  Not supported proxy scheme None   Fix to this was to specify the http    in the proxy url thus    http proxy     http   194 62 145 248 8080  https proxy     https   194 62 145 248 8080  ftp proxy      10 10 1 10 3128   proxyDict                    http     http proxy                 https    https proxy                 ftp      ftp proxy                 I d be interested as to why the original works for some people but not me    Edit  I see the main answer is now updated to reflect this

User · Answer

here is my basic class in python for the requests module with some proxy configs and stopwatch    import requests import time class BaseCheck        def   init   self  url           self http proxy     http   user pw proxy 8080          self https proxy    http   user pw proxy 8080          self ftp proxy      http   user pw proxy 8080          self proxyDict                            http     self http proxy                         https    self https proxy                         ftp      self ftp proxy                               self url   url         def makearr tsteps               global stemps             global steps             stemps                  for step in tsteps                  stemps step       start   0   end   0               steps   tsteps         makearr   init   check            def starttime typ                    for stemp in stemps                  if typ                            stemps stemp   start     time time                   else                      stemps stemp  typ    time time           starttime       def   str   self           return str self url      def getrequests self           g requests get self url proxies self proxyDict          print g status code         print g content         print self url         stemps  init    end     time time            print stemps  init    end   - stemps  init    start           x  stemps  init    end   - stemps  init    start           print x   test BaseCheck url  http   google com   test getrequests

User · Answer

8 years late  But I like   import os import requests  os environ  HTTP PROXY     os environ  http proxy      http   http-connect-proxy 3128   os environ  HTTPS PROXY     os environ  https proxy      http   http-connect-proxy 3128   os environ  NO PROXY     os environ  no proxy      127 0 0 1 localhost  local   r   requests get  https   example com        verify False

User · Answer

I share some code how to fetch proxies from the site  quot https   free-proxy-list net quot  and store data to a file compatible with tools like  quot Elite Proxy Switcher quot  format IP PORT     PROXY UPDATER - get free proxies from https   free-proxy-list net  from lxml html import fromstring import requests from itertools import cycle import traceback import re                        FIND PROXIES                                          def get proxies        url    https   free-proxy-list net       response   requests get url      parser   fromstring response text      proxies   set       for i in parser xpath    tbody tr    299      299 proxies max         proxy    quot   quot  join  i xpath     td 1  text               0  i xpath     td 2  text     0            proxies add proxy      return proxies                          write to file in format   IP PORT                       try      proxies   get proxies       f open  proxy list txt   w       for proxy in proxies          f write proxy   n       f close       print   quot DONE quot   except      print   quot MAJOR ERROR quot

User · Answer

The proxies  dict syntax is   protocol   ip port          With it you can specify different  or the same  proxie s  for requests using http  https  and ftp protocols   http proxy     http   10 10 1 10 3128  https proxy    https   10 10 1 11 1080  ftp proxy      ftp   10 10 1 10 3128   proxyDict                     http     http proxy                  https    https proxy                  ftp      ftp proxy                r   requests get url  headers headers  proxies proxyDict    Deduced from the requests documentation      Parameters    method     method for the new Request object    url     URL for the new Request object          proxies      optional  Dictionary mapping protocol to the URL of the proxy              On linux you can also do this via the HTTP PROXY  HTTPS PROXY  and FTP PROXY environment variables   export HTTP PROXY 10 10 1 10 3128 export HTTPS PROXY 10 10 1 11 1080 export FTP PROXY 10 10 1 10 3128   On Windows   set http proxy 10 10 1 10 3128 set https proxy 10 10 1 11 1080 set ftp proxy 10 10 1 10 3128     Thanks  Jay for pointing this out  The syntax changed with requests 2 0 0  You ll need to add a schema to the url   https   2 python-requests org en latest user advanced  proxies

User · Answer

i just made a proxy graber and also can connect with same grabed proxy without any input  here is     Import Modules  from termcolor import colored from selenium import webdriver import requests import os import sys import time   Proxy Grab  options   webdriver ChromeOptions   options add argument  headless   driver   webdriver Chrome chrome options options  driver get  https   www sslproxies org    tbody   driver find element by tag name  tbody   cell   tbody find elements by tag name  tr   for column in cell           column   column text split              print colored column 0      column 1   yellow    driver quit   print      os system  clear   os system  cls     Proxy Connection  print colored  Getting Proxies from graber      green    time sleep 2  os system  clear   os system  cls   proxy     http    http      column 0      column 1   url    https   mobile facebook com login  r   requests get url   proxies proxy  print     print colored  Connecting using proxy    green    print     sts   r status code

User · Answer

You can refer to the proxy documentation here   If you need to use a proxy  you can configure individual requests with the proxies argument to any request method   import requests  proxies        http    http   10 10 1 10 3128      https    https   10 10 1 10 1080      requests get  http   example org   proxies proxies    To use HTTP Basic Auth with your proxy  use the http   user password host com  syntax   proxies          http    http   user pass 10 10 1 10 3128

[python] Proxies with Python 'Requests' module

Examples related to python

Examples related to http-request

Examples related to python-requests