Running Selenium Webdriver with a proxy in Python

Question

I am trying to run a Selenium Webdriver script in Python to do some basic tasks  I can get the robot to function perfectly when running it through the Selenium IDE inteface  ie  when simply getting the GUI to repeat my actions   However when I export the code as a Python script and try to execute it from the command line  the Firefox browser will open but cannot ever access the starting URL  an error is returned to command line and the program stops   This is happening me regardless of what website etc I am trying to access   I have included a very basic code here for demonstration purposes  I don t think that I have included the proxy section of the code correctly as the error being returned seems to be generated by the proxy   Any help would be hugely appreciated    The below code is simply meant to open www google ie and search for the word  selenium   For me it opens a blank firefox browser and stops   from selenium import webdriver from selenium webdriver common by import By from selenium webdriver support ui import Select from selenium common exceptions import NoSuchElementException import unittest  time  re from selenium webdriver common proxy import    class Testrobot2 unittest TestCase       def setUp self            myProxy    http   149 215 113 110 70           proxy   Proxy            proxyType   ProxyType MANUAL           httpProxy   myProxy           ftpProxy   myProxy           sslProxy   myProxy           noProxy                self driver   webdriver Firefox proxy proxy          self driver implicitly wait 30          self base url    https   www google ie           self verificationErrors              self accept next alert   True      def test robot2 self           driver   self driver         driver get self base url      gs rn 17 amp gs ri psy-ab amp suggest p amp cp 6 amp gs id ix amp xhr t amp q selenium amp es nrs true amp pf p amp output search amp sclient psy-ab amp oq seleni amp gs l  amp pbx 1 amp bav on 2 or r qf  amp bvm bv 47883778 d ZGU amp fp 7c0d9024de9ac6ab amp biw 592 amp bih 665           driver find element by id  gbqfq   clear           driver find element by id  gbqfq   send keys  selenium        def is element present self  how  what           try  self driver find element by how  value what          except NoSuchElementException  e  return False         return True      def is alert present self           try  self driver switch to alert           except NoAlertPresentException  e  return False         return True      def close alert and get its text self           try              alert   self driver switch to alert               alert text   alert text             if self accept next alert                  alert accept               else                  alert dismiss               return alert text         finally  self accept next alert   True      def tearDown self           self driver quit           self assertEqual     self verificationErrors   if   name         main         unittest main

User · Answer

The answers above and on this question either didn't work for me with Selenium 3.14 and Firefox 68.9 on Linux, or are unnecessarily complex. I needed to use a WPAD configuration, sometimes behind a proxy (on a VPN), and sometimes not. After studying the code a bit, I came up with:

from selenium import webdriver
from selenium.webdriver.common.proxy import Proxy
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile

proxy = Proxy({'proxyAutoconfigUrl': 'http://wpad/wpad.dat'})
profile = FirefoxProfile()
profile.set_proxy(proxy)
driver = webdriver.Firefox(firefox_profile=profile)

The Proxy initialization sets proxyType to ProxyType.PAC (autoconfiguration from a URL) as a side-effect.

It also worked with Firefox's autodetect, using:

from selenium.webdriver.common.proxy import ProxyType

proxy = Proxy({'proxyType': ProxyType.AUTODETECT})

But I don't think this would work with both internal URLs (not proxied) and external (proxied) the way WPAD does. Similar proxy settings should work for manual configuration as well. The possible proxy settings can be seen in the code here.

Note that directly passing the Proxy object as proxy=proxy to the driver does NOT work--it's accepted but ignored (there should be a deprecation warning, but in my case I think Behave is swallowing it).

User · Answer

If anyone is looking for a solution here s how    from selenium import webdriver PROXY    YOUR PROXY ADDRESS HERE  webdriver DesiredCapabilities FIREFOX  proxy          httpProxy  PROXY       ftpProxy  PROXY       sslProxy  PROXY       noProxy  None       proxyType   MANUAL        autodetect  False   driver   webdriver Firefox   driver get  http   www whatsmyip org

User · Answer

My solution   def my proxy PROXY HOST PROXY PORT           fp   webdriver FirefoxProfile             Direct   0  Manual   1  PAC   2  AUTODETECT   4  SYSTEM   5         print PROXY PORT         print PROXY HOST         fp set preference  network proxy type   1          fp set preference  network proxy http  PROXY HOST          fp set preference  network proxy http port  int PROXY PORT           fp set preference  general useragent override   whater useragent           fp update preferences           return webdriver Firefox firefox profile fp    Then call in your code   my proxy PROXY HOST PROXY PORT    I had issues with this code because I was passing a string as a port      PROXY PORT  31280    This is important   int  31280     You must pass an integer instead of a string or your firefox profile will not be set to a properly port and connection through proxy will not work

User · Answer

How about something like this   PROXY    149 215 113 110 70   webdriver DesiredCapabilities FIREFOX  proxy            httpProxy  PROXY       ftpProxy  PROXY       sslProxy  PROXY       noProxy  None       proxyType   MANUAL        class   org openqa selenium Proxy        autodetect  False      you have to use remote  otherwise you ll have to code it yourself in python to  driver   webdriver Remote  http   localhost 4444 wd hub   webdriver DesiredCapabilities FIREFOX    You can read more about it here

User · Answer

As stated by  Dugini  some config entries have been removed  Maximal   webdriver DesiredCapabilities FIREFOX  proxy            httpProxy  PROXY       ftpProxy  PROXY       sslProxy  PROXY       noProxy           proxyType   MANUAL

User · Answer

This worked for me and allow to use an headless browser   you just need to call the method passing your proxy  def setProxy proxy           options   Options           options headless   True          options add argument  quot --window-size 1920 1200 quot           options add argument  quot --disable-dev-shm-usage quot           options add argument  quot --no-sandbox quot           prox   Proxy           prox proxy type   ProxyType MANUAL         prox http proxy   proxy         prox ssl proxy   proxy         capabilities   webdriver DesiredCapabilities CHROME         prox add to capabilities capabilities          return webdriver Chrome desired capabilities capabilities  options options  executable path DRIVER PATH

User · Answer

Proxy with verification  This is a whole new python script in reference from a  Mykhail Martsyniuk sample script     Load webdriver from selenium import webdriver    Load proxy option from selenium webdriver common proxy import Proxy  ProxyType    Configure Proxy Option prox   Proxy   prox proxy type   ProxyType MANUAL    Proxy IP  amp  Port prox http proxy      0 0 0 0 00000    prox socks proxy      0 0 0 0 00000    prox ssl proxy      0 0 0 0 00000       Configure capabilities  capabilities   webdriver DesiredCapabilities CHROME prox add to capabilities capabilities     Configure ChromeOptions driver   webdriver Chrome executable path   usr local share chromedriver  desired capabilities capabilities     Verify proxy ip driver get  http   www whatsmyip org

User · Answer

Try setting up sock5 proxy too  I was facing the same problem and it is solved by using the socks proxy  def install proxy PROXY HOST PROXY PORT           fp   webdriver FirefoxProfile           print PROXY PORT         print PROXY HOST         fp set preference  network proxy type   1          fp set preference  network proxy http  PROXY HOST          fp set preference  network proxy http port  int PROXY PORT           fp set preference  network proxy https  PROXY HOST          fp set preference  network proxy https port  int PROXY PORT           fp set preference  network proxy ssl  PROXY HOST          fp set preference  network proxy ssl port  int PROXY PORT             fp set preference  network proxy ftp  PROXY HOST          fp set preference  network proxy ftp port  int PROXY PORT              fp set preference  network proxy socks  PROXY HOST          fp set preference  network proxy socks port  int PROXY PORT              fp set preference  general useragent override   Mozilla 5 0  Macintosh  Intel Mac OS X 10 9 3  AppleWebKit 537 75 14  KHTML  like Gecko  Version 7 0 3 Safari 7046A194A           fp update preferences           return webdriver Firefox firefox profile fp    Then call  install proxy   ip   port    from your program

User · Answer

The result stated above may be correct  but isn t working with the latest webdriver   Here is my solution for the above question  Simple and sweet           http proxy     ip addr port          https proxy    ip addr port           webdriver DesiredCapabilities FIREFOX  proxy                  httpProxy  http proxy               sslProxy  https proxy               proxyType   MANUAL                     driver   webdriver Firefox     OR      http proxy     http   ip port      https proxy    https   ip port       proxyDict                          http     http proxy                       https    https proxy                         driver   webdriver Firefox proxy proxyDict

User · Answer

Works for me this way  similar to  Amey and  user4642224 code  but shorter a bit    from selenium import webdriver from selenium webdriver common proxy import Proxy  ProxyType  prox   Proxy   prox proxy type   ProxyType MANUAL prox http proxy    ip addr port  prox socks proxy    ip addr port  prox ssl proxy    ip addr port   capabilities   webdriver DesiredCapabilities CHROME prox add to capabilities capabilities   driver   webdriver Chrome desired capabilities capabilities

User · Answer

Try by Setting up FirefoxProfile  from selenium import webdriver import time    Define Both ProxyHost and ProxyPort as String  ProxyHost    54 84 95 51   ProxyPort    8083     def ChangeProxy ProxyHost  ProxyPort        Define Firefox Profile with you ProxyHost and ProxyPort      profile   webdriver FirefoxProfile       profile set preference  network proxy type   1      profile set preference  network proxy http   ProxyHost       profile set preference  network proxy http port   int ProxyPort       profile update preferences       return webdriver Firefox firefox profile profile    def FixProxy          Reset Firefox Profile       profile   webdriver FirefoxProfile       profile set preference  network proxy type   0      return webdriver Firefox firefox profile profile    driver   ChangeProxy ProxyHost  ProxyPort  driver get  http   whatismyipaddress com    time sleep 5   driver   FixProxy   driver get  http   whatismyipaddress com     This program tested on both Windows 8 and Mac OSX  If you are using Mac OSX and if you don t have selenium updated then you may face selenium common exceptions WebDriverException  If so  then try again after upgrading your selenium  pip install -U selenium

User · Answer

try running tor service  add the following function to your code   def connect tor port    socks set default proxy socks PROXY TYPE SOCKS5   127 0 0 1   port  True  socket socket   socks socksocket  def main     connect tor   driver   webdriver Firefox

[python] Running Selenium Webdriver with a proxy in Python

Examples related to python

Examples related to selenium

Examples related to proxy

Examples related to selenium-webdriver

Examples related to selenium-ide