Can a website detect when you are using Selenium with chromedriver

Question

I ve been testing out Selenium with Chromedriver and I noticed that some pages can detect that you re using Selenium even though there s no automation at all  Even when I m just browsing manually just using Chrome through Selenium and Xephyr I often get a page saying that suspicious activity was detected  I ve checked my user agent  and my browser fingerprint  and they are all exactly identical to the normal Chrome browser  When I browse to these sites in normal Chrome everything works fine  but the moment I use Selenium I m detected  In theory  chromedriver and Chrome should look literally exactly the same to any webserver  but somehow they can detect it  If you want some testcode try out this  from pyvirtualdisplay import Display from selenium import webdriver  display   Display visible 1  size  1600  902   display start   chrome options   webdriver ChromeOptions   chrome options add argument  --disable-extensions   chrome options add argument  --profile-directory Default   chrome options add argument  quot --incognito quot   chrome options add argument  quot --disable-plugins-discovery quot    chrome options add argument  quot --start-maximized quot   driver   webdriver Chrome chrome options chrome options  driver delete all cookies   driver set window size 800 800  driver set window position 0 0  print  arguments done  driver get  http   stubhub com    If you browse around stubhub you ll get redirected and  blocked  within one or two requests  I ve been investigating this and I can t figure out how they can tell that a user is using Selenium  How do they do it  I installed the Selenium IDE plugin in Firefox and I got banned when I went to stubhub com in the normal Firefox browser with only the additional plugin  When I use Fiddler to view the HTTP requests being sent back and forth I ve noticed that the  fake browser s  requests often have  no-cache  in the response header  Results like this Is there a way to detect that I  39 m in a Selenium Webdriver page from JavaScript suggest that there should be no way to detect when you are using a webdriver  But this evidence suggests otherwise  The site uploads a fingerprint to their servers  but I checked and the fingerprint of Selenium is identical to the fingerprint when using Chrome  This is one of the fingerprint payloads that they send to their servers    quot appName quot   quot Netscape quot   quot platform quot   quot Linuxx86 64 quot   quot cookies quot  1  quot syslang quot   quot en-US quot   quot userlang quot   quot en-US quot   quot cpu quot   quot  quot   quot productSub quot   quot 20030107 quot   quot setTimeout quot  1  quot setInterval quot  1  quot plugins quot    quot 0 quot   quot ChromePDFViewer quot   quot 1 quot   quot ShockwaveFlash quot   quot 2 quot   quot WidevineContentDecryptionModule quot   quot 3 quot   quot NativeClient quot   quot 4 quot   quot ChromePDFViewer quot    quot mimeTypes quot    quot 0 quot   quot application pdf quot   quot 1 quot   quot ShockwaveFlashapplication x-shockwave-flash quot   quot 2 quot   quot FutureSplashPlayerapplication futuresplash quot   quot 3 quot   quot WidevineContentDecryptionModuleapplication x-ppapi-widevine-cdm quot   quot 4 quot   quot NativeClientExecutableapplication x-nacl quot   quot 5 quot   quot PortableNativeClientExecutableapplication x-pnacl quot   quot 6 quot   quot PortableDocumentFormatapplication x-google-chrome-pdf quot    quot screen quot    quot width quot  1600  quot height quot  900  quot colorDepth quot  24   quot fonts quot    quot 0 quot   quot monospace quot   quot 1 quot   quot DejaVuSerif quot   quot 2 quot   quot Georgia quot   quot 3 quot   quot DejaVuSans quot   quot 4 quot   quot TrebuchetMS quot   quot 5 quot   quot Verdana quot   quot 6 quot   quot AndaleMono quot   quot 7 quot   quot DejaVuSansMono quot   quot 8 quot   quot LiberationMono quot   quot 9 quot   quot NimbusMonoL quot   quot 10 quot   quot CourierNew quot   quot 11 quot   quot Courier quot    It s identical in Selenium and in Chrome  VPNs work for a single use  but they get detected after I load the first page  Clearly some JavaScript is being run to detect Selenium

User · Answer

It seems to me the simplest way to do it with Selenium is to intercept the XHR that sends back the browser fingerprint.

But since this is a Selenium-only problem, its better just to use something else. Selenium is supposed to make things like this easier, not way harder.

User · Answer

It sounds like they are behind a web application firewall  Take a look at modsecurity and OWASP to see how those work  In reality  what you are asking is how to do bot detection evasion  That is not what Selenium WebDriver is for   It is for testing your web application not hitting other web applications   It is possible  but basically  you d have to look at what a WAF looks for in their rule set and specifically avoid it with selenium if you can   Even then  it might still not work because you don t know what WAF they are using  You did the right first step  that is  faking the user agent   If that didn t work though  then a WAF is in place and you probably need to get more tricky  Point taken from other answer   Make sure your user agent is actually being set correctly first   Maybe have it hit a local web server or sniff the traffic going out

User · Answer

I ve found changing the JavaScript  quot key quot  variable like this    Fools the website into believing a human is navigating it   JavascriptExecutor driver  executeScript  quot window key     quot blahblah  quot   quot     works for some websites when using Selenium WebDriver along with Google Chrome  since many sites check for this variable in order to avoid being scraped by Selenium

User · Answer

Additionally to the great answer of Erti-Chris Eelmaa - there s annoying window navigator webdriver and it is read-only  Event if you change the value of it to false it will still have true  That s why the browser driven by automated software can still be detected  MDN The variable is managed by the flag --enable-automation in chrome  The chromedriver launches Chrome with that flag and Chrome sets the window navigator webdriver to true  You can find it here  You need to add to  quot exclude switches quot  the flag  For instance  Go   package main  import        quot github com tebeka selenium quot       quot github com tebeka selenium chrome quot     func main      caps    selenium Capabilities       quot browserName quot    quot chrome quot      chromeCaps    chrome Capabilities      Path              quot  path to chrome-binary quot       ExcludeSwitches    string  quot enable-automation quot      caps AddChrome chromeCaps   wd  err    selenium NewRemote caps  fmt Sprintf  quot http   localhost  d wd hub quot   4444

User · Answer

Try to use selenium with a specific user profile of chrome  That way you can use it as specific user and define any thing you want  When doing so it will run as a  real  user  look at chrome process with some process explorer and you ll see the difference with the tags   For example   username   os getenv  USERNAME   userProfile    C   Users      username      AppData  Local  Google  Chrome  User Data  Default  options   webdriver ChromeOptions   options add argument  user-data-dir     format userProfile     add here any tag you want  options add experimental option  excludeSwitches     ignore-certificate-errors    safebrowsing-disable-download-protection    safebrowsing-disable-auto-update    disable-client-side-phishing-detection    chromedriver    C  Python27 chromedriver chromedriver exe  os environ  webdriver chrome driver     chromedriver browser   webdriver Chrome executable path chromedriver  chrome options options    chrome tag list here

User · Answer

A lot have been analyzed and discussed about a website being detected being driven by Selenium controlled ChromeDriver  Here are my two cents  According to the article Browser detection using the user agent serving different webpages or services to different browsers is usually not among the best of ideas  The web is meant to be accessible to everyone  regardless of which browser or device an user is using  There are best practices outlined to develop a website to progressively enhance itself based on the feature availability rather than by targeting specific browsers  However  browsers and standards are not perfect  and there are still some edge cases where some websites still detects the browser and if the browser is driven by Selenium controled WebDriver  Browsers can be detected through different ways and some commonly used mechanisms are as follows   Implementing captcha   recaptcha to detect the automatic bots    You can find a relevant detailed discussion in How does recaptcha 3 know I m using selenium chromedriver    Detecting the term HeadlessChrome within headless Chrome UserAgent   You can find a relevant detailed discussion in Access Denied page with headless Chrome on Linux while headed Chrome works on windows using Selenium through Python   Using Bot Management service from Distil Networks   You can find a relevant detailed discussion in Unable to use Selenium to automate Chase site login   Using Bot Manager service from Akamai   You can find a relevant detailed discussion in Dynamic dropdown doesn t populate with auto suggestions on https   www nseindia com  when values are passed using Selenium and Python   Using Bot Protection service from Datadome   You can find a relevant detailed discussion in Website using DataDome gets captcha blocked while scraping using Selenium and Python  However  using the user-agent to detect the browser looks simple but doing it well is in fact a bit tougher   Note  At this point it s worth to mention that  it s very rarely a good idea to use user agent sniffing  There are always better and more broadly compatible way to address a certain issue    Considerations for browser detection The idea behind detecting the browser can be either of the following   Trying to work around a specific bug in some specific variant or specific version of a webbrowser  Trying to check for the existence of a specific feature that some browsers don t yet support  Trying to provide different HTML depending on which browser is being used    Alternative of browser detection through UserAgents Some of the alternatives of browser detection are as follows   Implementing a test to detect how the browser implements the API of a feature and determine how to use it from that  An example was Chrome unflagged experimental lookbehind support in regular expressions  Adapting the design technique of Progressive enhancement which would involve developing a website in layers  using a bottom-up approach  starting with a simpler layer and improving the capabilities of the site in successive layers  each using more features  Adapting the top-down approach of Graceful degradation in which we build the best possible site using all the features we want and then tweak it to make it work on older browsers    Solution To prevent the Selenium driven WebDriver from getting detected  a niche approach would include either all of the below mentioned approaches   Rotating the UserAgent in every execution of your Test Suite using fake useragent module as follows  from selenium import webdriver from selenium webdriver chrome options import Options from fake useragent import UserAgent  options   Options   ua   UserAgent   userAgent   ua random print userAgent  options add argument f user-agent  userAgent    driver   webdriver Chrome chrome options options  executable path r C  WebDrivers ChromeDriver chromedriver win32 chromedriver exe   driver get  quot https   www google co in quot   driver quit       You can find a relevant detailed discussion in Way to change Google Chrome user agent in Selenium    Rotating the UserAgent in each of your Tests using Network setUserAgentOverride through execute cdp cmd   as follows  from selenium import webdriver  driver   webdriver Chrome executable path r C  WebDrivers chromedriver exe   print driver execute script  quot return navigator userAgent  quot      Setting user agent as Chrome 83 0 4103 97 driver execute cdp cmd  Network setUserAgentOverride     quot userAgent quot    Mozilla 5 0  Windows NT 10 0  Win64  x64  AppleWebKit 537 36  KHTML  like Gecko  Chrome 83 0 4103 97 Safari 537 36    print driver execute script  quot return navigator userAgent  quot        You can find a relevant detailed discussion in How to change the User Agent using Selenium and Python   Changing the property value of navigator for webdriver to undefined as follows  driver execute cdp cmd  quot Page addScriptToEvaluateOnNewDocument quot        quot source quot    quot  quot  quot      Object defineProperty navigator   webdriver           get       gt  undefined           quot  quot  quot         You can find a relevant detailed discussion in Selenium webdriver  Modifying navigator webdriver flag to prevent selenium detection   Changing the values of navigator plugins  navigator languages  WebGL  hairline feature  missing image  etc    You can find a relevant detailed discussion in Is there a version of selenium webdriver that is not detectable    Changing the conventional Viewport   You can find a relevant detailed discussion in How to bypass Google captcha with Selenium and python    Dealing with reCAPTCHA While dealing with 2captcha and recaptcha-v3 rather clicking on checkbox associated to the text I m not a robot  it may be easier to get authenticated extracting and using the data-sitekey   You can find a relevant detailed discussion in How to identify the 32 bit data-sitekey of ReCaptcha V2 to obtain a valid response programmatically using Selenium and Python Requests

User · Answer

Write an html page with the following code  You will see that in the DOM selenium applies a webdriver attribute in the outerHTML   x000D   x000D   lt html gt  x000D   lt head gt  x000D     lt script type  text javascript  gt  x000D     lt  -- x000D      function showWindow    x000D        javascript  alert document documentElement outerHTML    x000D        x000D      -- gt  x000D     lt  script gt  x000D   lt  head gt  x000D   lt body gt  x000D     lt form gt  x000D       lt input type  button  value  Show outerHTML  onclick  showWindow    gt  x000D     lt  form gt  x000D   lt  body gt  x000D   lt  html gt  x000D   x000D   x000D

User · Answer

Obfuscating JavaScripts result  I have checked the chromedriver source code  That injects some javascript files to the browser   Every javascript file on this link is injected to the web pages  https   chromium googlesource com chromium src   master chrome test chromedriver js   So I used reverse engineering and obfuscated the js files by Hex editing  Now i was sure that no more javascript variable  function names and fixed strings were used to uncover selenium activity  But still some sites and reCaptcha detect selenium   Maybe they check the modifications that are caused by chromedriver js execution       Edit 1   Chrome  navigator  parameters modification  I discovered there are some parameters in  navigator  that briefly uncover using of chromedriver   These are the parameters     navigator webdriver  On non-automated mode it is  undefined   On automated mode it s  true    navigator plugins  On headless chrome has 0 length  So I added some fake elements to fool the plugin length checking process   navigator languages  was set to default chrome value    en-US    en    es        So what i needed was a chrome extension to run javascript on the web pages  I made an extension with the js code provided in the article and used another article to add the zipped extension to my project  I have successfully changed the values  But still nothing changed   I didn t find other variables like these but it doesn t mean that they don t exist  Still reCaptcha detects chromedriver  So there should be more variables to change  The next step should be reverse engineering of the detector services that i don t want to do   Now I m not sure does it worth to spend more time on this automation process or search for alternative methods

User · Answer

One more thing I found is that some websites uses a platform that checks the User Agent  If the value contains   HeadlessChrome  the behavior can be weird when using headless mode   The workaround for that will be to override the user agent value  for example in Java   chromeOptions addArguments  --user-agent Mozilla 5 0  Macintosh  Intel Mac OS X 10 13 6  AppleWebKit 537 36  KHTML  like Gecko  Chrome 73 0 3683 86 Safari 537 36

User · Answer

As we ve already figured out in the question and the posted answers  there is an anti Web-scraping and a Bot detection service called  Distil Networks  in play here  And  according to the company CEO s interview      Even though they can create new bots  we figured out a way to identify   Selenium the a tool they   re using  so we   re blocking Selenium no   matter how many times they iterate on that bot  We   re doing that now   with Python and a lot of different technologies  Once we see a pattern   emerge from one type of bot  then we work to reverse engineer the   technology they use and identify it as malicious    It ll take time and additional challenges to understand how exactly they are detecting Selenium  but what can we say for sure at the moment    it s not related to the actions you take with selenium - once you navigate to the site  you get immediately detected and banned  I ve tried to add artificial random delays between actions  take a pause after the page is loaded - nothing helped it s not about browser fingerprint either - tried it in multiple browsers with clean profiles and not  incognito modes - nothing helped since  according to the hint in the interview  this was  reverse engineering   I suspect this is done with some JS code being executed in the browser revealing that this is a browser automated via selenium webdriver   Decided to post it as an answer  since clearly      Can a website detect when you are using selenium with chromedriver    Yes     Also  what I haven t experimented with is older selenium and older browser versions - in theory  there could be something implemented added to selenium at a certain point that Distil Networks bot detector currently relies on  Then  if this is the case  we might detect  yeah  let s detect the detector  at what point version a relevant change was made  look into changelog and changesets and  may be  this could give us more information on where to look and what is it they use to detect a webdriver-powered browser  It s just a theory that needs to be tested

User · Answer

Replacing cdc  string You can use vim or perl to replace the cdc  string in chromedriver  See answer by  Erti-Chris Eelmaa to learn more about that string and how it s a detection point  Using vim or perl prevents you from having to recompile source code or use a hex-editor  Make sure to make a copy of the original chromedriver before attempting to edit it  Our goal is to alter the cdc  string  which looks something like  cdc lasutopfhvcZLmcfl  The methods below were tested on chromedriver version 2 41 578706   Using Vim vim  path to chromedriver  After running the line above  you ll probably see a bunch of gibberish  Do the following   Replace all instances of cdc  with dog  by typing   s cdc  dog  g   dog  is just an example  You can choose anything as long as it has the same amount of characters as the search string  e g   cdc    otherwise the chromedriver will fail    To save the changes and quit  type  wq  and press return   If you need to quit without saving changes  type  q  and press return      Using Perl The line below replaces all cdc  occurrences with dog   Credit to Vic Seedoubleyew  perl -pi -e  s cdc  dog  g   path to chromedriver  Make sure that the replacement string  e g   dog   has the same number of characters as the search string  e g   cdc    otherwise the chromedriver will fail   Wrapping Up To verify that all occurrences of cdc  were replaced  grep  quot cdc  quot   path to chromedriver  If no output was returned  the replacement was successful  Go to the altered chromedriver and double click on it  A terminal window should open up  If you don t see killed in the output  you ve successfully altered the driver  Make sure that the name of the altered chromedriver binary is chromedriver  and that the original binary is either moved from its original location or renamed   My Experience With This Method I was previously being detected on a website while trying to log in  but after replacing cdc  with an equal sized string  I was able to log in  Like others have said though  if you ve already been detected  you might get blocked for a plethora of other reasons even after using this method  So you may have to try accessing the site that was detecting you using a VPN  different network  etc

User · Answer

partial interface Navigator         readonly attribute boolean webdriver            The webdriver IDL attribute of the Navigator interface must return the value of the webdriver-active flag  which is initially false       This property allows websites to determine that the user agent is under control by WebDriver  and can be used to help mitigate denial-of-service attacks    Taken directly from the 2017 W3C Editor s Draft of WebDriver  This heavily implies that at the very least  future iterations of selenium s drivers will be identifiable to prevent misuse  Ultimately  it s hard to tell without the source code  what exactly causes chrome driver in specific to be detectable

User · Answer

Basically  the way the Selenium detection works  is that they test for predefined JavaScript variables which appear when running with Selenium  The bot detection scripts usually look anything containing word  quot selenium quot     quot webdriver quot  in any of the variables  on window object   and also document variables called  cdc  and  wdc   Of course  all of this depends on which browser you are on  All the different browsers expose different things  For me  I used Chrome  so  all that I had to do was to ensure that  cdc  didn t exist anymore as a document variable  and voil    download chromedriver source code  modify chromedriver and re-compile  cdc  under different name   This is the function I modified in chromedriver  File call function js  function getPageCache opt doc      var doc   opt doc    document      var key     cdc asdjflasutopfhvcZLmcfl      var key    randomblabla      if    key in doc       doc key    new Cache      return doc key       Note the comment  All I did I turned  cdc  to randomblabla    Here is pseudocode which demonstrates some of the techniques that bot networks might use  runBotDetection   function          var documentDetectionKeys              quot   webdriver evaluate quot            quot   selenium evaluate quot            quot   webdriver script function quot            quot   webdriver script func quot            quot   webdriver script fn quot            quot   fxdriver evaluate quot            quot   driver unwrapped quot            quot   webdriver unwrapped quot            quot   driver evaluate quot            quot   selenium unwrapped quot            quot   fxdriver unwrapped quot               var windowDetectionKeys              quot  phantom quot            quot   nightmare quot            quot  selenium quot            quot callPhantom quot            quot callSelenium quot            quot  Selenium IDE Recorder quot               for  const windowDetectionKey in windowDetectionKeys            const windowDetectionKeyValue   windowDetectionKeys windowDetectionKey           if  window windowDetectionKeyValue                 return true                       for  const documentDetectionKey in documentDetectionKeys            const documentDetectionKeyValue   documentDetectionKeys documentDetectionKey           if  window  document   documentDetectionKeyValue                 return true                        for  const documentKey in window  document              if  documentKey match     a-z dc     amp  amp  window  document   documentKey   cache                   return true                       if  window  external    amp  amp  window  external   toString    amp  amp   window  external   toString    indexOf    Sequentum      -1   return true       if  window  document    documentElement    getAttribute    selenium    return true      if  window  document    documentElement    getAttribute    webdriver    return true      if  window  document    documentElement    getAttribute    driver    return true       return false      According to user szx  it is also possible to simply open chromedriver exe in a hex editor  and just do the replacement manually  without actually doing any compiling

User · Answer

You can try to use the parameter  enable-automation   var options   new ChromeOptions        hide selenium options AddExcludedArguments new List lt string gt       enable-automation       var driver   new ChromeDriver ChromeDriverService CreateDefaultService    options     But  I want to warn that this ability was fixed in ChromeDriver 79 0 3945 16   So probably you should use older versions of chrome   Also  as another option  you can try using InternetExplorerDriver instead of Chrome  As for me  IE does not block at all without any hacks   And for more info try to take a look here   Selenium webdriver  Modifying navigator webdriver flag to prevent selenium detection  Unable to hide  quot Chrome is being controlled by automated software quot  infobar within Chrome v76

User · Answer

Firefox is said to set window navigator webdriver     true if working with a webdriver  That was according to one of the older specs  e g   archive org  but I couldn t find it in the new one except for some very vague wording in the appendices   A test for it is in the selenium code in the file fingerprint test js where the comment at the end says  Currently only implemented in firefox  but I wasn t able to identify any code in that direction with some simple greping  neither in the current  41 0 2  Firefox release-tree nor in the Chromium-tree   I also found a comment for an older commit regarding fingerprinting in the firefox driver b82512999938 from January 2015  That code is still in the Selenium GIT-master downloaded yesterday at javascript firefox-driver extension content server js with a comment linking to the slightly differently worded appendix in the current w3c webdriver spec

User · Answer

Even if you are sending all the right data  e g  Selenium doesn t show up as an extension  you have a reasonable resolution bit-depth   amp c   there are a number of services and tools which profile visitor behaviour to determine whether the actor is a user or an automated system   For example  visiting a site then immediately going to perform some action by moving the mouse directly to the relevant button  in less than a second  is something no user would actually do   It might also be useful as a debugging tool to use a site such as https   panopticlick eff org  to check how unique your browser is  it ll also help you verify whether there are any specific parameters that indicate you re running in Selenium

User · Answer

Example of how it s implemented on wellsfargo com    try    if  window document documentElement getAttribute  webdriver    return        catch  IDLMrxxel     try    if    Selenium IDE Recorder  in window  return        catch  KknKsUayS     try    if     webdriver script fn  in document  return

User · Answer

The bot detection I ve seen seems more sophisticated or at least different than what I ve read through in the answers below   EXPERIMENT 1    I open a browser and web page with Selenium from a Python console   The mouse is already at a specific location where I know a link will appear once the page loads  I never move the mouse   I press the left mouse button once  this is necessary to take focus from the console where Python is running to the browser    I press the left mouse button again  remember  cursor is above a given link   The link opens normally  as it should    EXPERIMENT 2    As before  I open a browser and the web page with Selenium from a Python console   This time around  instead of clicking with the mouse  I use Selenium  in the Python console  to click the same element with a random offset   The link doesn t open  but I am taken to a sign up page    IMPLICATIONS    opening a web browser via Selenium doesn t preclude me from appearing human moving the mouse like a human is not necessary to be classified as human clicking something via Selenium with an offset still raises the alarm   Seems mysterious  but I guess they can just determine whether an action originates from Selenium or not  while they don t care whether the browser itself was opened via Selenium or not  Or can they determine if the window has focus  Would be interesting to hear if anyone has any insights

User · Answer

It works for some websites  remove property webdriver from navigator from selenium import webdriver driver   webdriver Chrome   driver execute cdp cmd  quot Page addScriptToEvaluateOnNewDocument quot          quot source quot            quot const newProto   navigator   proto    quot           quot delete newProto webdriver  quot           quot navigator   proto     newProto  quot

User · Answer

Some sites are detecting this   function d     try       if  window document  cdc asdjflasutopfhvcZLmcfl  cache           return  0   catch  e      try         if  window document documentElement getAttribute decodeURIComponent   77 65 62 64 72 69 76 65 72         if  window document documentElement getAttribute  webdriver            return  0   catch  e      try         if  decodeURIComponent   5F 53 65 6C 65 6E 69 75 6D 5F 49 44 45 5F 52 65 63 6F 72 64 65 72   in window      if    Selenium IDE Recorder  in window          return  0   catch  e      try         if  decodeURIComponent   5F 5F 77 65 62 64 72 69 76 65 72 5F 73 63 72 69 70 74 5F 66 6E   in document      if     webdriver script fn  in document          return  0   catch  e

[javascript] Can a website detect when you are using Selenium with chromedriver?

Examples related to javascript

Examples related to python

Examples related to google-chrome

Examples related to selenium

Examples related to selenium-chromedriver