[javascript] Can a website detect when you are using Selenium with chromedriver?

A lot have been analyzed and discussed about a website being detected being driven by Selenium controlled ChromeDriver. Here are my two cents:

According to the article Browser detection using the user agent serving different webpages or services to different browsers is usually not among the best of ideas. The web is meant to be accessible to everyone, regardless of which browser or device an user is using. There are best practices outlined to develop a website to progressively enhance itself based on the feature availability rather than by targeting specific browsers.

However, browsers and standards are not perfect, and there are still some edge cases where some websites still detects the browser and if the browser is driven by Selenium controled WebDriver. Browsers can be detected through different ways and some commonly used mechanisms are as follows:

You can find a relevant detailed discussion in How does recaptcha 3 know I'm using selenium/chromedriver?

  • Detecting the term HeadlessChrome within headless Chrome UserAgent

You can find a relevant detailed discussion in Access Denied page with headless Chrome on Linux while headed Chrome works on windows using Selenium through Python

You can find a relevant detailed discussion in Unable to use Selenium to automate Chase site login

  • Using Bot Manager service from Akamai

You can find a relevant detailed discussion in Dynamic dropdown doesn't populate with auto suggestions on https://www.nseindia.com/ when values are passed using Selenium and Python

  • Using Bot Protection service from Datadome

You can find a relevant detailed discussion in Website using DataDome gets captcha blocked while scraping using Selenium and Python

However, using the to detect the browser looks simple but doing it well is in fact a bit tougher.

Note: At this point it's worth to mention that: it's very rarely a good idea to use user agent sniffing. There are always better and more broadly compatible way to address a certain issue.


Considerations for browser detection

The idea behind detecting the browser can be either of the following:

  • Trying to work around a specific bug in some specific variant or specific version of a webbrowser.
  • Trying to check for the existence of a specific feature that some browsers don't yet support.
  • Trying to provide different HTML depending on which browser is being used.

Alternative of browser detection through UserAgents

Some of the alternatives of browser detection are as follows:

  • Implementing a test to detect how the browser implements the API of a feature and determine how to use it from that. An example was Chrome unflagged experimental lookbehind support in regular expressions.
  • Adapting the design technique of Progressive enhancement which would involve developing a website in layers, using a bottom-up approach, starting with a simpler layer and improving the capabilities of the site in successive layers, each using more features.
  • Adapting the top-down approach of Graceful degradation in which we build the best possible site using all the features we want and then tweak it to make it work on older browsers.

Solution

To prevent the Selenium driven WebDriver from getting detected, a niche approach would include either/all of the below mentioned approaches:

  • Rotating the UserAgent in every execution of your Test Suite using fake_useragent module as follows:

    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options
    from fake_useragent import UserAgent
    
    options = Options()
    ua = UserAgent()
    userAgent = ua.random
    print(userAgent)
    options.add_argument(f'user-agent={userAgent}')
    driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\ChromeDriver\chromedriver_win32\chromedriver.exe')
    driver.get("https://www.google.co.in")
    driver.quit()
    

You can find a relevant detailed discussion in Way to change Google Chrome user agent in Selenium?

  • Rotating the UserAgent in each of your Tests using Network.setUserAgentOverride through execute_cdp_cmd() as follows:

    from selenium import webdriver
    
    driver = webdriver.Chrome(executable_path=r'C:\WebDrivers\chromedriver.exe')
    print(driver.execute_script("return navigator.userAgent;"))
    # Setting user agent as Chrome/83.0.4103.97
    driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'})
    print(driver.execute_script("return navigator.userAgent;"))
    

You can find a relevant detailed discussion in How to change the User Agent using Selenium and Python

  • Changing the property value of navigator for webdriver to undefined as follows:

    driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
      "source": """
        Object.defineProperty(navigator, 'webdriver', {
          get: () => undefined
        })
      """
    })
    

You can find a relevant detailed discussion in Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection

  • Changing the values of navigator.plugins, navigator.languages, WebGL, hairline feature, missing image, etc.

You can find a relevant detailed discussion in Is there a version of selenium webdriver that is not detectable?

You can find a relevant detailed discussion in How to bypass Google captcha with Selenium and python?


Dealing with reCAPTCHA

While dealing with and rather clicking on associated to the text I'm not a robot, it may be easier to get authenticated extracting and using the data-sitekey.

You can find a relevant detailed discussion in How to identify the 32 bit data-sitekey of ReCaptcha V2 to obtain a valid response programmatically using Selenium and Python Requests?

Examples related to javascript

need to add a class to an element How to make a variable accessible outside a function? Hide Signs that Meteor.js was Used How to create a showdown.js markdown extension Please help me convert this script to a simple image slider Highlight Anchor Links when user manually scrolls? Summing radio input values How to execute an action before close metro app WinJS javascript, for loop defines a dynamic variable name Getting all files in directory with ajax

Examples related to python

programming a servo thru a barometer Is there a way to view two blocks of code from the same file simultaneously in Sublime Text? python variable NameError Why my regexp for hyphenated words doesn't work? Comparing a variable with a string python not working when redirecting from bash script is it possible to add colors to python output? Get Public URL for File - Google Cloud Storage - App Engine (Python) Real time face detection OpenCV, Python xlrd.biffh.XLRDError: Excel xlsx file; not supported Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

Examples related to google-chrome

SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 81 SameSite warning Chrome 77 What's the net::ERR_HTTP2_PROTOCOL_ERROR about? session not created: This version of ChromeDriver only supports Chrome version 74 error with ChromeDriver Chrome using Selenium Jupyter Notebook not saving: '_xsrf' argument missing from post How to fix 'Unchecked runtime.lastError: The message port closed before a response was received' chrome issue? Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser How to make audio autoplay on chrome How to handle "Uncaught (in promise) DOMException: play() failed because the user didn't interact with the document first." on Desktop with Chrome 66?

Examples related to selenium

SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 81 session not created: This version of ChromeDriver only supports Chrome version 74 error with ChromeDriver Chrome using Selenium Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser Class has been compiled by a more recent version of the Java Environment How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium? How to make Firefox headless programmatically in Selenium with Python? element not interactable exception in selenium web automation Selenium Web Driver & Java. Element is not clickable at point (x, y). Other element would receive the click How do you fix the "element not interactable" exception?

Examples related to selenium-chromedriver

SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 81 session not created: This version of ChromeDriver only supports Chrome version 74 error with ChromeDriver Chrome using Selenium Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium? Python Selenium Chrome Webdriver selenium - chromedriver executable needs to be in PATH Only local connections are allowed Chrome and Selenium webdriver How to set "value" to input web element using selenium? Can a website detect when you are using Selenium with chromedriver?