[webdriver] setting request headers in selenium

I'm attempting to set the request header 'Referer' to spoof a request coming from another site. We need the ability test that a specific referrer is used, which returns a specific form to the user, otherwise an alternative form is given.

I can do this within poltergeist by:

page.driver.headers = {"Referer" => referer_string}

but I can't find the equivalent functionality for the selemium driver.

How can I set request headers in the capybara selenium driver?

This question is related to webdriver capybara selenium-webdriver

The answer is


You can do it with PhantomJSDriver.

PhantomJSDriver pd = ((PhantomJSDriver) ((WebDriverFacade) getDriver()).getProxiedDriver());
pd.executePhantomJS(
            "this.onResourceRequested = function(request, net) {" +
            "   net.setHeader('header-name', 'header-value')" +
            "};");

Using the request object, you can filter also so the header won't be set for every request.


I had the same issue. I solved it downloading modify-headers firefox add-on and activate it with selenium.

The code in python is the following

fp = webdriver.FirefoxProfile()
path_modify_header = 'C:/xxxxxxx/modify_headers-0.7.1.1-fx.xpi'
fp.add_extension(path_modify_header)

fp.set_preference("modifyheaders.headers.count", 1)
fp.set_preference("modifyheaders.headers.action0", "Add")
fp.set_preference("modifyheaders.headers.name0", "Name_of_header") # Set here the name of the header
fp.set_preference("modifyheaders.headers.value0", "value_of_header") # Set here the value of the header
fp.set_preference("modifyheaders.headers.enabled0", True)
fp.set_preference("modifyheaders.config.active", True)
fp.set_preference("modifyheaders.config.alwaysOn", True)

driver = webdriver.Firefox(firefox_profile=fp)

I wanted something a bit slimmer for RSpec/Ruby so that the custom code only had to live in one place. Here's my solution:

/spec/support/selenium.rb
...
RSpec.configure do |config|
  config.after(:suite) do
    $custom_headers = nil
  end
end

module RequestWithExtraHeaders
  def headers
    $custom_headers.each do |key, value|
      self.set_header "HTTP_#{key}", value
    end if $custom_headers

    super
  end
end
class ActionDispatch::Request
  prepend RequestWithExtraHeaders
end

Then in my specs:

/specs/features/something_spec.rb
...
$custom_headers = {"Referer" => referer_string}

check this out: chrome_options = Options()

chrome_options.add_argument('--headless')
chrome_options.add_argument('user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"')

Problem solved!


Had the same issue today, except that I needed to set different referer per test. I ended up using a middleware and a class to pass headers to it. Thought I'd share (or maybe there's a cleaner solution?):

lib/request_headers.rb:

class CustomHeadersHelper
  cattr_accessor :headers
end

class RequestHeaders
  def initialize(app, helper = nil)
    @app, @helper = app, helper
  end

  def call(env)
    if @helper
      headers = @helper.headers

      if headers.is_a?(Hash)
        headers.each do |k,v|
          env["HTTP_#{k.upcase.gsub("-", "_")}"] = v
        end
      end
    end

    @app.call(env)
  end
end

config/initializers/middleware.rb

require 'request_headers'

if %w(test cucumber).include?(Rails.env)
  Rails.application.config.middleware.insert_before Rack::Lock, "RequestHeaders", CustomHeadersHelper
end

spec/support/capybara_headers.rb

require 'request_headers'

module CapybaraHeaderHelpers
  shared_context "navigating within the site" do
    before(:each) { add_headers("Referer" => Capybara.app_host + "/") }
  end

  def add_headers(custom_headers)
    if Capybara.current_driver == :rack_test
      custom_headers.each do |name, value|
        page.driver.browser.header(name, value)
      end
    else
      CustomHeadersHelper.headers = custom_headers
    end
  end
end

spec/spec_helper.rb

...
config.include CapybaraHeaderHelpers

Then I can include the shared context wherever I need, or pass different headers in another before block. I haven't tested it with anything other than Selenium and RackTest, but it should be transparent, as header injection is done before the request actually hits the application.


For those people using Python, you may consider using Selenium Wire, which can set request headers, as well as provide you with the ability to inspect requests and responses.

from seleniumwire import webdriver  # Import from seleniumwire

# Create a new instance of the Firefox driver (or Chrome)
driver = webdriver.Firefox()

# Create a request interceptor
def interceptor(request):
    del request.headers['Referer']  # Delete the header first
    request.headers['Referer'] = 'some_referer'

# Set the interceptor on the driver
driver.request_interceptor = interceptor

# All requests will now use 'some_referer' for the referer
driver.get('https://mysite')

With the solutions already discussed above the most reliable one is using Browsermob-Proxy

But while working with the remote grid machine, Browsermob-proxy isn't really helpful.

This is how I fixed the problem in my case. Hopefully, might be helpful for anyone with a similar setup.

  1. Add the ModHeader extension to the chrome browser

How to download the Modheader? Link

ChromeOptions options = new ChromeOptions();
options.addExtensions(new File(C://Downloads//modheader//modheader.crx));

// Set the Desired capabilities 
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(ChromeOptions.CAPABILITY, options);

// Instantiate the chrome driver with capabilities
WebDriver driver = new RemoteWebDriver(new URL(YOUR_HUB_URL), options);
  1. Go to the browser extensions and capture the Local Storage context ID of the ModHeader

Capture ID from ModHeader

  1. Navigate to the URL of the ModHeader to set the Local Storage Context

.

// set the context on the extension so the localStorage can be accessed
driver.get("chrome-extension://idgpnmonknjnojddfkpgkljpfnnfcklj/_generated_background_page.html");

Where `idgpnmonknjnojddfkpgkljpfnnfcklj` is the value captured from the Step# 2
  1. Now add the headers to the request using Javascript

.

   ((Javascript)driver).executeScript(
         "localStorage.setItem('profiles', JSON.stringify([{  title: 'Selenium', hideComment: true, appendMode: '', 
             headers: [                        
               {enabled: true, name: 'token-1', value: 'value-1', comment: ''},
               {enabled: true, name: 'token-2', value: 'value-2', comment: ''}
             ],                          
             respHeaders: [],
             filters: []
          }]));");

Where token-1, value-1, token-2, value-2 are the request headers and values that are to be added.

  1. Now navigate to the required web-application.

    driver.get("your-desired-website");


If you use the HtmlUnitDriver, you can set request headers by modifying the WebClient, like so:

final case class Header(name: String, value: String)

final class HtmlUnitDriverWithHeaders(headers: Seq[Header]) extends HtmlUnitDriver {
  super.modifyWebClient {
    val client = super.getWebClient
    headers.foreach(h => client.addRequestHeader(h.name, h.value))
    client
  }
}

The headers will then be on all requests made by the web browser.


Examples related to webdriver

WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser Selenium Web Driver & Java. Element is not clickable at point (x, y). Other element would receive the click How do you fix the "element not interactable" exception? ImportError: No module named 'selenium' how to use List<WebElement> webdriver How to get attribute of element from Selenium? Selenium Finding elements by class name in python Open web in new tab Selenium + Python When running WebDriver with Chrome browser, getting message, "Only local connections are allowed" even though browser launches properly How can I start InternetExplorerDriver using Selenium WebDriver

Examples related to capybara

How to select option in drop down using Capybara setting request headers in selenium

Examples related to selenium-webdriver

SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 81 Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium? How to make Firefox headless programmatically in Selenium with Python? Selenium Web Driver & Java. Element is not clickable at point (x, y). Other element would receive the click How do you fix the "element not interactable" exception? Scrolling to element using webdriver? Only local connections are allowed Chrome and Selenium webdriver Check if element is clickable in Selenium Java