[java] Selenium WebDriver: Wait for complex page with JavaScript to load

I have a web application to test with Selenium. There is a lot of JavaScript running on page load.
This JavaScript code is not so well written but I can't change anything. So waiting for an element to appear in the DOM with findElement() method is not an option.
I want to create a generic function in Java to wait for a page to load, a possible solution would be:

  • run a JavaScript script form WebDriver and store the result of document.body.innerHTML in a string variable body.
  • compare the body variable to the previous version of body. if they are the same then set increment a counter notChangedCount otherwise set notChangedCount to zero.
  • wait for a litte time (50 ms for example).
  • if the page has not changed for some time (500 ms for example) so notChangedCount >= 10 then exit the loop otherwise loop to the first step.

Do you think it's a valid solution?

This question is related to java c# selenium selenium-webdriver webdriver

The answer is


You need to wait for Javascript and jQuery to finish loading. Execute Javascript to check if jQuery.active is 0 and document.readyState is complete, which means the JS and jQuery load is complete.

public boolean waitForJStoLoad() {

    WebDriverWait wait = new WebDriverWait(driver, 30);

    // wait for jQuery to load
    ExpectedCondition<Boolean> jQueryLoad = new ExpectedCondition<Boolean>() {
      @Override
      public Boolean apply(WebDriver driver) {
        try {
          return ((Long)executeJavaScript("return jQuery.active") == 0);
        }
        catch (Exception e) {
          return true;
        }
      }
    };

    // wait for Javascript to load
    ExpectedCondition<Boolean> jsLoad = new ExpectedCondition<Boolean>() {
      @Override
      public Boolean apply(WebDriver driver) {
        return executeJavaScript("return document.readyState")
            .toString().equals("complete");
      }
    };

  return wait.until(jQueryLoad) && wait.until(jsLoad);
}

If all you need to do is wait for the html on the page to become stable before trying to interact with elements, you can poll the DOM periodically and compare the results, if the DOMs are the same within the given poll time, you're golden. Something like this where you pass in the maximum wait time and the time between page polls before comparing. Simple and effective.

public void waitForJavascript(int maxWaitMillis, int pollDelimiter) {
    double startTime = System.currentTimeMillis();
    while (System.currentTimeMillis() < startTime + maxWaitMillis) {
        String prevState = webDriver.getPageSource();
        Thread.sleep(pollDelimiter); // <-- would need to wrap in a try catch
        if (prevState.equals(webDriver.getPageSource())) {
            return;
        }
    }
}

For the nodejs Selenium library, I used the following snippet. In my case, I was looking for two objects that are added to the window, which in this example are <SOME PROPERTY>, 10000 is the timeout milliseconds, <NEXT STEP HERE> is what happens after the properties are found on the window.

driver.wait( driver => {
    return driver.executeScript( 'if(window.hasOwnProperty(<SOME PROPERTY>) && window.hasOwnProperty(<SOME PROPERTY>)) return true;' ); }, 10000).then( ()=>{
        <NEXT STEP HERE>
}).catch(err => { 
    console.log("looking for window properties", err);
});

Does the JS library define/initialize any well known variable on the window?

If so you could wait for the variable to appear. You can use

((JavascriptExecutor)driver).executeScript(String script, Object... args)

to test for this condition (something like: window.SomeClass && window.SomeClass.variable != null) and return a boolean true / false.

Wrap this in a WebDriverWait, and wait until the script returns true.


I had a same issue. This solution works for me from WebDriverDoku:

WebDriverWait wait = new WebDriverWait(driver, 10);
WebElement element = wait.until(ExpectedConditions.elementToBeClickable(By.id("someid")));

http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp


To do it properly, you need to handle the exceptions.

Here is how I do a wait for an iFrame. This requires that your JUnit test class pass the instance of RemoteWebDriver into the page object :

public class IFrame1 extends LoadableComponent<IFrame1> {

    private RemoteWebDriver driver;

    @FindBy(id = "iFrame1TextFieldTestInputControlID" )
    public WebElement iFrame1TextFieldInput;

    @FindBy(id = "iFrame1TextFieldTestProcessButtonID" )
    public WebElement copyButton;

    public IFrame1( RemoteWebDriver drv ) {
        super();
        this.driver = drv;
        this.driver.switchTo().defaultContent();
        waitTimer(1, 1000);
        this.driver.switchTo().frame("BodyFrame1");
        LOGGER.info("IFrame1 constructor...");
    }

    @Override
    protected void isLoaded() throws Error {        
        LOGGER.info("IFrame1.isLoaded()...");
        PageFactory.initElements( driver, this );
        try {
            assertTrue( "Page visible title is not yet available.", driver
     .findElementByCssSelector("body form#webDriverUnitiFrame1TestFormID h1")
                    .getText().equals("iFrame1 Test") );
        } catch ( NoSuchElementException e) {
            LOGGER.info("No such element." );
            assertTrue("No such element.", false);
        }
    }

    @Override
    protected void load() {
        LOGGER.info("IFrame1.load()...");
        Wait<WebDriver> wait = new FluentWait<WebDriver>( driver )
                .withTimeout(30, TimeUnit.SECONDS)
                .pollingEvery(5, TimeUnit.SECONDS)
                .ignoring( NoSuchElementException.class ) 
                .ignoring( StaleElementReferenceException.class ) ;
            wait.until( ExpectedConditions.presenceOfElementLocated( 
            By.cssSelector("body form#webDriverUnitiFrame1TestFormID h1") ) );
    }
....

NOTE: You can see my entire working example here.


Here's from my own code:
Window.setTimeout executes only when browser is idle.
So calling the function recursively (42 times) will take 100ms if there is no activity in the browser and much more if the browser is busy doing something else.

    ExpectedCondition<Boolean> javascriptDone = new ExpectedCondition<Boolean>() {
        public Boolean apply(WebDriver d) {
            try{//window.setTimeout executes only when browser is idle,
                //introduces needed wait time when javascript is running in browser
                return  ((Boolean) ((JavascriptExecutor) d).executeAsyncScript( 

                        " var callback =arguments[arguments.length - 1]; " +
                        " var count=42; " +
                        " setTimeout( collect, 0);" +
                        " function collect() { " +
                            " if(count-->0) { "+
                                " setTimeout( collect, 0); " +
                            " } "+
                            " else {callback(" +
                            "    true" +                            
                            " );}"+                             
                        " } "
                    ));
            }catch (Exception e) {
                return Boolean.FALSE;
            }
        }
    };
    WebDriverWait w = new WebDriverWait(driver,timeOut);  
    w.until(javascriptDone);
    w=null;

As a bonus the counter can be reset on document.readyState or on jQuery Ajax calls or if any jQuery animations are running (only if your app uses jQuery for ajax calls...)
...

" function collect() { " +
                            " if(!((typeof jQuery === 'undefined') || ((jQuery.active === 0) && ($(\":animated\").length === 0))) && (document.readyState === 'complete')){" +
                            "    count=42;" +
                            "    setTimeout( collect, 0); " +
                            " }" +
                            " else if(count-->0) { "+
                                " setTimeout( collect, 0); " +
                            " } "+ 

...

EDIT: I notice executeAsyncScript doesn't work well if a new page loads and the test might stop responding indefinetly, better to use this on instead.

public static ExpectedCondition<Boolean> documentNotActive(final int counter){ 
    return new ExpectedCondition<Boolean>() {
        boolean resetCount=true;
        @Override
        public Boolean apply(WebDriver d) {

            if(resetCount){
                ((JavascriptExecutor) d).executeScript(
                        "   window.mssCount="+counter+";\r\n" + 
                        "   window.mssJSDelay=function mssJSDelay(){\r\n" + 
                        "       if((typeof jQuery != 'undefined') && (jQuery.active !== 0 || $(\":animated\").length !== 0))\r\n" + 
                        "           window.mssCount="+counter+";\r\n" + 
                        "       window.mssCount-->0 &&\r\n" + 
                        "       setTimeout(window.mssJSDelay,window.mssCount+1);\r\n" + 
                        "   }\r\n" + 
                        "   window.mssJSDelay();");
                resetCount=false;
            }

            boolean ready=false;
            try{
                ready=-1==((Long) ((JavascriptExecutor) d).executeScript(
                        "if(typeof window.mssJSDelay!=\"function\"){\r\n" + 
                        "   window.mssCount="+counter+";\r\n" + 
                        "   window.mssJSDelay=function mssJSDelay(){\r\n" + 
                        "       if((typeof jQuery != 'undefined') && (jQuery.active !== 0 || $(\":animated\").length !== 0))\r\n" + 
                        "           window.mssCount="+counter+";\r\n" + 
                        "       window.mssCount-->0 &&\r\n" + 
                        "       setTimeout(window.mssJSDelay,window.mssCount+1);\r\n" + 
                        "   }\r\n" + 
                        "   window.mssJSDelay();\r\n" + 
                        "}\r\n" + 
                        "return window.mssCount;"));
            }
            catch (NoSuchWindowException a){
                a.printStackTrace();
                return true;
            }
            catch (Exception e) {
                e.printStackTrace();
                return false;
            }
            return ready;
        }
        @Override
        public String toString() {
            return String.format("Timeout waiting for documentNotActive script");
        }
    };
}

Using implicit wait works for me.

driver.Manage().Timeouts().ImplicitWait = TimeSpan.FromSeconds(10);

Refer to this answer Selenium c# Webdriver: Wait Until Element is Present


Thanks Ashwin !

In my case I should need wait for a jquery plugin execution in some element.. specifically "qtip"

based in your hint, it worked perfectly for me :

wait.until( new Predicate<WebDriver>() {
            public boolean apply(WebDriver driver) {
                return ((JavascriptExecutor)driver).executeScript("return document.readyState").equals("complete");
            }
        }
    );

Note: I'm using Webdriver 2


I like your idea of polling the HTML until it's stable. I may add that to my own solution. The following approach is in C# and requires jQuery.

I'm the developer for a SuccessFactors (SaaS) test project where we have no influence at all over the developers or the characteristics of the DOM behind the web page. The SaaS product can potentially change its underlying DOM design 4 times a year, so the hunt is permanently on for robust, performant ways to test with Selenium (including NOT testing with Selenium where possi ble!)

Here's what I use for "page ready". It works in all my own tests currently. The same approach also worked for a big in-house Java web app a couple of years ago, and had been robust for over a year at the time I left the project.

  • Driver is the WebDriver instance that communicates with the browser
  • DefaultPageLoadTimeout is a timeout value in ticks (100ns per tick)

public IWebDriver Driver { get; private set; }

// ...

const int GlobalPageLoadTimeOutSecs = 10;
static readonly TimeSpan DefaultPageLoadTimeout =
    new TimeSpan((long) (10_000_000 * GlobalPageLoadTimeOutSecs));
Driver = new FirefoxDriver();

In what follows, note the order of waits in method PageReady (Selenium document ready, Ajax, animations), which makes sense if you think about it:

  1. load the page containing the code
  2. use the code to load the data from somewhere via Ajax
  3. present the data, possibly with animations

Something like your DOM comparison approach could be used between 1 and 2 to add another layer of robustness.


public void PageReady()
{
    DocumentReady();
    AjaxReady();
    AnimationsReady();
}


private void DocumentReady()
{
    WaitForJavascript(script: "return document.readyState", result: "complete");
}

private void WaitForJavascript(string script, string result)
{
    new WebDriverWait(Driver, DefaultPageLoadTimeout).Until(
        d => ((IJavaScriptExecutor) d).ExecuteScript(script).Equals(result));
}

private void AjaxReady()
{
    WaitForJavascript(script: "return jQuery.active.toString()", result: "0");
}

private void AnimationsReady()
{
    WaitForJavascript(script: "return $(\"animated\").length.toString()", result: "0");
}

The below code works perfectly in my case - my page contains complex java scripts

public void checkPageIsReady() {

  JavascriptExecutor js = (JavascriptExecutor)driver;


  //Initially bellow given if condition will check ready state of page.
  if (js.executeScript("return document.readyState").toString().equals("complete")){ 
   System.out.println("Page Is loaded.");
   return; 
  } 

  //This loop will rotate for 25 times to check If page Is ready after every 1 second.
  //You can replace your value with 25 If you wants to Increase or decrease wait time.
  for (int i=0; i<25; i++){ 
   try {
    Thread.sleep(1000);
    }catch (InterruptedException e) {} 
   //To check page ready state.
   if (js.executeScript("return document.readyState").toString().equals("complete")){ 
    break; 
   }   
  }
 }

Source - How To Wait For Page To Load/Ready In Selenium WebDriver


You can write some logic to handle this. I have write a method that will return the WebElement and this method will be called three times or you can increase the time and add a null check for WebElement Here is an example

public static void main(String[] args) {
        WebDriver driver = new FirefoxDriver();
        driver.get("https://www.crowdanalytix.com/#home");
        WebElement webElement = getWebElement(driver, "homekkkkkkkkkkkk");
        int i = 1;
        while (webElement == null && i < 4) {
            webElement = getWebElement(driver, "homessssssssssss");
            System.out.println("calling");
            i++;
        }
        System.out.println(webElement.getTagName());
        System.out.println("End");
        driver.close();
    }

    public static WebElement getWebElement(WebDriver driver, String id) {
        WebElement myDynamicElement = null;
        try {
            myDynamicElement = (new WebDriverWait(driver, 10))
                    .until(ExpectedConditions.presenceOfElementLocated(By
                            .id(id)));
            return myDynamicElement;
        } catch (TimeoutException ex) {
            return null;
        }
    }

Don't know how to do that but in my case, end of page load & rendering match with FAVICON displayed in Firefox tab.

So if we can get the favicon image in the webbrowser, the web page is fully loaded.

But how perform this ....


Two conditions can be used to check if the page is loaded before finding any element on the page:

WebDriverWait wait = new WebDriverWait(driver, 50);

Using below readyState will wait till page load

wait.until((ExpectedCondition<Boolean>) wd ->
   ((JavascriptExecutor) wd).executeScript("return document.readyState").equals("complete"));

Below JQuery will wait till data has not been loaded

  int count =0;
            if((Boolean) executor.executeScript("return window.jQuery != undefined")){
                while(!(Boolean) executor.executeScript("return jQuery.active == 0")){
                    Thread.sleep(4000);
                    if(count>4)
                        break;
                    count++;
                }
            }

After these JavaScriptCode try to findOut webElement.

WebElement we = wait.until(ExpectedConditions.presenceOfElementLocated(by));

I asked my developers to create a JavaScript variable "isProcessing" that I can access (in the "ae" object) that they set when things start running and clear when things are done. I then run it in an accumulator that checks it every 100 ms until it gets five in a row for a total of 500 ms without any changes. If 30 seconds pass, I throw an exception because something should have happened by then. This is in C#.

public static void WaitForDocumentReady(this IWebDriver driver)
{
    Console.WriteLine("Waiting for five instances of document.readyState returning 'complete' at 100ms intervals.");
    IJavaScriptExecutor jse = (IJavaScriptExecutor)driver;
    int i = 0; // Count of (document.readyState === complete) && (ae.isProcessing === false)
    int j = 0; // Count of iterations in the while() loop.
    int k = 0; // Count of times i was reset to 0.
    bool readyState = false;
    while (i < 5)
    {
        System.Threading.Thread.Sleep(100);
        readyState = (bool)jse.ExecuteScript("return ((document.readyState === 'complete') && (ae.isProcessing === false))");
        if (readyState) { i++; }
        else
        {
            i = 0;
            k++;
        }
        j++;
        if (j > 300) { throw new TimeoutException("Timeout waiting for document.readyState to be complete."); }
    }
    j *= 100;
    Console.WriteLine("Waited " + j.ToString() + " milliseconds. There were " + k + " resets.");
}

Here's how I do it:

new WebDriverWait(driver, 20).until(
       ExpectedConditions.jsReturnsValue(
                   "return document.readyState === 'complete' ? true : false"));

Examples related to java

Under what circumstances can I call findViewById with an Options Menu / Action Bar item? How much should a function trust another function How to implement a simple scenario the OO way Two constructors How do I get some variable from another class in Java? this in equals method How to split a string in two and store it in a field How to do perspective fixing? String index out of range: 4 My eclipse won't open, i download the bundle pack it keeps saying error log

Examples related to c#

How can I convert this one line of ActionScript to C#? Microsoft Advertising SDK doesn't deliverer ads How to use a global array in C#? How to correctly write async method? C# - insert values from file into two arrays Uploading into folder in FTP? Are these methods thread safe? dotnet ef not found in .NET Core 3 HTTP Error 500.30 - ANCM In-Process Start Failure Best way to "push" into C# array

Examples related to selenium

SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 81 session not created: This version of ChromeDriver only supports Chrome version 74 error with ChromeDriver Chrome using Selenium Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser Class has been compiled by a more recent version of the Java Environment How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium? How to make Firefox headless programmatically in Selenium with Python? element not interactable exception in selenium web automation Selenium Web Driver & Java. Element is not clickable at point (x, y). Other element would receive the click How do you fix the "element not interactable" exception?

Examples related to selenium-webdriver

SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 81 Selenium: WebDriverException:Chrome failed to start: crashed as google-chrome is no longer running so ChromeDriver is assuming that Chrome has crashed WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium? How to make Firefox headless programmatically in Selenium with Python? Selenium Web Driver & Java. Element is not clickable at point (x, y). Other element would receive the click How do you fix the "element not interactable" exception? Scrolling to element using webdriver? Only local connections are allowed Chrome and Selenium webdriver Check if element is clickable in Selenium Java

Examples related to webdriver

WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser Selenium Web Driver & Java. Element is not clickable at point (x, y). Other element would receive the click How do you fix the "element not interactable" exception? ImportError: No module named 'selenium' how to use List<WebElement> webdriver How to get attribute of element from Selenium? Selenium Finding elements by class name in python Open web in new tab Selenium + Python When running WebDriver with Chrome browser, getting message, "Only local connections are allowed" even though browser launches properly How can I start InternetExplorerDriver using Selenium WebDriver