How can I scroll a web page using selenium webdriver in python

Question

I am currently using selenium webdriver to parse through facebook user friends page and extract all ids from the AJAX script  But I need to scroll down to get all the friends  How can I scroll down in Selenium  I am using python

User · Answer

same method as shown here    in python you can just use   driver execute script  window scrollTo 0  Y       Y is the vertical position you want to scroll to

User · Answer

You can use send keys to simulate an END  or PAGE DOWN  key press  which normally scroll the page    from selenium webdriver common keys import Keys html   driver find element by tag name  html   html send keys Keys END

User · Answer

scroll loading pages  Example  medium  quora etc last height   driver execute script  quot return document body scrollHeight quot       while True          driver execute script  quot window scrollTo 0  document body scrollHeight-1000   quot             Wait to load the page          driver implicitly wait 30    seconds         new height   driver execute script  quot return document body scrollHeight quot                if new height    last height              break         last height   new height           sleep for 30s         driver implicitly wait 30    seconds     driver quit

User · Answer

You can use  driver execute script  window scrollTo 0  Y       where Y is the height  on a fullhd monitor it s 1080    Thanks to  lukeis   You can also use  driver execute script  window scrollTo 0  document body scrollHeight       to scroll to the bottom of the page   If you want to scroll to a page with infinite loading  like social network ones  facebook etc   thanks to  Cuong Tran   SCROLL PAUSE TIME   0 5    Get scroll height last height   driver execute script  return document body scrollHeight    while True        Scroll down to bottom     driver execute script  window scrollTo 0  document body scrollHeight            Wait to load page     time sleep SCROLL PAUSE TIME         Calculate new scroll height and compare with last scroll height     new height   driver execute script  return document body scrollHeight       if new height    last height          break     last height   new height   another method  thanks to Juanse  is  select an object and   label sendKeys Keys PAGE DOWN

User · Answer

If you want to scroll down to bottom of infinite page  like linkedin com   you can use this code   SCROLL PAUSE TIME   0 5    Get scroll height last height   driver execute script  return document body scrollHeight    while True        Scroll down to bottom     driver execute script  window scrollTo 0  document body scrollHeight            Wait to load page     time sleep SCROLL PAUSE TIME         Calculate new scroll height and compare with last scroll height     new height   driver execute script  return document body scrollHeight       if new height    last height          break     last height   new height   Reference  https   stackoverflow com a 28928684 1316860

User · Answer

This code scrolls to the bottom but doesn t require that you wait each time  It ll continually scroll  and then stop at the bottom  or timeout   from selenium import webdriver import time  driver   webdriver Chrome executable path  chromedriver exe   driver get  https   example com    pre scroll height   driver execute script  return document body scrollHeight    run time  max run time   0  1 while True      iteration start   time time         Scroll webpage  the 100 allows for a more  aggressive  scroll     driver execute script  window scrollTo 0  100 document body scrollHeight          post scroll height   driver execute script  return document body scrollHeight         scrolled   post scroll height    pre scroll height     timed out   run time  gt   max run time      if scrolled          run time   0         pre scroll height   post scroll height     elif not scrolled and not timed out          run time    time time   - iteration start     elif not scrolled and timed out          break    closing the driver is optional  driver close     This is much faster than waiting 0 5-3 seconds each time for a response  when that response could take 0 1 seconds

User · Answer

The ScrollTo   function doesn t work anymore  This is what I used and it worked fine   driver execute script  document getElementById  mydiv   scrollIntoView

User · Answer

insert this line driver execute script  quot window scrollBy 0 925  quot    quot  quot

User · Answer

Here s an example selenium code snippet that you could use for this type of purpose  It goes to the url for youtube search results on  Enumerate python tutorial  and scrolls down until it finds the video with the title   Enumerate python tutorial 2020    driver get  https   www youtube com results search query enumerate python   target   driver find element by link text  Enumerate python tutorial 2020     target location once scrolled into view

User · Answer

When working with youtube the floating elements give the value  0  as the scroll height so rather than using  return document body scrollHeight  try using this one  return document documentElement scrollHeight  adjust the scroll pause time as per your internet speed  else it will run for only one time and then breaks after that   SCROLL PAUSE TIME   1    Get scroll height    last height   driver execute script  return document body scrollHeight    this dowsnt work due to floating web elements on youtube      last height   driver execute script  return document documentElement scrollHeight   while True        Scroll down to bottom     driver execute script  window scrollTo 0 document documentElement scrollHeight            Wait to load page     time sleep SCROLL PAUSE TIME         Calculate new scroll height and compare with last scroll height     new height   driver execute script  return document documentElement scrollHeight       if new height    last height         print  break          break     last height   new height

User · Answer

None of these answers worked for me  at least not for scrolling down a facebook search result page  but I found after a lot of testing this solution   while driver find element by tag name  div        driver execute script  window scrollTo 0  document body scrollHeight         Divs driver find element by tag name  div   text     if  End of Results  in Divs          print  end          break     else          continue

User · Answer

I was looking for a way of scrolling through a dynamic webpage  and automatically stopping once the end of the page is reached  and found this thread   The post by  Cuong Tran  with one main modification  was the answer that I was looking for  I thought that others might find the modification helpful  it has a pronounced effect on how the code works   hence this post    The modification is to move the statement that captures the last page height inside the loop  so that each check is comparing to the previous page height    So  the code below      Continuously scrolls down a dynamic webpage   scrollTo     only stopping when  for one iteration  the page height stays the same     There is another modification  where the break statement is inside another condition  in case the page  sticks   which can be removed        SCROLL PAUSE TIME   0 5       while True             Get scroll height             This is the difference  Moving this  inside  the loop             means that it checks if scrollTo is still scrolling          last height   driver execute script  return document body scrollHeight              Scroll down to bottom         driver execute script  window scrollTo 0  document body scrollHeight                Wait to load page         time sleep SCROLL PAUSE TIME             Calculate new scroll height and compare with last scroll height         new height   driver execute script  return document body scrollHeight           if new height    last height                 try again  can be removed              driver execute script  window scrollTo 0  document body scrollHeight                    Wait to load page             time sleep SCROLL PAUSE TIME                 Calculate new scroll height and compare with last scroll height             new height   driver execute script  return document body scrollHeight                  check if the page height has remained the same             if new height    last height                    if so  you are done                 break               if not  move on to the next loop             else                  last height   new height                 continue

User · Answer

For my purpose  I wanted to scroll down more  keeping the windows position in mind  My solution was similar and used window scrollY  driver execute script  window scrollTo 0  window scrollY   200      which will go to the current y scroll position   200

User · Answer

driver execute script  document getElementById  your ID Element   scrollIntoView        it s working for my case

User · Answer

You can use send keys to simulate a PAGE DOWN key press  which normally scroll the page   from selenium webdriver common keys import Keys html   driver find element by tag name  html   html send keys Keys PAGE DOWN

User · Answer

This is how you scroll down the webpage   driver execute script  window scrollTo 0  1000

User · Answer

if you want to scroll within a particular view frame  WebElement   what you only need to do is to replace  body  with a particular element that you intend to scroll within  i get that element via  getElementById  in the example below   self driver execute script  window scrollTo 0  document getElementById  page-manager   scrollHeight       this is the case on YouTube  for example

User · Answer

element find element by xpath  xpath of the li you are trying to access    element location once scrolled into view   this helped when I was trying to access a  li  that was not visible

User · Answer

The easiest way i found to solve that problem was to select a label and then send   label sendKeys Keys PAGE DOWN     Hope it works

[python] How can I scroll a web page using selenium webdriver in python?

Examples related to python

Examples related to selenium

Examples related to selenium-webdriver

Examples related to automated-tests