I want to scroll a webpage upto a certain length I am trying to scroll the webpage with this line of code
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
It is executing very well but this code scroll the webpage till last. But I want to scroll till 90% of the page. Any idea how to approach this kind of problem. A little advice will also be appreciated. Thanks!
Related
I am trying to scrape with Selenium but I need to load all the content of the page by moving to the end of the website. But when I execute: driver.execute_script("window.scrollTo(0,document.body.scrollHeight);")
the program doesn't do anything at all. I wonder if it's because the page that I'm scraping has a personalized scrollbar.
That's because document.body.scrollHeight is zero, so it doesn't scroll anything.
You can scroll to an arbitrary large value to scroll down or instead use document.documentElement.scrollHeight.
Check this question for more details.
How to scroll down specific area of webpage.
Because i need to scroll down on linkedin message section.
NOT scroll down entire screen. Scroll down on specific area.
Please help me.
Please check image. CLICK HERE
Use :
selenium driver.execute_script() for running js script
javascript or jQuery method scroll
css selectors for finding your element in the list
I need to get to the last page in the following site link. This is because when I right click on a row I would be able to export all previous rows along with it as a csv file. That way, I can download the complete data present in the website.
But the problem is that the Go to last page option doesn't work. So I am currently using selenium to click through the next page button to eventually reach the last page. But it takes a lot of time.
This is the code I am currently using.
from selenium import webdriver
import time
url = "https://mahabocw.in/essential-kit-benefits-distribution/"
driver = webdriver.Chrome()
driver.get(url)
for i in range(1000000):
next_button = '/html/body/div[1]/div[6]/div/article/div/div/div/div/div[2]/div/div/div[2]/div/div[4]/span[2]/div[3]/button'
click_next = driver.find_element_by_xpath(next_button)
click_next.click()
Is there any way I could modify the code in the website and maybe make the particular button : Go to last page to work so I can go to that page and download all the data. Or is there some better technique I could adopt using Selenium.
Any help or suggestions would be really helpful. Thanks a lot in advance.
I have a fully working Python script for taking screenshots of webpages using selenium and the FF driver(windows).
The script is taking full page screenshots instead of let's say 1024x768, which make the process slow. Here is my code:
driver = webdriver.Firefox()
driver.set_window_size(1024,768)
driver.get(url)
driver.get_screenshot_as_file("%s/%s.png"%(screen_dir,o.netloc))
driver.close()
Any ideas on how to make selenium/firefox take above the fold screenshots and not full page?
Thanks!
This is how the "take screenshot" functionality is working in Firefox - it would take a screenshot of a complete page from top to bottom even if part of it is not visible. This is actually not how it supposed to work by the specification.
Chrome would take a screenshot of the visible area / viewport only. Switch to it.
I'm trying to make a little script which takes a look at main page of web and finds adds.
The problem is that there are web pages which contains infinite scroll. If this code was built for particular web page, I could use locating elements and scrolling.
But I can't figure out how to make Selenium to scroll at the very bottom of any page?
self.driver.execute_script("window.scrollTo(0, something);")
PS: If there is very huge page, break it down after several seconds of scrolling.
Do you know how to do that?
Here's another method that i used for Java, get the window size and then scroll to that position using javascript. Here's how to do it in Java (hope you can implement the concept in python too) -
double pageHeight = testBase.TestBase.driver.manage().window().getSize().getHeight();
driver.executeScript("window.scrollBy(0,"+pageHeight+")");
If you are implementing an infinite scroll then you can put the executeScript() lines in a loop. Hope it helps.