I am trying to automate downloads from a webpage using selenium.
So far my strategy is to instantiate a firefox driver that loads the page and clicks the download button. However, to be clickable, the button needs to be visible, i.e. not covered by any banners on the page (at least in my understanding). Therefore I need to scroll down (I use scrollIntoView()). If I run button.click() immediately after scrolling down, the download doesn't start, if I hard code a sufficient timeout in between it works out fine. Can somebody help me to set a timeout conditioned on the scroll down?
Here is my code:
from selenium import webdriver
profilePath = '/path/to/my/firefox/profile'
profile = webdriver.FirefoxProfile(profilePath)
driver = webdriver.Firefox(firefox_profile=profile)
driver.get("https://www.happyscribe.com/public/lex-fridman-podcast-artificial-intelligence-ai/164-andrew-huberman-sleep-dreams-creativity-the-limits-of-the-human-mind")
button = driver.find_element_by_id('btn-download')
target=driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", button)
time.sleep(10)
button.click()
and this is the html code of the button:
<button class="hs-btn-secondary small" id="btn-download" type="button">Download</button>
I would be very thankful for any direct help or suggestions on how to tackle the problem from a different angle.
It sounds like you may need to induce WebDriverWait for the element to by clickable. You could do so like this. It will wait for the element to be both visible and enabled before clicking.
from selenium import webdriver
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
profilePath = '/path/to/my/firefox/profile'
profile = webdriver.FirefoxProfile(profilePath)
driver = webdriver.Firefox(firefox_profile=profile)
driver.get("https://www.happyscribe.com/public/lex-fridman-podcast-artificial-intelligence-ai/164-andrew-huberman-sleep-dreams-creativity-the-limits-of-the-human-mind")
button = driver.find_element_by_id('btn-download')
target=driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", button)
WebDriverWait(driver, 10).until(EC.element_to_be_clickable(button))
button.click()
Related
I'm trying to scrap the list of services we have for us from this site but not able to click to the next page.
This is what I've tried so far using selenium & bs4,
#attempt1
next_pg_btn = browser.find_elements(By.CLASS_NAME, 'ui-lib-pagination_item_nav')
next_pg_btn.click() # nothing happens
#attemp2
browser.find_element(By.XPATH, "//div[#role = 'button']").click() # nothing happens
#attempt3 - saw in some stackoverflow post that sometimes we need to scroll to the
#bottom of page to have the button clickable, so tried that
browser.execute_script("window.scrollTo(0,2500)")
browser.find_element(By.XPATH, "//div[#role = 'button']").click() # nothing happens
I'm not so experienced with scrapping, pls advice how to handle this and where I'm going wrong.
Thanks
Several issues with your code:
You tried wrong locators.
You probably need to wait for the element to be loaded before clicking it. But if before clicking the pagination you performing some actions on the page this is not needed since during you scraping the page content web elements are already got loaded.
Pagination button is on the buttom of the page, so you need to scroll the page to bring the pagination button into the visible screen.
After scrolling some delay should be added, as you can see in the code below.
Now pagination element can be clicked.
The following code works
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://www.tamm.abudhabi/en/life-events/individual/HousingProperties"
driver.get(url)
pagination = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "ui-lib-pagination__item_nav")))
pagination.location_once_scrolled_into_view
time.sleep(0.5)
pagination.click()
I want to scrape links to news articles using scrapy + selenium. The website I am using uses a 'Load more' button, so I obviously want selenium to click on this button to load all articles.
I have looked for similar questions and tried various options already such as
element = driver.find_element(By.XPATH, value='//*[#id="fusion-app"]/main/div/div/div/div/div[4]/div/div/button')
driver.execute_script("arguments[0].click();", element)
and
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ais-InfiniteHits-loadMore")))
ActionChains(driver).move_to_element(element).click().perform()
All to no result. I've also inserted some print statements in between to check whether it does run the code, and that seems to work fine; I think it's just a matter of the button not being located/clicked on.
This is the html of the button btw:
<button class="ais-InfiniteHits-loadMore">Load more </button>
And when I print element, this is what I get: <selenium.webdriver.remote.webelement.WebElement (session="545716eef622a12bdbeddef99e02bdef", element="551741ec-4616-4bd4-b8fd-57c2f4bffb00")>
Is someone able to help me out? Thank you in advance.
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get('https://www.businessoffashion.com/search/?q=Louis+Vuitton&f=Articles%2CFashion+Shows%2CNews')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'button.ab-close-button'))).click()
elem=wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ais-InfiniteHits-loadMore")))
driver.execute_script("arguments[0].click()", elem)
You hit two different errors with a pop up and an element click intereception when you can just use javascript to click that element.
Import:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
So I'm scraping using selenium and I want to click 'next' button in 'Defensive' section but the code I wrote clicks 'next' on 'Summary'.
Here's the url for you to try :
https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019
So it's selecting 'Defensive' and I can see it selected in the window but the next page doesnt appear. On clicking 'Summary' I found out next function is actually happening there.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser= webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe")
browser.get('https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019')
browser.find_element_by_xpath("""//*[#id="stage-top-player-stats-options"]/li[2]/a""").click()
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, """//*[#id="next"]""")))
browser.execute_script("arguments[0].click();", element)
The xpath for next button is not unique for this page. try this,
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//*[#id='stage-top-player-stats-defensive']//a[#id='next']")))
browser.execute_script("arguments[0].click();", element)
or
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//*[#id='stage-top-player-stats-defensive']//a[#id='next']")))
element.click()
For each tab (Summary, Defensive, ..) new next button with same id=next added to the DOM.
Select Defensive and you will see there will be two next buttons with same id=next, select Offensive and there will be three next buttons.
With basic id=next selector you always click to the first next button from Summary tab. Because you're using JavaScript and nothing happen, try to click with Selenium click method and you will get an error.
To solve the problem adjust your selector to be more specific to the dom - #statistics-paging-defensive #next.
Also when you first time open the page there's cookies acceptance screen appears and block the page, you can use method like below to skip it.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import selenium.common.exceptions as EX
def accept_cookies():
try:
WebDriverWait(browser, 20)\
.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.qc-cmp-button")))\
.click()
except EX.NoSuchElementException or EX.TimeoutException:
pass
#...
browser = webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe")
browser.get('https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019')
wait = WebDriverWait(browser, 20)
browser.get(baseUrl)
accept_cookies()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[href='#stage-top-player-stats-defensive']"))).click()
next_button = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "#statistics-paging-defensive #next")))
next_button.click()
Your elements locators must be unique
Avoid using XPath wildcards - * as it will cause performance degradation and prolonged elements lookup timings
Avoid using JavaScriptExecutor for clicking, well-behaved Selenium test must do what real user does and I doubt that real user will be opening browser console and typing something like document.getElementById('next').click(), he will use the mouse
Assuming all above you should come up with a selector which uniquely identifies next button on Defensive tab which would be something like:
//div[#id='statistics-paging-defensive']/descendant::a[#id='next']
References:
XPath Tutorial
XPath Axes
XPath Operators & Functions
I've written a script that successfully makes the login on Instagram.
When I should go on my account, at home, the website displays a popup that asks you if you want notifications.
At this point, I tried a lot of solutions, but I got nothing.
I just want that, when the pop-up is displayed, the script should click on "Not now".
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
ids = []
driver = webdriver.Chrome(executable_path = '/usr/local/bin/chromedriver')
driver.get("https://www.instagram.com/accounts/login/?source=auth_switcher")
usm = driver.find_element_by_name('username').send_keys("**")
pwd = driver.find_element_by_name('password').send_keys("**")
btnLog = driver.find_element_by_tag_name('form').submit()
acpt = driver.find_element_by_xpath("//*[contains(#class, 'aOOlW HoLwm ')]")
In the image, there's the line of the button highlighted that I want to click:
Try the following code for this:
from selenium.webdriver.support import ui
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
ui.WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".aOOlW.HoLwm"))).click()
PS: I have used 10-second wait for the element to be clickable before click on it.
Hope it helps you!
To click() on the element with text as Not now on Instagram popup notification you can use the following solution:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.instagram.com/accounts/login/?source=auth_switcher")
driver.find_element_by_name('username').send_keys("Giacomo")
driver.find_element_by_name('password').send_keys("Maraglino")
driver.find_element_by_tag_name('form').submit()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(.,'Non ora')]"))).click()
just add click to acpt, like this:
acpt = driver.find_element_by_xpath("//*[contains(#class, 'aOOlW HoLwm ')]")
acpt.click()
For some reason, the solution to this question didn't fully work for me. The script does click on "not now" then I'm redirected to the "Home" page only to find the 'activate notifications' pop-up waiting for me.
This is the solution that I came up with:
Go to https://instagram.com/
Wait for the page to load.
Find the "Not now" button on the button using the full XPATH.
Clicking on it.
Here's the code (inserted after identifying to my Instagram account):
driver.get("https://instagram.com/")
time.sleep(7)
acpt = browser.find_element(by=By.XPATH, value='/html/body/div[1]/div/div/div/div[2]/div/div/div[1]/div/div[2]/div/div/div/div/div/div/div/div[3]/button[2]')
acpt.click()
After handling a pop up overlay the rest of the code no longer works.
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('https://www.vapordna.com/login.asp')
#handling popup
browser.find_element_by_css_selector("div.age-verify-button.age-modal__btn.btn.btn-primary").click()
browser.find_element_by_link_text("Yes").click()
If I don't close the pop up the rest of the code works, if I close the popup none of the code after that point works.
emailElem = browser.find_element_by_name("email")
emailElem.send_keys("****#****.com")
passwordElem = browser.find_element_by_name("password")
passwordElem.send_keys("******")
passwordElem.submit()
I have tried adding this line but it doesn't help,
browser.switchTo().defaultContent();
To my understanding the popup isn't in a new iframe or window so I'm not sure why there would be any problem with continuing to interact with the page.
I also tried clicking on the body before continuing
browser.find_element_by_tag_name("body").click()
After seeing your website I observe this is neither an frame nor a window, I just a form so there is no need to use here browser.switchTo().defaultContent(); which actually using for switching from any frame to default, Actually when you are going to find this button on the opened popup which overlay all the page, It would not be full loaded on the page, so you should try using WebDriverWait to wait until this button visible on the page.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Firefox()
browser.get('https://www.vapordna.com/login.asp')
wait = WebDriverWait(driver, 10)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.age-verify-button.age-modal__btn.btn.btn-primary"))).click()
Now after clicking on this agreement button you can proceed further steps for login.