I am attempting to download a file from the CDC website by clicking a button in a dropdown menu (I would just access the file URL directly, but the blob URL seems to change every time the download button is clicked when checking my download history on Chrome). This button can be found by clikcing the header "Data Table for Trends in Number of COVID-19 Vaccinations in the US" below the chart on https://covid.cdc.gov/covid-data-tracker/#vaccination-trends.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
import time
driver_path = "some_path"
vac_trend_url = "https://covid.cdc.gov/covid-data-tracker/#vaccination-trends"
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.get(vac_trend_url)
table_title_xpath = "/html/body/div[7]/div[2]/main/div[2]/div[1]/h4"
table_title = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, table_title_xpath)))
table_title.click()
download_button_xpath = "/html/body/div[7]/div[2]/main/div[2]/div[2]/div[1]/button/.."
download_button = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, download_button_xpath)))
download_button.click()
# ActionChains(driver).move_to_element(WebDriverWait(driver, 5).until(
# EC.element_to_be_clickable((By.XPATH, download_button_xpath)))).click().perform()
Although the first click to reveal the actual download button in the dropdown works, the second click to actually download the file does not.
Message: element click intercepted: Element is not clickable at point (1445, 310)
The title explains my error running the code above. When I try to uncomment the ActionChain line, I get the error described here despite using the very same method described in the post that should fix said error. The button does not seem to be overlayed with some other HTML element, so I am even more confused about what the issue is. Is there a workaround?
EDIT: I also tried MoveTargetOutOfBoundsException problem with chromedriver version >74 at another user's suggestion, still no dice.
This should simply download the file for you. By sending click to the element.
download_button_xpath = "/html/body/div[7]/div[2]/main/div[2]/div[2]/div[1]/button"
download_button = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, download_button_xpath)))
driver.execute_script("arguments[0].click();", download_button )
Related
I want to scrape links to news articles using scrapy + selenium. The website I am using uses a 'Load more' button, so I obviously want selenium to click on this button to load all articles.
I have looked for similar questions and tried various options already such as
element = driver.find_element(By.XPATH, value='//*[#id="fusion-app"]/main/div/div/div/div/div[4]/div/div/button')
driver.execute_script("arguments[0].click();", element)
and
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ais-InfiniteHits-loadMore")))
ActionChains(driver).move_to_element(element).click().perform()
All to no result. I've also inserted some print statements in between to check whether it does run the code, and that seems to work fine; I think it's just a matter of the button not being located/clicked on.
This is the html of the button btw:
<button class="ais-InfiniteHits-loadMore">Load more </button>
And when I print element, this is what I get: <selenium.webdriver.remote.webelement.WebElement (session="545716eef622a12bdbeddef99e02bdef", element="551741ec-4616-4bd4-b8fd-57c2f4bffb00")>
Is someone able to help me out? Thank you in advance.
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get('https://www.businessoffashion.com/search/?q=Louis+Vuitton&f=Articles%2CFashion+Shows%2CNews')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'button.ab-close-button'))).click()
elem=wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ais-InfiniteHits-loadMore")))
driver.execute_script("arguments[0].click()", elem)
You hit two different errors with a pop up and an element click intereception when you can just use javascript to click that element.
Import:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I've been trying to scrape this website Link I'm interested in clicking the first DONWLOAD button. However whenever I try to find any element that is a button, I can't find any.
Here is the code :
url = 'https://ember-climate.org/data/carbon-price-viewer/'
webdriver = create_driver()
with webdriver as driver:
driver.maximize_window()
driver.get(url)
wait = WebDriverWait(driver, 30)
try:
wait.until(EC.element_to_be_clickable((By.XPATH, '//button')))
except:
pass
ids = driver.find_elements_by_xpath('//button')
for ii in ids:
print (ii.tag_name, ii.get_attribute('class'))
Is there anything wrong with the XPath or is it an issue with the website itself?
Your xpath is wrong, Download button is wrapped inside an span tag not button tag.
try this instead :
//span[contains(text(),'DOWNLOAD')]
also, I see it's in iframe, which can be located via
iframe[name='ETS']
CSS_SELECTOR, and we need to switch also to this iframe.
so in sequence the explanation would be :
You would have to click on cookies button.
Download button is in an iframe, we need to switch to iframe first and then we can interact with download button.
download button is a part of span tag not button tag.
Use Explicit waits.
Prefer id, css over xpath. (if they are unique in nature)
Launch browser in full screen mode.
Code :
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(30)
wait = WebDriverWait(driver, 50)
driver.get("https://ember-climate.org/data/carbon-price-viewer/")
wait.until(EC.element_to_be_clickable((By.ID, "cn-accept-cookie"))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[name='ETS']")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[contains(text(),'DOWNLOAD')]"))).click()
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I am making a screen scraper using the Selenium Python library and I have already made some code so that I can log in. For some reason I am now stuck on the main menu and can't select any of the options. I have tried using CSS Selector, Class Name, and XPATH and none have been able to select any of the possible choices. No matter what happens, I always get a TimeoutException even with a long delay.
The portion of the page I am trying to scrape from is here.
The relevant code is as follows:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
# Open browser and go to the Webex login page
driver = webdriver.Chrome()
driver.get('https://admin.webex.com')
delay = 10 # seconds
long_delay = 20
# Login portion removed
# Menu selection goes here.
# I have tried the following with no luck
# The following lines produce a TimeoutException error
# Selecting menu item
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.XPATH, "span[#class='left-nav-item__link']"))).click()
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.XPATH, '//mch-left-nav-item-group[4]/ul/mch-left-nav-item[3]/li/span'))).click()
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.XPATH, "//webex-root/webex-main[#class='control-hub-container']//webex-sidebar/mch-left-nav/nav/mch-left-nav-item-group[4]/ul/mch-left-nav-item[3]//span[#class='left-nav-item__link']"))).click()
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "span[class='left-nav-item__link']"))).click()
WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "[aria-label] mch-left-nav-item-group:nth-of-type(4) mch-left-nav-item:nth-of-type(3) .left-nav-item__link"))).click()
# Selecting group of items
WebDriverWait(driver, delay).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'left-nav-item')))
WebDriverWait(driver, delay).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'li.left-nav-item')))
# Selecting parent
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "li[data-test-name='calling']")))
Does anyone have an idea why I'm not able to select any element?
Try xpath for the third one:
WebDriverWait(driver, long_delay).until(EC.presence_of_element_located((By.XPATH, "span[#class='left-nav-item__link']"))).click()
To anyone having this issue.
When a new tab is opened while you are using Selenium, even though you are viewing the new page, Selenium is not. You have to switch to the correct window. Something such as the following can help you with that.
driver.switch_to.window(driver.window_handles[NUMBER])
I am having trouble selecting a load more button on a Linkedin page. I receive this error in finding the xpath: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element
I suspect that the issue is that the button is not visible on the page at that time. So I have tried actions.move_to_element. However, the page scrolls just below the element, so that the element is no longer visible, and the same error subsequently occurs.
I have also tried move_to_element_with_offset, but this hasn't changed where the page scrolls to.
How can I scroll to the right location on the page such that I can successfully select the element?
My relevant code:
import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('C:\\Users\\Root\\Downloads\\chromedriver.exe')
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)
username = driver.find_element_by_name('session_key')
username.send_keys(parameters.linkedin_username)
sleep(0.5)
password = driver.find_element_by_name('session_password')
password.send_keys(parameters.linkedin_password)
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//button[#class="btn__primary--large from__button--floating"]')
sign_in_button.click()
driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')
loadmore_skills=driver.find_element_by_xpath('//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')
actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()
After playing around with it, I seem to have figured out where the problem is stemming from. The error
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]"}
(Session info: chrome=81.0.4044.113)
always correctly states the problem its encountering and as such it's not able to find the element. The possible causes of this include:
Element not present at the time of execution
Dynamically generated
content Conflicting names
In your case, it was the second point. As the content that is displayed is loaded dynamically as you scroll down. So When it first loads your profile the skills sections aren't actually present in the DOM. So to solve this, you simply have to scroll to the section so that it gets applied in the DOM.
This line is the trick here. It will position it to the correct panel and thus loading and applying the data to the DOM.
driver.execute_script("window.scrollTo(0, 1800)")
Here's my code (Please change it as necessary)
from time import sleep
# import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('../chromedriver.exe')
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)
username = driver.find_element_by_name('session_key')
username.send_keys('')
sleep(0.5)
password = driver.find_element_by_name('session_password')
password.send_keys('')
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//button[#class="btn__primary--large from__button--floating"]')
sign_in_button.click()
driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')
sleep(3)
# driver.execute_script("window.scrollTo(0, 1800)")
sleep(3)
loadmore_skills=driver.find_element_by_xpath('//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')
actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()
Output
Update
In concerns to your newer problem, you need to implement a continuous scroll method that would enable you to dynamically update the skills section. This requires a lot of change and should ideally be asked as a another question.
I have also found a simple solution by setting the scroll to the correct threshold. For y=3200 seems to work fine for all the profiles I've checked including yours, mine and few others.
driver.execute_script("window.scrollTo(0, 3200)")
If the button is not visible on the page at the time of loading then use the until method to delay the execution
try:
myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
print "Button is rdy!"
except TimeoutException:
print "Loading took too much time!"
Example is taken from here
To get the exact location of the element, you can use the following method to do so.
element = driver.find_element_by_id('some_id')
element.location_once_scrolled_into_view
This actually intends to return you the coordinates (x, y) of the element on-page, but also scroll down right to target element. You can then use the coordinates to make a click on the button. You can read more on that here.
You are getting NoSuchElementException error when the locators (i.e. id / xpath/name/class_name/css selectors etc) we mentioned in the selenium program code is unable to find the web element on the web page.
How to resolve NoSuchElementException:
Apply WebDriverWait : allow webdriver to wait for a specific time
Try catch block
so before performing action on webelement you need to take web element into view, I have removed unwated code and also avoided use of hardcoded waits as its not good practice to deal with synchronization issue. Also while clicking on show more button you have to scroll down otherwise it will not work.
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome(executable_path="path of chromedriver.exe")
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("email id")
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("password ")
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//button[#class='btn__primary--large from__button--floating']"))).click()
driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()
driver.execute_script("scroll(0, 250);")
buttonClick = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//span[text()='Show more']")))
ActionChains(driver).move_to_element(buttonClick).click().perform()
Output:
So I'm scraping using selenium and I want to click 'next' button in 'Defensive' section but the code I wrote clicks 'next' on 'Summary'.
Here's the url for you to try :
https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019
So it's selecting 'Defensive' and I can see it selected in the window but the next page doesnt appear. On clicking 'Summary' I found out next function is actually happening there.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser= webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe")
browser.get('https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019')
browser.find_element_by_xpath("""//*[#id="stage-top-player-stats-options"]/li[2]/a""").click()
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, """//*[#id="next"]""")))
browser.execute_script("arguments[0].click();", element)
The xpath for next button is not unique for this page. try this,
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//*[#id='stage-top-player-stats-defensive']//a[#id='next']")))
browser.execute_script("arguments[0].click();", element)
or
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//*[#id='stage-top-player-stats-defensive']//a[#id='next']")))
element.click()
For each tab (Summary, Defensive, ..) new next button with same id=next added to the DOM.
Select Defensive and you will see there will be two next buttons with same id=next, select Offensive and there will be three next buttons.
With basic id=next selector you always click to the first next button from Summary tab. Because you're using JavaScript and nothing happen, try to click with Selenium click method and you will get an error.
To solve the problem adjust your selector to be more specific to the dom - #statistics-paging-defensive #next.
Also when you first time open the page there's cookies acceptance screen appears and block the page, you can use method like below to skip it.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import selenium.common.exceptions as EX
def accept_cookies():
try:
WebDriverWait(browser, 20)\
.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.qc-cmp-button")))\
.click()
except EX.NoSuchElementException or EX.TimeoutException:
pass
#...
browser = webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe")
browser.get('https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019')
wait = WebDriverWait(browser, 20)
browser.get(baseUrl)
accept_cookies()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[href='#stage-top-player-stats-defensive']"))).click()
next_button = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "#statistics-paging-defensive #next")))
next_button.click()
Your elements locators must be unique
Avoid using XPath wildcards - * as it will cause performance degradation and prolonged elements lookup timings
Avoid using JavaScriptExecutor for clicking, well-behaved Selenium test must do what real user does and I doubt that real user will be opening browser console and typing something like document.getElementById('next').click(), he will use the mouse
Assuming all above you should come up with a selector which uniquely identifies next button on Defensive tab which would be something like:
//div[#id='statistics-paging-defensive']/descendant::a[#id='next']
References:
XPath Tutorial
XPath Axes
XPath Operators & Functions