I hope you're doing well!
I want to scrap this website https://proadvisor.intuit.com/app/accountant/search?region=ca&distance=1000, if you click on it the site load normally but if I run my script appears something like a "pop-up" window (I attached the evidence).
pop-up window
I'm using Selenium and Python to do it. I want to do click on the 'x' button to close that and then I can start to scrap the website so I tried with the next two ways with the XPATH:
close_button = driver.find_element(By.XPATH, '/html/body/div[4]/button')
close_button.click()
driver.find_element(By.XPATH, '//div[contains(#class,"coach-marks")]//button[#aria-label="close"]').click()
But the result was the same, the next error is launched and the "pop-up" window doesn't close.
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[4]/button"}
The HTML code is:
<button class="a--close_button" aria-label="close" tabindex="0">
<svg class="svg--close_button" width="19" height="19" viewBox="0 0 19 19">
<path
fill-rule="evenodd"
d="M19 1.83L17.17 0 9.5 7.67 1.83 0 0 1.83 7.67 9.5 0 17.17 1.83 19l7.67-7.67L17.17 19 19 17.17 11.33 9.5z"
></path>
</svg>
</button>
All my Python code is:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from webdriver_manager.chrome import ChromeDriverManager
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
driver.get("https://proadvisor.intuit.com/app/accountant/search?region=ca&distance=1000")
# click to close button
close_button = driver.find_element(By.XPATH, '/html/body/div[4]/button')
close_button.click()
If you can help me I will very grateful with you.
Regards.
Wait for the element to become visible before trying to interact with it. You can do this by using an explicit wait with the expected_conditions module from "selenium.webdriver.support.ui".
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
driver.get("https://proadvisor.intuit.com/app/accountant/search?region=ca&distance=1000")
# wait for the close button to be visible and click it
wait = WebDriverWait(driver, 10)
close_button = wait.until(EC.visibility_of_element_located((By.XPATH, '//button[#aria-label="close"]')))
close_button.click()
Related
I'm using selenium to try and scrape a listing of products in this website:
https://www.zonacriativa.com.br/harry-potter
However, I'm having trouble getting the full listing of products. the page list 116 products, yet only a few are shown at a time. If I want to see the other ones, I need to click on the "Carregar mais Produtos" (load more products) button at the bottom a few times to get the full listing.
I'm having trouble locating this button, as it doesn't have an id and its class is a huge string. I've tried several things, like the examples below, but they don't seem to work. Any suggestions?
driver.find_element("xpath", "//button[text()='Carregar mais Produtos']").click()
driver.find_element("css selector", ".vtex-button__label.flex.items-center.justify-center.h-100.ph5").click()
driver.find_element(By.CLASS_NAME, "vtex-button.bw1.ba.fw5.v-mid.relative.pa0.lh-solid.br2.min-h-small.t-action--small.bg-action-primary.b--action-primary.c-on-action-primary.hover-bg-action-primary.hover-b--action-primary.hover-c-on-action-primary.pointer").click()
The element you trying to click is initially out of the visible screen so you can't click it. Also this XPath at least for me doesn't locate that element.
What you need to do is to scroll the page down untill that button becomes visible and clickable and then click it.
The following code clicks that button 1 time:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 5)
url = "https://www.zonacriativa.com.br/harry-potter"
driver.get(url)
while True:
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(#class,'buttonShowMore')]//button"))).click()
break
except:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
The above code can be simply modified to scroll and click that button until we reach the latest page where this button is not presented:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 5)
url = "https://www.zonacriativa.com.br/harry-potter"
driver.get(url)
while driver.find_elements(By.XPATH, "//div[contains(#class,'buttonShowMore')]//button"):
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(#class,'buttonShowMore')]//button"))).click()
except:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
I am trying to scraping in this URL, dealing with a Download button and I am having a problem, as the last line gives a ElementClickInterceptedException.
My actual goal is to download the CSV file.
The code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.chrome.options import Options
from ipykernel import kernelapp as app
import time
options = webdriver.ChromeOptions()
driver_path = 'C:\\Users\\Idener\\Downloads\\chromedriver_win32\\chromedriver.exe'
driver = webdriver.Chrome(driver_path, options=options)
url = "https://pubchem.ncbi.nlm.nih.gov/compound/2078"
driver.get(url)
wait = WebDriverWait(driver, 5)
wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="NIOSH-Toxicity-Data"]/div[1]/div/div/a'))).click()
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="Download"]'))).click()
enter image description here
Element you trying to click in initially out of the visible viewpoint. So, you need first to scroll the page and only then to click on that element.
By clicking the first element new tab is opened and the second element you want to click is there, on the second tab. So, you need to switch to the new tab to access that element.
No need to define wait = WebDriverWait(driver, 10) second time.
The following code is working:
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://pubchem.ncbi.nlm.nih.gov/compound/2078#section=Toxicity"
driver.get(url)
element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#NIOSH-Toxicity-Data a[title*='Open']")))
element.location_once_scrolled_into_view
time.sleep(1)
element.click()
driver.switch_to.window(driver.window_handles[1])
wait.until(EC.element_to_be_clickable((By.ID, "Download"))).click()
It does not download the file, only opens the downloading dialog
I'm trying to scrape href tags that are hidden under a #shadow-root (open) on this website: https://iltacon2022.expofp.com/. I'm new to coding and was wondering if someone could help me out.
Here is the code I've been trying to access the #shadow-root (open), but I'm in front of my skies on what to do next.
Code:
from bs4 import BeautifulSoup as bs
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
url = "https://iltacon2022.expofp.com/"
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("detach", True)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
driver.get(url)
time.sleep(6)
root1 = driver.find_element(By.XPATH,"/html/body/div[1]/div").shadow_root
root2 = driver.find_element(By.XPATH,"/html/body/div[1]/div//div")
print(root2)
driver.quit()
error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[1]/div//div"}
Desired output:
?access-corp
?accruent-inc
?aceds-association-of-certified-e-discovery-specialists
?actionstep
?aderant
etc
etc
etc
I'm trying to click on the "next page" button using selenium, but I'm having no success. Am I using the right CSS selector or should I change it to something else?
from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import pandas as pd
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument("--enable-javascript")
options.add_argument('--no-sandbox')
options.add_argument("window-size=1200x600")
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
driver.get(f'https://www.stakingrewards.com/cryptoassets')
driver.implicitly_wait(10)
button = driver.find_element(By.CSS_SELECTOR,"button[data-testid='next-page-button']")
button.click()
driver.quit()
change
button = driver.find_element(By.CSS_SELECTOR,"button[data-testid='next-page-button']")
to
button = driver.find_element(By.XPATH, "//button[#data-testid='next-page-button']")
You were previously trying to select with an XPATH, but had it as CSS selector. It also needed relative pathing/corrections. Let me know if there's anything else I can help with or missed.
I want to use Selenium in Python 3.9 to click on a button.
https://www.sepaq.com/en/reservation/national-parks/annual-card
I want to click on Effective Date and click on July 2021 then click ok.
The first two was working, then I added the code to click the OK button and I can no longer click on July 2021. Interesting the click OK step also works.
Error I am getting this error:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
(Session info: chrome=90.0.4430.212)
I have tried:
sleep statement to make it wait for the button to become available
Web Driver wait
Action Chain to make it scroll into view
I am using Selenium IDE with chrome to give me the css tags.
from time import sleep
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
def print_hi():
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
chrome_options.add_argument("start-maximized")
chrome_options.add_argument("disable-infobars")
chrome_options.add_argument("--disable-extensions")
annual_pass = webdriver.Chrome(options=chrome_options)
annual_pass.get("https://www.sepaq.com/en/reservation/national-parks/annual-card")
type = annual_pass.find_element_by_css_selector('.form-list:nth-child(2) > li:nth-child(1) .form-label')
type.click()
annual_pass.get("https://www.sepaq.com/en/reservation/national-parks/annual-card")
open_eff_date_panel = annual_pass.find_element_by_link_text('Select')
open_eff_date_panel.click()
annual_pass.get("https://www.sepaq.com/en/reservation/national-parks/annual-card")
selectJuly = annual_pass.find_element_by_css_selector('li:nth-child(3) .form-label')
#selectJuly = annual_pass.find_element_by_xpath('//*[#id="date2"]')
selectJuly.click() ## error here##
annual_pass.get("https://www.sepaq.com/en/reservation/national-parks/annual-card")
ok_button = annual_pass.find_element_by_css_selector('#reserver-date .bouton')
ok_button.click()
sleep(10)
annual_pass.close()
if __name__ == '__main__':
print_hi()
Clicking toggles may be unstable with Selenium.
First, try this approach:
selectJuly = annual_pass.find_element_by_css_selector('#reserver .panel.is-active>.panel-middle li:nth-child(3) .form-label')
annual_pass.execute_script("arguments[0].click();", selectJuly)
ok_button = annual_pass.find_element_by_css_selector('#reserver-date .bouton')
ok_button.click()
It works, but I do not like that the window is not completely rolled out.
Full code with debugging info left:
import time
from time import sleep
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.remote.webelement import WebElement
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
def print_hi():
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
chrome_options.add_argument("start-maximized")
chrome_options.add_argument("disable-infobars")
chrome_options.add_argument("--disable-extensions")
annual_pass = webdriver.Chrome(executable_path='/snap/bin/chromium.chromedriver', options=chrome_options)
annual_pass.get("https://www.sepaq.com/en/reservation/national-parks/annual-card")
open_eff_date_panel = annual_pass.find_element_by_link_text('Select')
open_eff_date_panel.click()
# wait = WebDriverWait(open_eff_date_panel, 10)
# wait.until(EC.visibility_of_element_located(
# (By.CSS_SELECTOR, "//html[#class='js history es5array video opacity csspointerevents placeholder inlinesvg es5date supports es5function es5object strictmode es5string json es5syntax es5undefined es5 no-touchevents cssvhunit mediaqueries vibrate csscolumns csscolumns-width csscolumns-span csscolumns-fill csscolumns-gap csscolumns-rule csscolumns-rulecolor csscolumns-rulestyle csscolumns-rulewidth csscolumns-breakbefore csscolumns-breakafter csscolumns-breakinside flexbox csstransforms csstransforms3d no-ios panel-active']")))
selectJuly = annual_pass.find_element_by_css_selector(
'#reserver .panel.is-active>.panel-middle li:nth-child(3) .form-label')
annual_pass.execute_script("arguments[0].click();", selectJuly)
ok_button = annual_pass.find_element_by_css_selector('#reserver-date .bouton')
ok_button.click()
# annual_pass.close()
if __name__ == '__main__':
print_hi()
2 Also, you can try implementing your own function to wait till some css property of your form is changed (for example width). For this use Selenium's value_of_css_property.
3
If you wait for all classes to become available in xpath, it will also work, but does not look robust. You can change this locator into css and exclude classes one by one to make it shorter. Waiting just for panel-active class won't work:
wait = WebDriverWait(open_eff_date_panel, 10)
wait.until(EC.visibility_of_element_located(
(By.XPATH, "//html[#class='js history es5array video opacity csspointerevents placeholder inlinesvg es5date supports es5function es5object strictmode es5string json es5syntax es5undefined es5 no-touchevents cssvhunit mediaqueries vibrate csscolumns csscolumns-width csscolumns-span csscolumns-fill csscolumns-gap csscolumns-rule csscolumns-rulecolor csscolumns-rulestyle csscolumns-rulewidth csscolumns-breakbefore csscolumns-breakafter csscolumns-breakinside flexbox csstransforms csstransforms3d no-ios panel-active']")))
selectJuly = annual_pass.find_element_by_css_selector('#reserver .panel.is-active>.panel-middle li:nth-child(3) .form-label')
selectJuly.click()
I tried as well, wait.until(lambda driver: driver.execute_script("return jQuery.active == 0")) but it did not work for this case.
I had a similar question: Cannot click on toggle with selenium because of ElementNotInteractableException