First post on here and brand new to Python. I am trying to learn how to scrape data from a website. When you first load the website, a disclaimer window shows up and all I am trying to do is hit the accept button using the browser.find_element_by_id.
I am using the webdriverwait command to wait for the page to load prior to clicking the "Accept" button but I keep getting a Timeoutexception. Here is the code that I currently have:
from selenium import webdriver
#get the chrome webdriver path file
browser = webdriver.Chrome(executable_path=r"C:/Program Files (x86)/Google/Chrome/Application/chromedriver.exe")
browser.get('http://foreclosures.guilfordcountync.gov/')
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
#wait until element is loaded
wait = WebDriverWait(browser, 10)
wait.until(EC.presence_of_element_located((By.ID, "cmdAccept")))
element = browser.find_element_by_id("cmdAccept")
element.click()
Here is the error I keep getting:
Traceback (most recent call last):
File "C:/Users/Abbas/Desktop/Foreclosure_Scraping/Foreclosure_Scraping.py", line 33, in <module>
wait.until(EC.presence_of_element_located((By.ID, "cmdAccept")))
File "C:\Users\Abbas\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
I believe it has something to do with the calling out the ID of the button itself from the website but I honestly do not know. Any help is greatly appreciated.
Your attempts to locate the element are unsuccessful because of they are nested within an iframe. One must tell selenium to switch to the iframe that contains the desired element before attempting to click it or use it in any way. Try the following:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
#get the chrome webdriver path file
browser = webdriver.Chrome(executable_path=r"C:/Program Files (x86)/Google/Chrome/Application/chromedriver.exe")
browser.get('http://foreclosures.guilfordcountync.gov/')
browser.switch_to.frame(browser.find_element_by_name("ctl06"))
wait = WebDriverWait(browser, 10)
wait.until(EC.presence_of_element_located((By.ID, "cmdAccept")))
element = browser.find_element_by_id("cmdAccept")
element.click()
Related
I'm very newbie in Python and I'm trying to make login on Dell OpenManage webpage below by using Selenium:
Dell OpenManage
I already tried too many ways to locate the xpath of the Username box (/html/body/div1/div[2]/div/div/div1/div[6]/form/table/tbody/tr[2]/td1/input), but without success.
In the latest test I tried to wait the element be loaded to use send_keys and I received the error message below:
Traceback (most recent call last):
File "C:\LEARN\PY\RepositoryGitHub\Ancora-Automations\Ancora-Automations\ServersCheck.py", line 18, in
wait.until(EC.presence_of_element_located((By.XPATH, "/html/body/div1/div[2]/div/div/div1/div[6]/form/table/tbody/tr[2]/td1/input"))).send_keys('ezequiel.ferreira')
File "C:\Users\ezequiel.ferreira\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\support\wait.py", line 90, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
My code is:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import tkinter as tk
from tkinter import simpledialog
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service)
driver.get('https://ancorabk02:1311/OMSALogin?msgStatus=null')
wait = WebDriverWait(driver, 2)
# Clicking on advanced and continue buttons to proceed on a untrusted page
driver.find_element('xpath', '/html/body/div/div[2]/button[3]').click()
driver.find_element('xpath', '/html/body/div/div[3]/p[2]/a').click()
wait.until(EC.presence_of_element_located((By.XPATH, "/html/body/div[1]/div[2]/div/div/div[1]/div[6]/form/table/tbody/tr[2]/td[1]/input"))).send_keys('ezequiel.ferreira')
Could you please help me to understand with this issue?
Thanks!
I am new to web scraping. For a little project i decided to try and log into instagram automatically. I am using chrome web driver to open the page and selenium to process it. i can get selenium to open Instagram but on instagram there is a cookie popup that i can't seem to automate a click for. I keep getting a error. what i want to do is just to pick the accept all.
my code is:
#Selenium imports here
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
#Other imports here
import os
#import wget
driver = webdriver.Chrome("/Users/kitchensink/Desktop/instab/chromedriver")
driver.get("https://www.instagram.com")
Accept_all = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Accept All)]"))).click()
There error that i get is:
Traceback (most recent call last):
File "/Users/kitchensink/PycharmProjects/instapost/main3.py", line 15, in <module>
Accept_all = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Accept All)]"))).click()
File "/Users/kitchensink/.local/share/virtualenvs/kitchensink-UJdjJQi/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
See you have a issue with xpath here : (Your xpath does not contain ' at the end) you are using this
//button[contains(text(), 'Accept All)]
but one should use this instead :
//button[contains(text(), 'Accept All')]
I would do this as an xpath: /html[#class='js not-logged-in client-root js-focus-visible sDN5V']/body/div[#class='RnEpo Yx5HN _4Yzd2']/div[#class='pbNvD FrS-d gD9tr ']/div[#class='_1XyCr ']/button[#class='aOOlW bIiDR ']
So your code would look like:
#Selenium imports here
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
#Other imports here
import os
#import wget
driver = webdriver.Chrome("./chromedriver.exe")
driver.get("https://www.instagram.com")
Accept_all = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html[#class='js not-logged-in client-root js-focus-visible sDN5V']/body/div[#class='RnEpo Yx5HN _4Yzd2']/div[#class='pbNvD FrS-d gD9tr ']/div[#class='_1XyCr ']/button[#class='aOOlW bIiDR ']"))).click()
EDIT: I tried it and it works fine for me
I am trying to scrape a website but when I try to run the program, I get the following error. Here is my code
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.expected_conditions import presence_of_element_located
driver = webdriver.Chrome(executable_path = '/home/danish-khan/webscraping/rgcrawler2/chromedriver')
driver.get('https://www.researchgate.net/institution/Islamia_College_Peshawar/department/Department_of_Computer_Science/members')
chrome_options = Options()
element = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[#id="rgw9_5fac070727fc2"]/div[3]/h5/a]')))
print(element.text)`
Traceback (most recent call last):
File "resgt3.py", line 14, in <module>
element = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[#id="rgw9_5fac070727fc2"]/div[3]/h5/a]')))
File "/home/danish-khan/miniconda3/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
element = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '//*[#id="rgw9_5fac070727fc2"]/div[3]/h5/a]')))
While page is loading your program waits 20 seconds for XPATH element '//*[#id="rgw9_5fac070727fc2"]/div[3]/h5/a]' to appear.
If the XPATH element doesn't appear after defined time you get a Timeout error. Which is a good thing, otherwise your program would stuck forever if the XPATH element won't appear at all.
I think you should double check whether the provided XPATH is correct, or that it doesn't change over time.
For automating downloading from a website, I am using this python script for automating click action. But the click does not work.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
browser = webdriver.Firefox()
browser.maximize_window()
browser.implicitly_wait(20)
browser.get('http://download.cnet.com/most-popular/windows/')
linkElem = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "CCleaner")))
linkElem.click() # follows the "CCleaner" link
I first got this error:
selenium.common.exceptions.ElementNotInteractableException: Message: Element could not be scrolled into view
Then I add this line:
linkElem = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "CCleaner")))
Now I get this error:
linkElem = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "CCleaner")))
File "/usr/lib/python2.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
I tried the things that people used to solve this problem but did not work.
Your code seems to work with Chrome, I only added the bottom two lines and it downloaded CCleaner.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
browser = webdriver.Chrome() # Firefox()
browser.maximize_window()
browser.implicitly_wait(20)
browser.get('http://download.cnet.com/most-popular/windows/')
linkElem = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "CCleaner")))
linkElem.click() # follows the "CCleaner" link
download = browser.find_element_by_id('download-now-btn-text')
download.click()
Im trying to add input to a text box when i do try it doesn't find the element and it gives an error i don't know if im selecting the right element this is what i have so far
Traceback (most recent call last):
File "./fl_bot.py", line 22, in <module>
ui.WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.ID, "#billFirstName")))
File "/Library/Python/2.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
the code abouve is the error message im getting
from selenium.webdriver.support import ui
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
def get_page(model, sku):
url = "https://www.footlocker.com/product/model:"+str(model)+"/sku:"+ str(sku)+"/"
return url
browser = webdriver.Firefox()
page=browser.get(get_page(277097,"8448001"))
browser.find_element_by_xpath("//*[#id='pdp_size_select_mask']").click()
shoesize = ui.WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'a.grid_size:nth-child(8)')))
shoesize.click()
browser.find_element_by_xpath("//*[#id='pdp_addtocart_button']").click()
checkout = browser.get('https://www.footlocker.com/shoppingcart/default.cfm?sku=')
checkoutbutton = browser.find_element_by_css_selector('#cart_checkout_button').click()
ui.WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.ID, "#billFirstName")))
browser.find_element_by_id("#billFirstName").send_keys(Keys.RETURN)
every time it makes it to the end of the script it dosent type and it just stops
[1]: https://www.footlocker.com/checkout/?uri=checkout this is the page im trying to check out on
The FIRST NAME is a mandatory field to be filled up, so instead of send_keys(Keys.RETURN) try to send some text as follows along with the expected_condition as element_to_be_clickable :
ui.WebDriverWait(browser, 10).until(EC.element_to_be_clickable((By.XPATH, "//input[#id='billFirstName']")))
browser.find_element_by_xpath("//input[#id='billFirstName']").click()
browser.find_element_by_xpath("//input[#id='billFirstName']").clear()
browser.find_element_by_xpath("//input[#id='billFirstName']").send_keys("user_first_name")