Following code runs as expected if manually stepped in debug while it misbehaves if just run.
Aside from some initial code to reach the search page, I am then requesting a value which I know does not exist ("ZQZZQ") and thus the code should branch into the last else statement.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(
"/Users/bob/Documents/work/AIFA/scraper/scrape_gu/chromedriver"
)
# navigate through the initial agreement screens
driver.get("https://farmaci.agenziafarmaco.gov.it/bancadatifarmaci/cerca-farmaco")
readunderstood = driver.find_element_by_id("conf")
readunderstood.click()
accept = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "/html/body/div[5]/div[3]/div/button"))
)
accept.click()
# end of the initial agreement screens and general preparation
##############################################################
string_to_search = "ZQZZQ" # we can safely assume this does not exist
find_textbox = driver.find_element_by_id("search")
find_textbox.clear() # after the first search the old value will still be there
find_textbox.send_keys(string_to_search)
find_textbox.send_keys(Keys.ENTER)
# check if any results found
element = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.ID, "noresultsfound"))
)
# if we did retrieve something we don't have an error message
if element.text != "Nessun risultato trovato":
# first we have to find out the total number of retrieved pages
print(f"Digram {string_to_search} found!")
else:
print(f"Digram {string_to_search} not found. Need to close modal window!")
# click on ok of name not found modal dialog
WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, "/html/body/div[4]/div[3]/div/button"))
).click()
print("Modal window hopefully closed")
When I run it at full speed though I find it goes into the "if" branch, printing "Digram ZQZZQ found!" which is wrong.
Is the WebDriverWait call wrong? Thank you
You need to change the "presence_of_element_located" method to "visibility_of_element_located".
Old Code:
# check if any results found
element = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.ID, "noresultsfound"))
Change:
element = WebDriverWait(driver, 5).until(
EC.visibility_of_element_located((By.XPATH, "//*[#id='noresultsfound']"))
)
Output:
Digram ZQZZQ not found. Need to close modal window!
Modal window hopefully closed
VisibilityOfElementLocated Vs presenceOfElementLocated:
https://stackoverflow.com/questions/38038920/visibilityofelementlocated-vs-presenceofelementlocated#:~:text=while%20the%20visibilityOfElementLocated%20has%20to,of%20presenceOfElementLocated%20will%20be%20enough.
Related
i want to log in to the Instagram page. I want to press the "Jetzt nicht" button, but my Selenium cant find it, no matter if im searching for the class name, or the tag name...
Could somebody please help me?
Btw: the code will be edited later :)
[see the class name etc]
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from time import sleep
#define webdriver
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.instagram.com/")
#wait 10s or till cookies loaded and accept
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.TAG_NAME, "button")))
element.click()
#type username account, wait max 10s until searchbar loaded
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.NAME, "username")))
element.send_keys("yeet")
#hit enter
element.send_keys(Keys.RETURN)
#type password
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.NAME, "password")))
element.send_keys("yeet")
#hit enter
element.send_keys(Keys.RETURN)
#press the login button, press enter
element.send_keys(Keys.RETURN)
#click false on safe ur login information
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, "B.sqdOP yWX7d y3zKF ")))
element.click()
sleep(30)
#quit
driver.quit()
driver.close()
(By.CLASS_NAME, "B.sqdOP yWX7d y3zKF ") locator is wrong!
By.CLASS_NAME locator receives single class name only, not multiple class names.
Also the long spaces between class names here absolutely not correct.
Also the class names you are using here seems to be dynamically changing, not something fixed.
And the B at the beginning looks to be absolutely irrelevant...
If you are using English version of instagram.com and wants to click on Not Now button which just appear once you login successfully. you can use the below code :
try:
# click false on save your login information
element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Not Now']")))
element.click()
except:
print('something went wrong')
pass
should work for you.
I have read several articles on this site regarding around the StaleElementReferenceException and am aware that this error is caused by the element no longer being in the site's DOM. What I am trying to do is click the bottom links on this webpage in order to go on and see the next page's listings. I have tried a few ways around this exception being given to me, and haven't found any to work. Here is an example of the code I have tried, and what I thought it might accomplish.
driver = webdriver.Chrome(r'C:\Users\Hank\Desktop\chromedriver_win32\chromedriver.exe')
driver.get('https://steamcommunity.com/market/listings/440/Unusual%20Old%20Guadalajara')
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support.expected_conditions import presence_of_element_located
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import StaleElementReferenceException
action = ActionChains(driver)
page_links = wait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '[class^=market_paging_pagelink]')))
try:
action.move_to_element(page_links[1]).click().perform()
except StaleElementReferenceException as Exception:
print("Exception received, trying again")
time.sleep(5)
page_links = wait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '[class^=market_paging_pagelink]')))
action.move_to_element(page_links[1]).click().perform()
I was hoping that this code segment would attempt to move to the element at the bottom, click it, or return the error message, and try again, succeeding the second time. Instead, the code simply throws the error again. If my question has already been answered, please direct me to the relevant link.
Thank you!
The approach I normally go for is to click Next page until the button gets disabled/invisible.
Here's a working example based on your page. You should obviously do whatever relevant in the while loop; I chose to capture prices for the sake of example.
url="https://steamcommunity.com/market/listings/440/Unusual%20Old%20Guadalajara"
driver.get(url)
next_button=wait(driver, 10).until(EC.presence_of_element_located((By.ID,'searchResults_btn_next')))
# capture the start value from "Showing x-xx of 22 results"
#need this to check against later
ref_val=wait(driver, 10).until(EC.presence_of_element_located((By.ID,'searchResults_start'))).text
while next_button.get_attribute('class') == 'pagebtn':
next_button.click()
#wait until ref_val has changed
wait(driver, 10).until(lambda driver: wait(driver, 10).until(EC.presence_of_element_located((By.ID,'searchResults_start'))).text != ref_val)
# ====== Do whatever relevant here =============================
page_num=wait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR,'.market_paging_pagelink.active'))).text
print(f"Prices from page {page_num}")
prices = wait(driver, 10).until(EC.presence_of_all_elements_located(
(By.XPATH, ".//span[#class='market_listing_price market_listing_price_with_fee']")))
for price in prices:
print(price.text)
#================================================================
#get the new reference value
ref_val = wait(driver, 10).until(EC.presence_of_element_located((By.ID, 'searchResults_start'))).text
I am having trouble selecting a load more button on a Linkedin page. I receive this error in finding the xpath: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element
I suspect that the issue is that the button is not visible on the page at that time. So I have tried actions.move_to_element. However, the page scrolls just below the element, so that the element is no longer visible, and the same error subsequently occurs.
I have also tried move_to_element_with_offset, but this hasn't changed where the page scrolls to.
How can I scroll to the right location on the page such that I can successfully select the element?
My relevant code:
import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('C:\\Users\\Root\\Downloads\\chromedriver.exe')
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)
username = driver.find_element_by_name('session_key')
username.send_keys(parameters.linkedin_username)
sleep(0.5)
password = driver.find_element_by_name('session_password')
password.send_keys(parameters.linkedin_password)
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//button[#class="btn__primary--large from__button--floating"]')
sign_in_button.click()
driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')
loadmore_skills=driver.find_element_by_xpath('//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')
actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()
After playing around with it, I seem to have figured out where the problem is stemming from. The error
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]"}
(Session info: chrome=81.0.4044.113)
always correctly states the problem its encountering and as such it's not able to find the element. The possible causes of this include:
Element not present at the time of execution
Dynamically generated
content Conflicting names
In your case, it was the second point. As the content that is displayed is loaded dynamically as you scroll down. So When it first loads your profile the skills sections aren't actually present in the DOM. So to solve this, you simply have to scroll to the section so that it gets applied in the DOM.
This line is the trick here. It will position it to the correct panel and thus loading and applying the data to the DOM.
driver.execute_script("window.scrollTo(0, 1800)")
Here's my code (Please change it as necessary)
from time import sleep
# import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('../chromedriver.exe')
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)
username = driver.find_element_by_name('session_key')
username.send_keys('')
sleep(0.5)
password = driver.find_element_by_name('session_password')
password.send_keys('')
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//button[#class="btn__primary--large from__button--floating"]')
sign_in_button.click()
driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')
sleep(3)
# driver.execute_script("window.scrollTo(0, 1800)")
sleep(3)
loadmore_skills=driver.find_element_by_xpath('//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')
actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()
Output
Update
In concerns to your newer problem, you need to implement a continuous scroll method that would enable you to dynamically update the skills section. This requires a lot of change and should ideally be asked as a another question.
I have also found a simple solution by setting the scroll to the correct threshold. For y=3200 seems to work fine for all the profiles I've checked including yours, mine and few others.
driver.execute_script("window.scrollTo(0, 3200)")
If the button is not visible on the page at the time of loading then use the until method to delay the execution
try:
myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
print "Button is rdy!"
except TimeoutException:
print "Loading took too much time!"
Example is taken from here
To get the exact location of the element, you can use the following method to do so.
element = driver.find_element_by_id('some_id')
element.location_once_scrolled_into_view
This actually intends to return you the coordinates (x, y) of the element on-page, but also scroll down right to target element. You can then use the coordinates to make a click on the button. You can read more on that here.
You are getting NoSuchElementException error when the locators (i.e. id / xpath/name/class_name/css selectors etc) we mentioned in the selenium program code is unable to find the web element on the web page.
How to resolve NoSuchElementException:
Apply WebDriverWait : allow webdriver to wait for a specific time
Try catch block
so before performing action on webelement you need to take web element into view, I have removed unwated code and also avoided use of hardcoded waits as its not good practice to deal with synchronization issue. Also while clicking on show more button you have to scroll down otherwise it will not work.
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome(executable_path="path of chromedriver.exe")
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("email id")
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("password ")
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//button[#class='btn__primary--large from__button--floating']"))).click()
driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()
driver.execute_script("scroll(0, 250);")
buttonClick = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//span[text()='Show more']")))
ActionChains(driver).move_to_element(buttonClick).click().perform()
Output:
I've written a script in python in combination with selenium to get some information from a webpage. To get the content it is necessary to do some clicks and fill in an inputbox to produce the result. The result gets displayed in a new tab. So, it is necessary to switch to that particular window to parse the information. My script can do all this very efficiently.
It's a follow-up post of this one.
The problem I'm facing is that I'm using few keywords to get information from. When a single keyword is used then driver.close() might do the trick but I'm using multiple keywords so when I use driver.close(), there is no more window left to move along.
Question: how can I make the scraper close the new tab (when it grabs information from there) and switch back to main window to do the process cyclically until there is no more keyword to check?
This is my script so far:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link = "https://officialrecords.broward.org/AcclaimWeb/search/SearchTypeName"
def get_information(driver,url):
for keyword in ['HMC DESIGN GROUP','HMC DESIGN GROUP']:
driver.get(url)
current = driver.current_window_handle
wait.until(EC.element_to_be_clickable((By.ID, "btnButton"))).click()
wait.until(EC.presence_of_element_located((By.ID,"SearchOnName"))).send_keys(keyword)
wait.until(EC.presence_of_element_located((By.ID, "btnSearch"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//td[contains(., 'HMC DESIGN GROUP')]"))).click()
wait.until(EC.new_window_is_opened)
driver.switch_to.window([window for window in driver.window_handles if window != current][0])
for items in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".listDocDetails"))):
print(items.text)
# driver.switch_to.default_content()
# driver.close()
if __name__ == "__main__":
driver = webdriver.Chrome()
wait = WebDriverWait(driver,10)
try:
get_information(driver,link)
finally:
driver.quit()
Try to implement below solution:
def get_information(driver,url):
driver.get(url)
current = driver.current_window_handle
wait.until(EC.element_to_be_clickable((By.ID, "btnButton"))).click()
for keyword in ['HMC DESIGN GROUP','HMC DESIGN GROUP']:
input_field = wait.until(EC.presence_of_element_located((By.ID,"SearchOnName")))
input_field.clear()
input_field.send_keys(keyword)
wait.until(EC.presence_of_element_located((By.ID, "btnSearch"))).click()
wait.until_not(EC.visibility_of_element_located((By.ID, "SearchingWaitImg")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//td[contains(., '%s')]" % keyword))).click()
driver.switch_to.window([window for window in driver.window_handles if window != current][0])
for items in wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".listDocDetails"))):
print(items.text)
driver.close()
driver.switch_to.window(current)
Code at Selenium by python
from selenium import webdriver
from selenium.webdriver.support.select import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome()
driver.maximize_window()
driver.get("https://motul.lubricantadvisor.com/Default.aspx?data=1&lang=ENG&lang=eng")
def getallcars():
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.ID, "ctl00_ContentPlaceHolder1_rptCategoryBtn_ctl01_btnImage")))
driver.find_element(By.ID, "ctl00_ContentPlaceHolder1_rptCategoryBtn_ctl01_btnImage").click()
wait.until(EC.presence_of_element_located((By.ID, "ctl00_ContentPlaceHolder1_lblSelectedMake")))
driver.find_element(By.ID, 'ctl00_ContentPlaceHolder1_lblSelectedMake').click()
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#ctl00_ContentPlaceHolder1_lstMake")))
el = driver.find_element(By.NAME,"ctl00$ContentPlaceHolder1$lstMake")
car =[]
for option in el.find_elements(By.TAG_NAME,'option'):
car.append((option.text).encode('utf8'))
return car
cars=getallcars()
for value in cars:
drop = driver.find_element(By.CSS_SELECTOR, '#ctl00_ContentPlaceHolder1_lstMake')
sel = Select(drop)
sel.select_by_visible_text(value)
time.sleep(2) #<---- THIS POINT!!
driver.find_element(By.ID,'ctl00_ContentPlaceHolder1_HeaderModel').click()
el2 = driver.find_element(By.NAME, "ctl00$ContentPlaceHolder1$lstModel")
print "The models for %s are:" %value
for option in el2.find_elements(By.TAG_NAME,'option'):
print option.text
action = ActionChains(driver)
action.move_to_element_with_offset(el2, 300, 200)
action.click()
action.perform()
driver.find_element(By.CSS_SELECTOR,'#ctl00_ContentPlaceHolder1_HeaderMake').click()
I have been make the crawler. I don't understand completely yet. so I have a question. maybe It's 34line at code. I was mark about #
it's use be the "time.sleep(2)" method. because It didn't detect the select box when It's change about "sel.select_by_visible_text(value)"
how can I do that? I don't want to use the "time.sleep(2)"method.
already I tried "expected_conditions.presence_of_element_located" It doesn't work. I guess It's problem about dropbox. this size is not basically because It did well when I tried another size tried "expected_conditions.presence_of_element_located"
Explicit wait will not work, because the conditions you can use are "element to be clickable", "element to be visible" and like that.
The element that you using for explicit wait is available and clickable also, but its failing because other element is overlapping it.
Since the other element's overlap takes time to disappear, we have to wait for the overlap to disappear before we can click on the element.
Explicit wait can wait for element to appear and clickable, which it is already is but it is being hidden by other element.
In this scenario, we have to use time.sleep() to put a hard wait