This page has a total of 790 products and I write selenium code to automatically click on the product load button until it will finish loading all 790 products. Unfortunately, my code is not working and getting an error. here is my full code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import time
driver = webdriver.Chrome()
driver.maximize_window()
url ='https://www.billigvvs.dk/maerker/grohe/produkter?min_price=1'
driver.get(url)
time.sleep(5)
#accept cookies
try:
driver.find_element_by_xpath("//button[#class='coi-banner__accept']").click()
except:
pass
print('cookies not accepted')
# Wait 20 seconds for page to load.
timeout = 20
try:
WebDriverWait(driver, timeout).until(EC.visibility_of_element_located((By.XPATH, "//a[#class='productbox__info__name']")))
except TimeoutException:
print("Timed out waiting for page to load")
browser.quit()
#my page load button not working. I want to load all 790 product in this page
products_load_button = driver.find_element_by_xpath("//div[#class='filterlist__button']").click()
The error that I am getting:
Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#class='filterlist__button']"}
(Session info: chrome=87.0.4280.88)
The error message saying Unable to locate element but see the picture which saying I am selecting the right element.
You are missing an extra space at the end, try with this:
products_load_button = driver.find_element_by_xpath("//div[#class='filterlist__button ']").click()
when you work with selectors is always a good practice to copy and paste directly from the page, that will save a lot of headaches in the future.
Edit:
The while loop to check if all the elements are loaded looks similar to this:
progress_bar_text = driver.find_element_by_css("div.filterlist__pagination__text").text
# From here you could extract the total items and the loaded items
# Note: I am doing this because I don't have access to the page, probably
# there is a better way to found out if the items are loaded taking
# taking a look into the attributes of the progressBar
total_items = int(progress_bar_text.split()[4])
loaded_items = int(progress_bar_text.split()[1])
while loaded_items < total_items:
# Click the product load button until the products are loaded
product_load_button.click()
# Get the progress bar text and updates the loaded_items count
progress_bar_text = driver.find_element_by_css("div.filterlist__pagination__text").text
loaded_items = int(progress_bar_text.split()[1])
This is a very simple example and does not consider a lot of scenarios that you will need to handle to make it stable, some of them are:
The elements might disappear or reload after you click the products_load_button. For this I'll recommend that you take a look to explicit waits in selenium docs.
Is possible that the progress bar could disappear/hide after the load is complete.
Related
I am trying to navigate a scheduling website to eventually auto populate a schedule using the following script:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
# Create a Chrome webdriver
driver = webdriver.Chrome(r'C:\Users\chromedriver_win32\chromedriver.exe')
# Navigate to https://www.qgenda.com/
driver.get('https://www.qgenda.com/')
# Wait for the page to load
driver.implicitly_wait(5) # 5 seconds
# You can now interact with the page using the webdriver
# Locate the sign in button
sign_in_button = driver.find_element(By.XPATH,'/html/body/div[1]/div/header[3]/div/div[3]/div/div/div/div/a')
# Click the sign in button
sign_in_button.click()
# Find the input element
input_email = driver.find_element(By.XPATH,'//*[#id="Input_Email"]')
# Send text
input_email.send_keys('Josh')
However, I cannot seem to find the Input_Email object. I've tried all the Xpaths and Id's that make sense and also tried waiting until the object is clickable with no luck. Would really appreciate some guidance on this.
I was expecting Selenium to find the html object form box and pass in text but instead I get an error:
NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="Input_Email"]"}
even though the Xpath definitely exists.
The XPath seems fine. I am guessing you need to do some explicit wait or implicit wait to ensure the page is fully loaded before allocating the element.
Another thing I would like to point out is that given the login URL is available. Locating the sign in button seems to be redundant. You can access it directly via driver.get('https://login.qgenda.com/')
For instance,
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.get('https://login.qgenda.com/')
input_email = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '//*[#id="Input_Email"]'))
)
input_email.send_keys('Josh')
You can read more about it here.
Good morning,
I am trying to automatize simple online procedure: go to this site http://nvidia-research-mingyuliu.com/gaugan/
Check the little box, update a picture, render it and download it clicking the download button.
I have written my code, which is extremely intuitive and very short, but for some reasons sometimes it works and sometimes it doesn't.
Sure enough, sometimes python returns the following error:
no such element: Unable to locate element: {"method":"css selector","selector":"[id="myCheck"]"}
My code is:
import selenium
import time
from selenium import webdriver
# Using Chrome to access web
driver = webdriver.Chrome(executable_path='/Users/Marco/Downloads/chromedriver')
# Open the website
driver.get('http://nvidia-research-mingyuliu.com/gaugan/')
# Select the checkbox
time.sleep(5)
check_box = driver.find_element_by_id('myCheck')
check_box.click()
# Upload File button
element = driver.find_element_by_id("myCheck")
driver.execute_script("arguments[0].click();", element)
choose_file = driver.find_element_by_id('segmapfile')
# Send the file location to the button
choose_file.send_keys('/Users/Marco/Desktop/Foto upload/Schermata 2020-10-31 alle 00.07.03.png')
#Locate submit button and click
submit_assignment = driver.find_element_by_id('btnSegmapLoad')
submit_assignment.click()
#render
render_button = driver.find_element_by_id('render')
render_button.click()
#download
save_box = driver.find_element_by_id('save_render')
time.sleep(5)
save_box.click()
Why is this happening? How should i solve it?
Thank you
Use explicit waits which wait for something to happen, as opposed to 'time.sleep(5)' which just waits for 5 seconds:
import selenium
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
# Using Chrome to access web
driver = webdriver.Chrome(executable_path='/Users/Marco/Downloads/chromedriver')
# Open the website
driver.get('http://nvidia-research-mingyuliu.com/gaugan/')
# Wait for checkbox to be located
check_box_wait = EC.presence_of_element_located((By.ID, 'myCheck'))
WebDriverWait(driver, 10).until(check_box_wait)
# Select the checkbox
check_box = driver.find_element_by_id('myCheck')
check_box.click()
I am having trouble selecting a load more button on a Linkedin page. I receive this error in finding the xpath: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element
I suspect that the issue is that the button is not visible on the page at that time. So I have tried actions.move_to_element. However, the page scrolls just below the element, so that the element is no longer visible, and the same error subsequently occurs.
I have also tried move_to_element_with_offset, but this hasn't changed where the page scrolls to.
How can I scroll to the right location on the page such that I can successfully select the element?
My relevant code:
import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('C:\\Users\\Root\\Downloads\\chromedriver.exe')
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)
username = driver.find_element_by_name('session_key')
username.send_keys(parameters.linkedin_username)
sleep(0.5)
password = driver.find_element_by_name('session_password')
password.send_keys(parameters.linkedin_password)
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//button[#class="btn__primary--large from__button--floating"]')
sign_in_button.click()
driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')
loadmore_skills=driver.find_element_by_xpath('//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')
actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()
After playing around with it, I seem to have figured out where the problem is stemming from. The error
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]"}
(Session info: chrome=81.0.4044.113)
always correctly states the problem its encountering and as such it's not able to find the element. The possible causes of this include:
Element not present at the time of execution
Dynamically generated
content Conflicting names
In your case, it was the second point. As the content that is displayed is loaded dynamically as you scroll down. So When it first loads your profile the skills sections aren't actually present in the DOM. So to solve this, you simply have to scroll to the section so that it gets applied in the DOM.
This line is the trick here. It will position it to the correct panel and thus loading and applying the data to the DOM.
driver.execute_script("window.scrollTo(0, 1800)")
Here's my code (Please change it as necessary)
from time import sleep
# import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('../chromedriver.exe')
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)
username = driver.find_element_by_name('session_key')
username.send_keys('')
sleep(0.5)
password = driver.find_element_by_name('session_password')
password.send_keys('')
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//button[#class="btn__primary--large from__button--floating"]')
sign_in_button.click()
driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')
sleep(3)
# driver.execute_script("window.scrollTo(0, 1800)")
sleep(3)
loadmore_skills=driver.find_element_by_xpath('//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')
actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()
Output
Update
In concerns to your newer problem, you need to implement a continuous scroll method that would enable you to dynamically update the skills section. This requires a lot of change and should ideally be asked as a another question.
I have also found a simple solution by setting the scroll to the correct threshold. For y=3200 seems to work fine for all the profiles I've checked including yours, mine and few others.
driver.execute_script("window.scrollTo(0, 3200)")
If the button is not visible on the page at the time of loading then use the until method to delay the execution
try:
myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
print "Button is rdy!"
except TimeoutException:
print "Loading took too much time!"
Example is taken from here
To get the exact location of the element, you can use the following method to do so.
element = driver.find_element_by_id('some_id')
element.location_once_scrolled_into_view
This actually intends to return you the coordinates (x, y) of the element on-page, but also scroll down right to target element. You can then use the coordinates to make a click on the button. You can read more on that here.
You are getting NoSuchElementException error when the locators (i.e. id / xpath/name/class_name/css selectors etc) we mentioned in the selenium program code is unable to find the web element on the web page.
How to resolve NoSuchElementException:
Apply WebDriverWait : allow webdriver to wait for a specific time
Try catch block
so before performing action on webelement you need to take web element into view, I have removed unwated code and also avoided use of hardcoded waits as its not good practice to deal with synchronization issue. Also while clicking on show more button you have to scroll down otherwise it will not work.
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome(executable_path="path of chromedriver.exe")
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("email id")
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("password ")
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//button[#class='btn__primary--large from__button--floating']"))).click()
driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()
driver.execute_script("scroll(0, 250);")
buttonClick = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//span[text()='Show more']")))
ActionChains(driver).move_to_element(buttonClick).click().perform()
Output:
I have a div which contains the results for a certain search query. The text contained in this div changes as a button to go to the next page is clicked.
In the text contained in this div, there is also the corresponding number of the page. After clicking in the button to go to the next page, the results still take a bit to load, so I want to make the driver wait the content to load.
As such, I want to wait until the string "Page x" appears inside the div, where x corresponds to the number of the next page to be loaded.
For this, I tried:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
# Search page
driver.get('http://searchpage.com')
timeout = 120
current_page = 1
while True:
try:
# Wait until the results have been loaded
WebDriverWait(driver, timeout).until(
EC.text_to_be_present_in_element(
locator=(By.ID, "SEARCHBASE"),
text_="Page {:}".format(current_page)))
# Click to go to the next page. This calls some javascript.
driver.find_element_by_xpath('//a[#href="#"]').click
current_page += 1
except:
driver.quit()
Although, this always fails to match the text. What am I doing wrong here?
To just detect if anything whatsoever had changed in the page would also do the job, but I haven't found any way to do that.
Try to apply below solution to wait for partial text match:
WebDriverWait(driver, timeout).until(lambda driver: "Page {:}".format(current_page) in driver.find_element(By.ID, "SEARCHBASE").text)
I am trying to extract links from a infinite scroll website
It's my code for scrolling down the page
driver = webdriver.Chrome('C:\\Program Files (x86)\\Google\\Chrome\\chromedriver.exe')
driver.get('http://seekingalpha.com/market-news/top-news')
for i in range(0,2):
driver.implicitly_wait(15)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(20)
I aim at extracting specific links from this page. With class = "market_current_title" and HTML like the following :
<a class="market_current_title" href="/news/3223955-dow-wraps-best-week-since-2011-s-and-p-strongest-week-since-2014" sasource="titles_mc_top_news" target="_self">Dow wraps up best week since 2011; S&P in strongest week since 2014</a>
When I used
URL = driver.find_elements_by_class_name('market_current_title')
I ended up with the error that says "stale element reference: element is not attached to the page document". Then I tried
URL = driver.find_elements_by_xpath("//div[#id='a']//a[#class='market_current_title']")
but it says that there is no such a link !!!
Do you have any idea about solving this problem?
You're probably trying to interact with an element that is already changed (probably elements above your scrolling and off screen). Try this answer for some good options on how to overcome this.
Here's a snippet:
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
import selenium.webdriver.support.expected_conditions as EC
import selenium.webdriver.support.ui as ui
# return True if element is visible within 2 seconds, otherwise False
def is_visible(self, locator, timeout=2):
try:
ui.WebDriverWait(driver, timeout).until(EC.visibility_of_element_located((By.CSS_SELECTOR, locator)))
return True
except TimeoutException:
return False