I am writing a script that deals with google. I have successfully searched for what I wanted using the selenium web driver however I would like to navigate to the next page of results. my code looks as follows:
import parameters
import csv
from parsel import Selector
from selenium import webdriver
from time import sleep
from selenium.webdriver.common.keys import Keys
empty=""
def validate_field(field):
if field == "":
field= 'No results'
return field
writer=csv.writer(open(parameters.file_name, 'w'))
writer.writerow(('Name','Job Title','Company','College','Location','URL'))
driver=webdriver.Chrome('/usr/local/bin/chromedriver')
driver.implicitly_wait(10)
driver.get('https://www.linkedin.com')
username=driver.find_element_by_id('session_key') #locating the email form using the class name
username.send_keys('')
sleep(0.5)
password=driver.find_element_by_id('session_password') #locating the password form using the class name
password.send_keys('')
sleep(0.5)
log_in_button=driver.find_element_by_class_name('sign-in-form__submit-button') #locating submit button by class name
log_in_button.click() #here we are mimicing a click
sleep(0.5)
driver.get('https:www.google.com') #navigating to google
sleep(3)
search_gog=driver.find_element_by_name('q')
search_gog.send_keys(parameters.search_gog)
sleep(0.5)
search_gog.send_keys(Keys.RETURN)
sleep(3)
list_links = [link.get_attribute('href') for link in driver.find_elements_by_xpath("//div[#class='g']//div[#class='r']/a[contains(#href, 'https://www.linkedin.com')]")]
for link in list_links:
driver.get(link)
sel=Selector(text=driver.page_source)
name = sel.xpath('//*[starts-with(#class, "inline t-24 t-black t-normal break-words")]/text()').extract_first()
if name:
name=name.strip()
job_title= sel.xpath('//*[starts-with(#class, "mt1 t-18 t-black t-normal break-words")]/text()').extract_first()
if job_title:
job_title=job_title.strip()
education = sel.xpath('//*[starts-with(#class, "pv-profile-section-pager ember-view")]/text()').extract_first()
if education:
education=education.strip()
name=validate_field(name)
job_title=validate_field(job_title)
education=validate_field(education)
print('\n')
print('Name: ' + name)
print('Job Title: ' + job_title)
print('Education: ' + education)
print(education)
print('URL: ' + driver.current_url)
print('\n')
driver.find_element_by_link_text("Next").click()
the parameters file contains the google search query which states:
search_gog = 'site:linkedin.com/in/ AND "data analyst" AND "California"'
upon running the above I receive the following error:
Traceback (most recent call last):
File "app2.py", line 79, in <module>
driver.find_element_by_link_text("Next").click()
File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 428, in find_element_by_link_text
return self.find_element(by=By.LINK_TEXT, value=link_text)
File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"Next"}
(Session info: chrome=85.0.4183.102)
I have also tried using the html code for the element which is:
<a href="/search?q=site:linkedin.com/in/+AND+%22data+analyst%22+AND+%22California%22&ei=lbpbX4GhDoPs9APDmITID
A&start=10&sa=N&ved=2ahUKEwjBv57e1uHrAhUDNn0KHUMMAckQ8NMDegQICxA_"
id="pnnext" style="text-align:left"><span class="SJajHc NVbCr"
style="background:url(/images/nav_logo299_hr.webp) no-repeat;background-position:-96px
0;background-size:167px;width:71px"></span><span style="display:block;margin-
left:53px">Next</span></a>
using the above html element I have tried finding the element using the element id "pnnext' by doing the following find_element_by_id("pnnext").click and it has been unsuccessful.
Any ideas?
First, if you have driver.implicitly_wait(10), then when you do driver.find_element_by_link_text("Next").click() following search_gog.send_keys(Keys.RETURN), the driver will wait for up to 10 seconds for the next page to load and a link with the text "Next" to appear before throwing an exception. So, the call to sleep(3) that you invoke between the two calls is rather superfluous. If for whatever reason you thought 10 seconds was not enough (hard to believe for google.com), then just increase the amount of time on the call to driver.implicitly_wait.
I cannot tell what your search argument to Google is, but it just may be possible that the number of returned results isn't enough to warrant a "Next" link. Modify your code as follows:
search_gog.send_keys(Keys.RETURN)
next_links = driver.find_elements_by_link_text("Next") # note it is find_elements with an s
if len(next_links):
next_links[0].click()
else:
print('There is no "Next" link')
Update
I have run the following code on my desktop 5 times in a row and each time it has successfully navigated to the second page (that is, it has found the "Next" link and clicked on it successfully).
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
options = webdriver.ChromeOptions()
#options.add_argument("headless")
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)
try:
driver.implicitly_wait(10)
driver.get('https://www.google.com') #navigating to google
search_gog=driver.find_element_by_name('q')
search_gog.send_keys('site:linkedin.com/in/ AND "data analyst" AND "California"')
search_gog.send_keys(Keys.RETURN)
next_links = driver.find_elements_by_link_text("Next") # note it is find_elements with an s
if len(next_links):
print('Found "Next" link')
next_links[0].click()
else:
print('There is no "Next" link')
finally:
input('pausing (hit enter to terminate) ...')
driver.quit()
selenium.common.exceptions.NoSuchElementException means it can't find the element it may be caused by elements takings too long to load or a wrong XPath. There might be an iframe as well.
elem=WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.ID('pnnext')))
elem.click()
Import these.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Related
I'm pretty new to selenium. I'm working on a autocheckout scrip using selenium python. Script executes very smooth till the final checkout page. However, I'm unable to access element on the final page('Razorpay Checkout'), which is a kind of pop up/iframe window.
Any help accessing the same would be highly appreciated.
Thanks in advance!
Below is the code I'm using
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome('C:/Users/hamhatre/Desktop/chromedriver.exe')
driver.get("https://shopatsc.com/collections/playstation-5/products/ps5-horizon-forbidden-west")
driver.maximize_window()
driver.implicitly_wait(20)
elem = driver.find_element_by_xpath('//*[#id="pincode_input"]')
elem.send_keys("400708")
driver.find_element_by_xpath('//*[#id="check-delivery-submit"]').click()
driver.find_element_by_xpath('/html/body/div[2]/main/div[1]/div/div/div/div[2]/div[1]/form/div[3]/div/button[2]').click()
driver.find_element_by_xpath('//*[#id="checkout_email"]').send_keys('abc#gmail.com')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_first_name"]').send_keys('abc')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_last_name"]').send_keys('xyz')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_address1"]').send_keys('Rd1, Flat no 2, Apt 1')
driver.find_element_by_xpath(('//*[#id="checkout_shipping_address_address2"]')).send_keys('Nothing')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_city"]').send_keys('Mumbai')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_province"]').send_keys('Maharashtra')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_zip"]').send_keys('400708')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_phone_custom"]').send_keys('9876543210')
driver.find_element_by_id('continue_to_shipping_button_custom').click()
driver.find_element_by_id('continue_button').click()
driver.find_element_by_xpath('/html/body/div/div/div/main/div[1]/div/form/div[3]/div[1]/button').click()
print(driver.title)
seq = driver.find_elements_by_tag_name('iframe')
print(seq)
print("No of frames present in the web page are: ", len(seq))
for index in range(len(seq)):
iframe = driver.find_elements_by_tag_name('iframe')[index]
driver.switch_to.frame(iframe)
driver.find_element_by_id('language-dropdown').click()
Below is the error
Traceback (most recent call last):
File "C:\Users\hamhatre\Desktop\Algotron\WebScrap_Sample.py", line 47, in <module>
driver.find_element_by_id('language-dropdown').click()
File "C:\Users\hamhatre\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "C:\Users\hamhatre\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Users\hamhatre\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\hamhatre\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"[id="language-dropdown"]"}
(Session info: chrome=99.0.4844.84)
I am using chromedriver version 100. Your code throws a NoSuchElementException. Thats because your driver tries to find an element which does not exist. So, what you need to do is to put time.sleep(4) statements after your click() statements (Your browser needs time to open the window/s). If you are done using the driver, use driver.quit().
Followed code worked for me:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome('C:/Users/hamhatre/Desktop/chromedriver.exe')
driver.get("https://shopatsc.com/collections/playstation-5/products/ps5-horizon-forbidden-west")
driver.maximize_window()
driver.implicitly_wait(20)
elem = driver.find_element_by_xpath('//*[#id="pincode_input"]')
elem.send_keys("400708")
driver.find_element_by_xpath('//*[#id="check-delivery-submit"]').click()
driver.find_element_by_xpath('/html/body/div[2]/main/div[1]/div/div/div/div[2]/div[1]/form/div[3]/div/button[2]').click()
driver.find_element_by_xpath('//*[#id="checkout_email"]').send_keys('abc#gmail.com')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_first_name"]').send_keys('abc')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_last_name"]').send_keys('xyz')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_address1"]').send_keys('Rd1, Flat no 2, Apt 1')
driver.find_element_by_xpath(('//*[#id="checkout_shipping_address_address2"]')).send_keys('Nothing')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_city"]').send_keys('Mumbai')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_province"]').send_keys('Maharashtra')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_zip"]').send_keys('400708')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_phone_custom"]').send_keys('9876543210')
driver.find_element_by_id('continue_to_shipping_button_custom').click()
time.sleep(4)
driver.find_element_by_id('continue_button').click()
time.sleep(4)
driver.find_element_by_xpath('/html/body/div/div/div/main/div[1]/div/form/div[3]/div[1]/button').click()
time.sleep(4)
print(driver.title)
seq = driver.find_elements_by_tag_name('iframe')
print(seq)
print("No of frames present in the web page are: ", len(seq))
for index in range(len(seq)):
iframe = driver.find_elements_by_tag_name('iframe')[index]
driver.switch_to.frame(iframe)
driver.find_element_by_id('language-dropdown').click()
A few things you could do in your next project:
Use WebDriverWait to allow enough time for elements to be loaded
Use switch_to.default_content() before switching to another frame
Use time.sleep() while you switch frames
Try using relative xpaths (ex: //div[#id="user_address"]). It's easy to right click and copy the absolute xpath for an element from the inspect window, however if there's another element added in between, your xpath would no longer point to the desired element. Moreover, relative xpaths would empower you with a deeper understanding allowing you to easily find elements.
This code should work:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import StaleElementReferenceException
import time
driver = webdriver.Chrome('D://chromedriver/100/chromedriver.exe')
driver.get("https://shopatsc.com/collections/playstation-5/products/ps5-horizon-forbidden-west")
driver.maximize_window()
wait = WebDriverWait(driver, 20)
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="pincode_input"]'))).send_keys('400708')
wait.until(EC.presence_of_element_located((By.XPATH, '//span[#id="check-delivery-submit"]'))).click()
wait.until(EC.presence_of_element_located((By.XPATH, '//button[normalize-space(text())="Buy Now"]'))).click()
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_email"]'))).send_keys('abc#gmail.com')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_first_name"]'))).send_keys('abc')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_last_name"]'))).send_keys('xyz')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_address1"]'))).send_keys('Rd1, Flat no 2, Apt 1')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_address2"]'))).send_keys('Nothing')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_city"]'))).send_keys('Mumbai')
wait.until(EC.presence_of_element_located((By.XPATH, '//select[#id="checkout_shipping_address_province"]'))).send_keys('Maharashtra')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_zip"]'))).send_keys('400708')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_phone_custom"]'))).send_keys('9876543210')
wait.until(EC.presence_of_element_located((By.XPATH, '//button[#id="continue_to_shipping_button_custom"]'))).click()
wait.until(EC.presence_of_element_located((By.XPATH, '//button[#id="continue_button"]'))).click()
wait.until(EC.presence_of_element_located((By.XPATH, '//span[normalize-space(text())="Complete order"]/parent::button'))).click()
print(driver.title)
# finding all iframes
iframes = wait.until(EC.presence_of_all_elements_located((By.TAG_NAME, 'iframe')))
print(f'No. of iframes: {len(iframes)}')
print('Looping through frames to check language dropdown')
found = False
frameno = 0
tries = 1
maxtries = 5
# Attempt at least 5 times to click on the language dropdown
while not found and frameno < len(iframes) and tries <= maxtries:
try:
print(f"Frame# {frameno} \t {tries} of {maxtries} tries")
print('Switching to frame')
driver.switch_to.frame(iframes[frameno])
time.sleep(2)
wait.until(EC.presence_of_element_located((By.XPATH, '//div[#id="language-dropdown"]/div/div'))).click()
found = True
print('Found and clicked')
except StaleElementReferenceException:
print("Exception occured. Capturing iframes again")
driver.switch_to.default_content()
iframes = wait.until(EC.presence_of_all_elements_located((By.TAG_NAME, 'iframe')))
except Exception as ex:
print("Couldn't find/click on the language dropdown")
frameno += 1
finally:
print('Switching to default content')
driver.switch_to.default_content()
time.sleep(2)
tries += 1
Error: Element is not currently interactable and may not be manipulated.
So I have selenium requesting a page and then filling out a few input boxes then clicking a button to submit the form.
The issue I'm having is that selenium isn't waiting for the element to be visible so I have to manually place a time.sleep(2) to fix that but I would prefer to use a wait until they show up in case it takes longer or less time to load or if they don't load at all.
I have it setup like this currently to request the page, find the input field, check if the element exists and type the text into it, I get this error at the element.clear() part which doesn't happen when I include a time.sleep(2) after requesting the url:
error
Traceback (most recent call last):
File "C:\Users\example\AppData\Roaming\JetBrains\PyCharmCE2021.2\scratches\example.py", line 125, in <module>
completeForm(
File "C:\Users\example\AppData\Roaming\JetBrains\PyCharmCE2021.2\scratches\example.py", line 89, in completeForm
typeText(Field, text)
File "C:\Users\example\AppData\Roaming\JetBrains\PyCharmCE2021.2\scratches\example.py", line 50, in typeText
element.clear()
File "C:\Users\example\Desktop\selenium-program\venv\lib\site-packages\selenium\webdriver\remote\webelement.py", line 92, in clear
self._execute(Command.CLEAR_ELEMENT)
File "C:\Users\example\Desktop\selenium-program\venv\lib\site-packages\selenium\webdriver\remote\webelement.py", line 693, in _execute
return self._parent.execute(command, params)
File "C:\Users\example\Desktop\selenium-program\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 400, in execute
self.error_handler.check_response(response)
File "C:\Users\example\Desktop\selenium-program\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 236, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidElementStateException: Message: invalid element state: Element is not currently interactable and may not be manipulated
(Session info: chrome=93.0.4577.63)
Code
from seleniumwire.undetected_chromedriver.v2 import Chrome, ChromeOptions
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver import ActionChains
import random
import time
#Setup selenium chrome browser with options
def setupBrowser():
global driver
options = {}
chrome_options = ChromeOptions()
driver = Chrome(seleniumwire_options=options, options=chrome_options)
#Type like a human
def typeText(element, text):
success = False
if doesElementExist(element):
while success == False:
actions = ActionChains(driver)
actions.move_to_element(element)
actions.click()
element.clear()
for character in text:
actions.send_keys(character)
actions.perform()
time.sleep(random.uniform(0.13, 0.4))
if element.get_attribute("data-initial-value") == text:
success = True
#check if element exists, times out after some time and returns error
def doesElementExist(element):
timeout = 10
try:
element_present = EC.visibility_of_element_located(element)
WebDriverWait(driver, timeout).until(element_present)
except TimeoutException:
print("Error: Timed out after {} seconds waiting for element to load!".format(timeout))
return False
finally:
return True
driver.get(url)
driver.implicitly_wait(60)
Field = driver.find_element_by_xpath('//*[#id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input')
text = "exampletexttowrite123"
typeText(Field, text)
Your typeText needs corrections. It does not have .perform() so what happens is all the item will get stored in Actioncahins queue and the moment you write .perform() they will be release one by one.
Code :
def typeText(element, text):
success = False
if doesElementExist(element):
while success == False:
actions = ActionChains(driver)
actions.move_to_element(element).perform()
element.click()
element.clear()
for character in text:
actions.send_keys(character).perform()
time.sleep(random.uniform(0.13, 0.4))
if element.get_attribute("data-initial-value") == text:
success = True
PS :
this method is directly accepting a web element, make sure the web element should be rendered properly.
Update :
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(50)
driver.get("https://docs.google.com/forms/d/e/1FAIpQLSfMJRFOPF0qeJG3DHrbzbDR7nETPF0qE2D-r_F3kjjqdP9B1w/viewform")
wait = WebDriverWait(driver, 20)
ActionChains(driver).move_to_element(wait.until(EC.visibility_of_element_located((By.XPATH, "//div[text()='Your answer']/preceding-sibling::input")))).perform()
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[text()='Your answer']/preceding-sibling::input"))).send_keys('cruise')
I am performing an automation bot, capable of sharing post. However, when it comes to performing the task more than 3 times, I get an error and program stops working. I am not sure why my code is able to perform the task 3 times and then stops.
Here is my code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as ECDS
import time
driver = webdriver.Chrome()
driver.get("https://www.poshmark.com") #Open webpage
Log_Field=(By.XPATH, "//a[contains(text(),'Log in')]")
Email= (By.XPATH, "//input[#placeholder='Username or Email']")
Pass= (By.XPATH, "//input[#placeholder='Password']")
Second_Log= (By.XPATH, "//button[#class='btn btn--primary']")
SF = (By.XPATH, "//img[#class='user-image user-image--s']")
MyCloset = (By.XPATH, "//a[contains(text(),'My Closet')]")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable(Log_Field)).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable(Email)).send_keys("xx#xx.com")
driver.find_element_by_xpath("//input[#placeholder='Password']").send_keys("xxx")
driver.find_element_by_xpath("//button[#class='btn blue btn-primary']").click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable(SF)).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable(MyCloset)).click()
for i in range(100):
driver.find_element_by_tag_name('body').send_keys(Keys.END)#Use send_keys(Keys.HOME)
driver.find_element_by_xpath("//div[6]//div[1]//div[2]//div[3]//i[1]").click()
driver.find_element_by_xpath("//div[#class='share-wrapper-container']").click()
driver.refresh()
time.sleep(20)
The error that I am getting is the following:
Traceback (most recent call last):
File "/home/pi/Documents/Bot_Poshmark.py", line 27, in <module>
driver.find_element_by_xpath("//div[6]//div[1]//div[2]//div[3]//i[1]").click()
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <i class="icon share-gray-large"></i> is not clickable at point (870, 163). Other element would receive the click: <div class="tile col-x12 col-l6 col-s8 p--2">...</div>
(Session info: chrome=78.0.3904.108)
Any ideas why my code is only working not more than 3 times?
Thank you
I had the same problem several times while testing my selenium web automation. As the exception tells, this object is not clickable. That means you have to dive deeper into the HTML tree to find an element that is always clickable. If you hover over the HTML-lines Chrome shows you the according piece of website.
However, if this is not possible, try to let your code sleep() for a bit :-)
You could do that for a certain amount of time
Or you use WebDriverWait() as in the comments described:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "your XPATH"))).click()
(You missed one (), this should remove your error: Without: Argument 1 = self [Keep that in mind with Python!!], Argument 2 = By.XPATH and 3 = "the xpath". With the (), Argument 2 and 3 are together)
WebDriverWait() requires a timeout parameter because selenium does not know whether the element exists or not. But you could easily create your own waiting-method. Pay attention: you have to know that the element exists or you will end up with an infinite loop.
Here's the code:
from time import sleep
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
def wait_for(driver, method):
"""Calls the method provided with the driver"""
while True:
try:
element = method(driver)
if element:
return element
except:
pass
sleep(0.5)
It tries to find the element and when it's found, it returns it.
You can use it like so:
el = wait_for(driver, EC.element_to_be_clickable((By.XPATH, "//input[#class='13e44'")))
el.click()
Disclaimer: This code is not fully my creation. I adapted the selenium source-code so it fits our needs :)
I have this site https://www.inc.com/inc5000/list/2017 where I want my script to insert a number in PAGE field and click GO, but I keep getting error:
File "/Users/anhvangiang/Desktop/PY/inc.py", line 34, in scrape
driver.find_element_by_xpath('//*[#id="page-input-button"]').click()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 77, in click
self._execute(Command.CLICK_ELEMENT)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 493, in _execute
return self._parent.execute(command, params)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error:
Element is not clickable at point (741, 697)
(Session info: chrome=61.0.3163.100)
(Driver info: chromedriver=2.30.477690
(c53f4ad87510ee97b5c3425a14c0e79780cdf262),platform=Mac OS X 10.12.6 x86_64)
This is my code:
def scrape(num):
driver = webdriver.Chrome('/Users/anhvangiang/Desktop/PY/chromedriver')
driver.get(main_site)
driver.find_element_by_id('page-input-field').send_keys(str(num))
driver.find_element_by_xpath('//*[#id="Welcome-59"]/div[2]/div[1]/span[2]').click()
time.sleep(5)
driver.find_element_by_xpath('//*[#id="page-input-button"]').click()
soup = BS(driver.page_source, 'lxml')
container = soup.find('section', {'id': 'data-container'})
return [source + tag.find('div', {'class': 'col i5 company'}).find('a')['href'] for tag in container.findAll('div', {'class': 'row'})]
If I put the function scrape inside a loop:
for i in range(1, 100):
print scrape(i)
For a few first i, it will go smoothly, but then it will throw the error like above.
Any suggestion how I can fix it?
This is because the button is not visible at that time, so the selenium WebDriver cannot access that.
As I run your code on my local machine , I found that the website shows a popup ad for 15-20 sec (See attached image : Popup_Ad), which is the actual cause of this error. For resolving this error you have to handle the popup ad, you can do this as.
check for the SKIP button, if button exist then first skip the add by
Clicking the skip button , then follow the normal flow of code.
Other Suggestions: You should use WebDriverWait to avoid the Element not found and element is not clickable like issues. For Example, you can write above code as
from selenium import webdriver
from bs4 import BeautifulSoup as BS
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support import expected_conditions as EC
import time
def scrape(num,wdriver):
# define webdriver
driver = wdriver
# naviagte to url
driver.get("https://www.inc.com/inc5000/list/2017")
# define default wait time
wait = WebDriverWait(driver, 10)
while True:
if EC.presence_of_all_elements_located:
break
else:
continue
# handle Ad Popup
try:
skip_button = wait.until(EC.element_to_be_clickable((By.XPATH,"//*[#id='Welcome-59']/div[2]/div[1]/span[2]")))
skip_button.click()
print("\nSkip Button Clicked")
except TimeoutException:
pass
time.sleep(5)
# find the page number input field
page_num_elem = wait.until(EC.visibility_of_element_located((By.ID,"page-input-field")))
page_num_elem.clear()
page_num_elem.send_keys(num)
time.sleep(2)
while True:
try:
# find go button
go_button = wait.until(EC.element_to_be_clickable((By.ID, "page-input-button")))
go_button.click()
break
except TimeoutException :
print("Retrying...")
continue
# scrape data
soup = BS(driver.page_source, 'lxml')
container = soup.find('section', {'id': 'data-container'})
return [tag.find('div', {'class': 'col i5 company'}).find('a')['href'] for tag in container.findAll('div', {'class': 'row'})]
if __name__ == "__main__":
# create webdriver instance
driver = webdriver.Chrome()
for num in range(5):
print("\nPage Number : %s" % (num+1))
print(scrape(num,driver))
print(90*"-")
driver.quit()
It works for me:
Actions action = new Actions(Browser.WebDriver); action.MoveToElement(element).Click().Perform();
open main(home) page and then go to login page using click() function , now i want to find element in this page how could I?
here is my code ...
import unittest,time,re
from selenium import webdriver
from selenium import selenium
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
class PythonOrgSearch(unittest.TestCase):
def setUp(self):
#self.selenium = selenium("localhost", 4444, "*firefox","http://www.google.com/")
self.driver = webdriver.Firefox()
def test_search_in_python_org(self):
driver = self.driver
driver.get("https://bitbucket.org/")
elem = driver.find_element_by_id("user-options")
elem = elem.find_element_by_class_name("login-link")
elem.click()
print "check"
#elem = WebDriverWait(driver, 30).until(EC.elementToBeVisible(By.name("username")));
#elem.send_keys("my_username#bitbucket.org")
user_name_field = driver.find_element_by_id('id_username')
password_field = driver.find_element_by_id('id_password')
user_name_field.send_keys('your_user_name')
password_field.send_keys('your_password')
def tearDown(self):
pass
#self.driver.close()
if __name__ == "__main__":
unittest.main()
I got this error (file name python_org_search.py)
E
======================================================================
ERROR: test_search_in_python_org (__main__.PythonOrgSearch)
----------------------------------------------------------------------
Traceback (most recent call last):
File "python_org_search.py", line 25, in test_search_in_python_org
user_name_field = driver.find_element_by_id('id_username')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 197, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 681, in find_element
{'using': by, 'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: u'Unable to locate element: {"method":"id","selector":"id_username"}' ; Stacktrace:
at FirefoxDriver.prototype.findElementInternal_ (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/driver_component.js:8860)
at FirefoxDriver.prototype.findElement (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/driver_component.js:8869)
at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/command_processor.js:10831)
at DelayedCommand.prototype.executeInternal_ (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/command_processor.js:10836)
at DelayedCommand.prototype.execute/< (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/command_processor.js:10778)
----------------------------------------------------------------------
Ran 1 test in 30.994s
FAILED (errors=1)
I tried other solutions as well but still same error .
You better should try to retrieve unique elements like the username field and password field by it's id. The name might not be unique and misleading.
Try the following find_element_by_id('id_username') and find_element_by_id('id_password').
Sometimes you need to wait some time that your browser renders the page
and retrieves all contents - so it can be useful to wait some time,
e.g. three seconds before you look deeper for elements.
Code
import time
time.sleep(3)
user_name_field = driver.find_element_by_id('id_username')
password_field = driver.find_element_by_id('id_password')
user_name_field.send_keys('your_user_name')
password_field.send_keys('your_password')
password_field.send_keys(Keys.RETURN)
...
Moreover, I recommend you to use different variables names for the different fields. The variable elem could lead to difficult bugs.
Three seconds of waiting time like mentioned above can be unreliable - a more structured way of assuring some waiting time is it to tell it your driver through implicitly_wait:
driver = webdriver.Firefox()
driver.implicitly_wait(30) # maximally wait 30 seconds before raising an exception.
# ...
# go to the page etc.
There are two ways of waiting time a driver can do:
Wait explicitly: The driver waits this time before executing every next step.
Wait implicitly: If you look for an element the driver searches for this time before he raises an exception.
For more details see the documentation: http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp#implicit-waits
The following code use elem of previous page, which does not exist.
elem = elem.find_element_by_name("username")
Instead, you should use driver:
elem = driver.find_element_by_name("username")
Try
elem = new WebDriverWait(Driver, 30).until(ExpectedConditions.visibilityOfElementLocated(By.name("username")));
elem.send_keys("my_username#bitbucket.org");
To fill any type of form...the best method is go through its name in any case where ever available.
driver=self.webdriver
driver.find_element_by_name("username").click()
driver.find_element_by_name("username").send_keys("XXXXX")
time.sleep(5)
driver.find_element_by_name("password").click()
driver.find_element_by_name("password").send_keys("XXXXX")