Basically, I am trying to create a bot that can find an element on a page with a given raw_input text, finding the button, and clicking it. The purpose of this is to click on links (buttons) that may not be on the web page yet, but will appear after the site refreshes. Because of this, I cannot find elements by XPATH, because the XPATH will be unknown until the second the link becomes available. So, my question is: is there a way for Selenium to find a button based on text and click it? Here is some of my code:
key1 = raw_input('Enter the first keyword: ')
key2 = raw_input('Enter the second keyword: ')
key3 = raw_input('Enter the third keyword: ')
...
elem1 =driver.find_elements_by_xpath("//*[contains(text(), key1)]")
if elem1.is_displayed():
elem1.click()
else:
print "Cannot find element to click. Retrying..."
driver.refresh()
I need the program to find the text based on the keywords, and then click the button / link with these keywords in it. Is this possible? or only possible with XPATHs? The example code i gave with giving an elem has not been working.
thanks in advance
Ok first of all, the only one of selecting an element by text is sadly xpath, but do not worry, there is a workaround you could use waits
#Libraries (those comes with selenium, you don't have to install anything)
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#Way
elem1 = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, '//*[contains(text(),
{})]'.format(key1))
)
elem1.click()
this will make it wait for the element to be clickable if the element is not clickable after 10 seconds it will raise an error if it gets clickable before that time it will return the element and break the wait automatically.
Related
The company has a list of 100+ sites that I am trying to use Selenium webdriver to automatically take a user into that site. I am fairly new to programming so please forgive me if my question is worded poorly.. But, I am trying to take the name of a site such as "Alpharetta - Cemex" in the example below from the user and find it in this long list and then select that link. Through testing I am pretty sure the element I need to click is the h3 class that also holds the name of the site under the data-hmi-name
Website Code Example:
I have tried to use the below and it never seems to work..
driver.find_element_by_css_selector("h3.tru-card-head-text uk-text-center[data-hmi-name='Alpharetta - Cemex']").click()
#For this one I tried to select the h3 class by searching for all those elements that has the name Alpharetta - Cemex
or
**theCards = main.find_elements_by_tag_name("h3")** #I tried both of these declarations for theCards
**#theCards = main.find_elements_by_class_name("tru-card-wrapper")**
#then used the loop below. This obviously didn't work and it just returns an error that card.text doesn't actually exist
for card in theCards:
#title = card.find_elements_by_tag_name("h3")
print(card.text)
if(card.text == theSite):
card.click()
Any help or guidance would be so appreciated! I am new to programming in Python and if you can explain what I am doing wrong I'd be forever thankful!
If you want to click a single link (e.g. Alpharetta - Cemex) , you can try like below:
theSite = "Alpharetta - Cemex" #You can store user inputted site Name here
linkXpath = "//a[h3[contains(text(),'"+theSite +"']]"
WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.XPATH, linkXpath))).click() #This will wait for element to be clickable before it clicks
In case above is not working. If your Link is not in screen / not visible. You can use java script to first scroll to element and the click like below:
ele = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, linkXpath)))
driver.execute_script("arguments[0].scrollIntoView();", ele )
driver.execute_script("arguments[0].click();", ele )
You need to Import:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
hi I'm new to web scraping and have been trying to use Selenium to scrape a forum in python
I am trying to get Selenium to click "Next" until the last page but I am not sure how to break the loop. and I having trouble with the locator:
When I locate the next button by partial link, the automated clicking will continue to next thread e.g page1->page2->next thread->page1 of next thread-->page2 of next thread
while True:
next_link = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, "Next")))
next_link.click()
When I locate the next button by class name, the automated clicking will click "prev" button when it reaches the last page
while True:
next_link = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "prevnext")))
next_link.click()
My questions are:
Which locator should I use? (by class or by partial link or any
other suggestion?
How do I break the loop so it stops clicking when it reaches the
last page?
There are a couple of things you need to consider as follows :
There are two elements on the page with text as Next one on Top and another at the Bottom, so you need to decide with which element you desire to interact and construct a unique Locator Strategy
Moving forward as you want to invoke click() on the element instead of expected-conditions as presence_of_element_located() you need to use element_to_be_clickable().
When there would be no element with text as Next you need to execute the remaining steps, so invoke the click() within try-catch block and incase of an exception break out.
As per your requirement xpath as a Locator Strategy looks good to me.
Here is the working code block :
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://forums.hardwarezone.com.sg/money-mind-210/hdb-fully-paid-up-5744914.html")
driver.find_element_by_xpath("//a[#id='poststop' and #name='poststop']//following::table[1]//li[#class='prevnext']/a").click()
while True:
try :
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[#id='poststop' and #name='poststop']//following::table[1]//li[#class='prevnext']/a[contains(.,'Next')]"))).click()
except :
print("No more pages left")
break
driver.quit()
Console Output :
No more pages left
You can use below code to click Next button until the last page reached and break the loop if the button is not present:
from selenium.common.exceptions import TimeoutException
while True:
try:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, "Next ›"))).click()
except TimeoutException:
break
You can use any locator which gives unique identification. Best practices says the following order.
Id
Name
Class Name
Css Selector
Xpath
Others
The come out of the while loop when it is not find the element you can use try block as given below. the break command is used for the same.
while True:
try:
next_link = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "prevnext")))
next_link.click()
except TimeoutException:
break
I am trying to go to the drought monitor website and tell it to select to show county data. I am able to get my code to navigate to the website, and it clicks the dropdown, but I cannot get it to type in "county". My code gets to the last line and then give the error: "Cannot focus element".
Any help would be greatly appreciated as I'm very new to Selenium.
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
browser = webdriver.Chrome()
browser.get('http://droughtmonitor.unl.edu/Data/DataDownload/ComprehensiveStatistics.aspx')
browser.maximize_window()
dropdown = browser.find_element_by_xpath("""//*
[#id="dnn_ctr1009_USDMservice_CompStats_2017_aoiType_chosen"]""")
dropdown.click()
dropdown.send_keys('county')
dropdown.submit()
print("I'm done")
You're sending keys to the <div> that contains the search <input>, rather than to the <input> element itself. You'll need to find the <input> and send it the keys.
(Note: You also don't need to use XPath for something as simple as a lookup by id.)
dropdown = browser.find_element_by_id("dnn_ctr1009_USDMservice_CompStats_2017_aoiType_chosen")
dropdown.click()
search = dropdown.find_element_by_tag_name("input")
search.send_keys("county", Keys.ENTER)
I'm learning how to use selenium and I'm stuck on figuring out how to scroll down in a website to verify an element exists.
I tried using the methods that was found in this question
Scrolling to element using webdriver?
but selenium won't scroll down the page. Instead it'll give me an error
"selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: element"
Heres the codes I am using
moveToElement:
element = driver.find_element_by_xpath('xpath')
actions = ActionChains(driver)
actions.move_to_element(element).perform()
Scrolling into View
element = driver.find_element_by_xpath('xpath')
driver.execute_script("arguments[1].scrollIntoView();", element)
The whole code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Firefox()
driver.get("https://www.linkedin.com/")
element =
driver.find_element_by_xpath('/html/body/div/main/div/div[1]/div/h1/img')
element = driver.find_element_by_xpath('//*[#id="login-email"]')
element.send_keys('')
element = driver.find_element_by_xpath('//*[#id="login-password"]')
element.send_keys('')
element = driver.find_element_by_xpath('//*[#id="login-submit"]')
element.click();
element = driver.find_element_by_xpath('')
actions = ActionChains(driver)
actions.move_to_element(element).perform()
There are two aspects to your is problem
Whether the element exist?
Whether the element displayed on the page?
Whether the element exist?
It may happen that the element exists on the page[i.e. it is part of the DOM] but it is not available right away for further selenium action becuase it is not visible[hidden], it's only gets visible after some actions or it may get displayed on scroll down the page.
Your code is throwing an exception here-
element = driver.find_element_by_xpath('xpath')
as WebDiver not able to find the element using mentioned xpath. Once you fix this, you can moe forward the next part.
Whether the element displayed on the page?
Once you fix the above issue, you should check whether the element is being displayed or not. If it's not displayed and avialble on the scroll then you can use code like
if !element.is_displayed():
driver.execute_script("arguments[1].scrollIntoView();", element)
Perfer using Action Class for very specific mouse action.
Update:-
If you are application using lazy loading and the element you are trying to find is available on scroll the you can try something like this -
You have to import exception like -
from selenium.common.exceptions import NoSuchElementException
and the create new recursion function which would scroll if element not found, something like this -
def search_element():
try:
elem = driver.find_element_by_xpath("your")
return elem
except NosSuchElementException:
driver.execute_script("window.scrollTo(0,Math.max(document.documentElement.scrollHeight,document.body.scrollHeight,document.documentElement.clientHeight));")
search_element()
I am not sure that having recurion for finding element here is good idea, also I have naver worked on the python so you need to lookout for the syntax
Amm may be this might help.Just send page down key and if you are sure that element exists definitely then this would work
from selenium.webdriver.common.keys import Keys
from selenium.webdriver import ActionChains
import time
while True:
ActionChains(driver).send_keys(Keys.PAGE_DOWN).perform()
time.sleep(2) #2 seconds
try:
element = driver.find_element_by_xpath("your_element_xpath")
break
except:
continue
I have a div which contains the results for a certain search query. The text contained in this div changes as a button to go to the next page is clicked.
In the text contained in this div, there is also the corresponding number of the page. After clicking in the button to go to the next page, the results still take a bit to load, so I want to make the driver wait the content to load.
As such, I want to wait until the string "Page x" appears inside the div, where x corresponds to the number of the next page to be loaded.
For this, I tried:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
# Search page
driver.get('http://searchpage.com')
timeout = 120
current_page = 1
while True:
try:
# Wait until the results have been loaded
WebDriverWait(driver, timeout).until(
EC.text_to_be_present_in_element(
locator=(By.ID, "SEARCHBASE"),
text_="Page {:}".format(current_page)))
# Click to go to the next page. This calls some javascript.
driver.find_element_by_xpath('//a[#href="#"]').click
current_page += 1
except:
driver.quit()
Although, this always fails to match the text. What am I doing wrong here?
To just detect if anything whatsoever had changed in the page would also do the job, but I haven't found any way to do that.
Try to apply below solution to wait for partial text match:
WebDriverWait(driver, timeout).until(lambda driver: "Page {:}".format(current_page) in driver.find_element(By.ID, "SEARCHBASE").text)