I've tried using getText() Referencing, but when I use that method I always get the error "WebElement object has no attribute getText()". I found .text through a YouTube video but I'm not getting results from it either. Any thoughts?
from selenium import webdriver
driverPath = 'C:\Program Files (x86)\chromedriver.exe'
driver = webdriver.Chrome(driverPath)
driver.implicitly_wait(20)
driver.get ('https://shop.pricechopper.com/search?search_term=lettuce')
pageSource = driver.page_source
search = driver.find_elements_by_class_name ("css-dr3s47")
print (search[0].text)
driver.quit()
You are using Selenium with Python, so correct method is .text
.getText() is for Selenium-Java Bindings.
The reason why are you getting null is cause elements are not rendered properly and you are trying to interact with them resulting in null.
Please use Explicit wait or time.sleep() which is worst case to get rid of this issue.
Also, there's a pop window that we will have to click on close web element, so that css-dr3s47 will be available to Selenium view port.
Code :
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)
driver.get("https://shop.pricechopper.com/search?search_term=lettuce")
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[#ng-click='processModalCloseClick()']"))).click()
print('Clicked on modal pop up closer button')
except:
pass
search = driver.find_elements_by_class_name ("css-dr3s47")
print (search[0].text)
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
getText returns elements with rendered size more than 0 * 0 , so if element is not displayed of has size zero , then it wont return anything .
Two things you could do is :
scroll to the element before getting text
Actions actionProvider = new Actions(driver);
// Performs mouse move action onto the element
actionProvider.moveToElement(search[0]).build().perform();
print(search[0].text)
Use textCOntent
Text content doesn't check for display or size , it just gives content
print(search[0].get_attribute("textContent"))
Related
import time
driver = webdriver.Chrome()
driver.get("https://www.canva.com/q/pro-signup/")
time.sleep(6)
driver.switch_to.frame(driver.find_element_by_class_name('rbV9vo63iaj7sGd7XwS4h'))
elem = driver.find_element_by_name("//iframe[contains(#name, '_hjRemote')]")
It cant find element last line. i tried contains, starts with and indexing but none worked.
Try using different elements such as the xpath or the id. If that fails then you could select the element by the css selector. If that fails then you could always use a lib like pyautogui to physically click on the web element.
There are total of 6 iframes, The elements you are looking, they are inside
iframe[src^='https://www.canva.com/']
this iframe.
so you need to switch to this frame first :
driver.switch_to.frame(driver.find_element_by_css_selector("iframe[src^='https://www.canva.com/']"))
I would use the below code to click on Sign up with email:
driver = webdriver.Chrome()
driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://www.canva.com/q/pro-signup/")
wait = WebDriverWait(driver, 10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[src^='https://www.canva.com/']")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Sign up with email']/.."))).click()
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
In case you want to have a predefined iframe stored, you could something like this :
remote_vars_frame = driver.find_element_by_css_selector("iframe[id='_hjRemoteVarsFrame']")
driver.switch_to.frame(remote_vars_frame)
You can handle this Exception by directly searching for the element without switching to any iframe as below
elem = driver.find_element_by_name("//iframe[contains(#name, '_hjRemote')]")
My code goes into this website, and clicks on each row of a table.
These rows open a window, which contain a field "keywords" at the bottom of the new window, which I am trying to get.
I dont think I have the xpath for this field right, though all I did was right click and "copy xpath".
Expected output is to print the value in the keywords.
from selenium import webdriver
from bs4 import BeautifulSoup
import time
import requests
driver = webdriver.Chrome()
driver.get('https://aaaai.planion.com/Web.User/SearchSessions?ACCOUNT=AAAAI&CONF=AM2021&USERPID=PUBLIC&ssoOverride=OFF')
time.sleep(3)
page_source = driver.page_source
soup = BeautifulSoup(page_source,'html.parser')
eachRow=driver.find_elements_by_class_name('clickdiv')
for item in eachRow:
item.click() #opens the new window per each row
time.sleep(2)
faculty = driver.find_element_by_xpath("//td[#valign='MIDDLE']/b") #this works fine
keywords=item.find_element_by_xpath('//*[#id="W1"]/div/div/div/div[2]/div[2]/table/tbody/tr[5]/td/text()') #this does not
print(keywords.text)
print(faculty.text)
driver.find_element_by_class_name('XX').click()#closes window
ERROR message - selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: The result of the xpath expression "//*[#id="W1"]/div/div/div/div[2]/div[2]/table/tbody/tr[5]/td/text()" is: [object Text]. It should be an element.
.....
Your XPath '//*[#id="W1"]/div/div/div/div[2]/div[2]/table/tbody/tr[5]/td/text()' is getting the actual text of the element, not the entire element itself. Just remove text() and it should work. XPath is notoriously finnicky, so be careful how you use it, and how you are defining your top level root.
I reworked your code very slightly here to give a better option than time.sleep(), explicit waits. It isn't necessary, but tends to make more reliable code and speeds up processing; it doesn't pause the process unless it has to, and only does so for as long as necessary.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('https://aaaai.planion.com/Web.User/SearchSessions?ACCOUNT=AAAAI&CONF=AM2021&USERPID=PUBLIC&ssoOverride=OFF')
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'clickdiv')))
eachRow = driver.find_elements_by_class_name('clickdiv')
for item in eachRow:
item.click() # opens the new window per each row
faculty = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//td[#valign='MIDDLE']/b")))
keywords = item.find_element_by_xpath('//*[#id="W1"]/div/div/div/div[2]/div[2]/table/tbody/tr[5]/td')
print(keywords.text)
print(faculty.text)
driver.find_element_by_class_name('XX').click() # closes window
I am having trouble selecting a load more button on a Linkedin page. I receive this error in finding the xpath: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element
I suspect that the issue is that the button is not visible on the page at that time. So I have tried actions.move_to_element. However, the page scrolls just below the element, so that the element is no longer visible, and the same error subsequently occurs.
I have also tried move_to_element_with_offset, but this hasn't changed where the page scrolls to.
How can I scroll to the right location on the page such that I can successfully select the element?
My relevant code:
import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('C:\\Users\\Root\\Downloads\\chromedriver.exe')
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)
username = driver.find_element_by_name('session_key')
username.send_keys(parameters.linkedin_username)
sleep(0.5)
password = driver.find_element_by_name('session_password')
password.send_keys(parameters.linkedin_password)
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//button[#class="btn__primary--large from__button--floating"]')
sign_in_button.click()
driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')
loadmore_skills=driver.find_element_by_xpath('//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')
actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()
After playing around with it, I seem to have figured out where the problem is stemming from. The error
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]"}
(Session info: chrome=81.0.4044.113)
always correctly states the problem its encountering and as such it's not able to find the element. The possible causes of this include:
Element not present at the time of execution
Dynamically generated
content Conflicting names
In your case, it was the second point. As the content that is displayed is loaded dynamically as you scroll down. So When it first loads your profile the skills sections aren't actually present in the DOM. So to solve this, you simply have to scroll to the section so that it gets applied in the DOM.
This line is the trick here. It will position it to the correct panel and thus loading and applying the data to the DOM.
driver.execute_script("window.scrollTo(0, 1800)")
Here's my code (Please change it as necessary)
from time import sleep
# import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('../chromedriver.exe')
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)
username = driver.find_element_by_name('session_key')
username.send_keys('')
sleep(0.5)
password = driver.find_element_by_name('session_password')
password.send_keys('')
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//button[#class="btn__primary--large from__button--floating"]')
sign_in_button.click()
driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')
sleep(3)
# driver.execute_script("window.scrollTo(0, 1800)")
sleep(3)
loadmore_skills=driver.find_element_by_xpath('//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')
actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()
Output
Update
In concerns to your newer problem, you need to implement a continuous scroll method that would enable you to dynamically update the skills section. This requires a lot of change and should ideally be asked as a another question.
I have also found a simple solution by setting the scroll to the correct threshold. For y=3200 seems to work fine for all the profiles I've checked including yours, mine and few others.
driver.execute_script("window.scrollTo(0, 3200)")
If the button is not visible on the page at the time of loading then use the until method to delay the execution
try:
myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
print "Button is rdy!"
except TimeoutException:
print "Loading took too much time!"
Example is taken from here
To get the exact location of the element, you can use the following method to do so.
element = driver.find_element_by_id('some_id')
element.location_once_scrolled_into_view
This actually intends to return you the coordinates (x, y) of the element on-page, but also scroll down right to target element. You can then use the coordinates to make a click on the button. You can read more on that here.
You are getting NoSuchElementException error when the locators (i.e. id / xpath/name/class_name/css selectors etc) we mentioned in the selenium program code is unable to find the web element on the web page.
How to resolve NoSuchElementException:
Apply WebDriverWait : allow webdriver to wait for a specific time
Try catch block
so before performing action on webelement you need to take web element into view, I have removed unwated code and also avoided use of hardcoded waits as its not good practice to deal with synchronization issue. Also while clicking on show more button you have to scroll down otherwise it will not work.
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome(executable_path="path of chromedriver.exe")
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("email id")
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("password ")
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//button[#class='btn__primary--large from__button--floating']"))).click()
driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()
driver.execute_script("scroll(0, 250);")
buttonClick = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//span[text()='Show more']")))
ActionChains(driver).move_to_element(buttonClick).click().perform()
Output:
So I'm scraping using selenium and I want to click 'next' button in 'Defensive' section but the code I wrote clicks 'next' on 'Summary'.
Here's the url for you to try :
https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019
So it's selecting 'Defensive' and I can see it selected in the window but the next page doesnt appear. On clicking 'Summary' I found out next function is actually happening there.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser= webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe")
browser.get('https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019')
browser.find_element_by_xpath("""//*[#id="stage-top-player-stats-options"]/li[2]/a""").click()
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, """//*[#id="next"]""")))
browser.execute_script("arguments[0].click();", element)
The xpath for next button is not unique for this page. try this,
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//*[#id='stage-top-player-stats-defensive']//a[#id='next']")))
browser.execute_script("arguments[0].click();", element)
or
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//*[#id='stage-top-player-stats-defensive']//a[#id='next']")))
element.click()
For each tab (Summary, Defensive, ..) new next button with same id=next added to the DOM.
Select Defensive and you will see there will be two next buttons with same id=next, select Offensive and there will be three next buttons.
With basic id=next selector you always click to the first next button from Summary tab. Because you're using JavaScript and nothing happen, try to click with Selenium click method and you will get an error.
To solve the problem adjust your selector to be more specific to the dom - #statistics-paging-defensive #next.
Also when you first time open the page there's cookies acceptance screen appears and block the page, you can use method like below to skip it.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import selenium.common.exceptions as EX
def accept_cookies():
try:
WebDriverWait(browser, 20)\
.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.qc-cmp-button")))\
.click()
except EX.NoSuchElementException or EX.TimeoutException:
pass
#...
browser = webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe")
browser.get('https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019')
wait = WebDriverWait(browser, 20)
browser.get(baseUrl)
accept_cookies()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[href='#stage-top-player-stats-defensive']"))).click()
next_button = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "#statistics-paging-defensive #next")))
next_button.click()
Your elements locators must be unique
Avoid using XPath wildcards - * as it will cause performance degradation and prolonged elements lookup timings
Avoid using JavaScriptExecutor for clicking, well-behaved Selenium test must do what real user does and I doubt that real user will be opening browser console and typing something like document.getElementById('next').click(), he will use the mouse
Assuming all above you should come up with a selector which uniquely identifies next button on Defensive tab which would be something like:
//div[#id='statistics-paging-defensive']/descendant::a[#id='next']
References:
XPath Tutorial
XPath Axes
XPath Operators & Functions
import webbrowser
from selenium import webdriver
browser = webdriver.Chrome()
browser.maximize_window()
browser.get('https://www.suntrust.com/')
browser.implicitly_wait(15)
elem = browser.find_element_by_css_selector('input#sign-on-3A69E29D-79E0-
403E-9352-5261239ADD89-user')
elem.click().send_keys('your-username')
element not visible exception error message:
I'm trying to sign into the login/password field automatically, but I keep getting this error message.
I have tried various "find_element_by" locators, but this one was recommended, so I don't think the css selector is the problem. What am I doing wrong?
It happens usually because the dom wouldn't have loaded and the Selenium script tries to find that element .. Make sure this element is not inside an Iframe . Use the selenium explicit wait until the element loads and then perform action on that button . You have to do something like this in python . The below code is just creating a wait object and then waiting for the element to load and perform next step
from selenium.webdriver.support.ui import WebDriverWait
myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
print "Page is ready!"
except TimeoutException:
print "Loading took too much time!"
Here is the Answer to your Question:
The css_selector you constructed was not unique and was matching to 2 elements on the HTML DOM. The first match was invisible while the second match was our expected element. Selenium was trying to click the first element. Hence the error. Here is your own code with the modified css_selector which works well at my end:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
browser = webdriver.Chrome(chrome_options=options, executable_path="C:\\Utility\\BrowserDrivers\\chromedriver.exe")
browser.get('https://www.suntrust.com/')
browser.implicitly_wait(15)
elem = browser.find_element_by_css_selector('section[role="main"] input[id="sign-on-3A69E29D-79E0-403E-9352-5261239ADD89-user"]')
elem.send_keys('your-username')
Let me know if this Answers your Question.