Selenium Webdriver How to scrape text? - python

Here's the website code:
<div class="question-row clearfix " lang="en">
<div class="qquestion qtext ">
five
</div>
</div>
I'm trying to scrape the text (in this example, it's 'five'). Here's my current code:
qtitle = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//*[#id='prompt-row'div/div"))).get_attribute("text")
However when I run it, this error code comes up:
File "C:\Users\lycop\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
That's all it says. Any help on solving this problem would be appreciated.

To print the text five you can use either of the following Locator Strategies:
Using css_selector and get_attribute():
print(driver.find_element_by_css_selector("div.qquestion.qtext").get_attribute("innerHTML"))
Using xpath and text attribute:
print(driver.find_element_by_xpath("//div[#class='qquestion qtext ']").text)
Ideally, to print the innerText of an element you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following solutions:
Using CSS_SELECTOR:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button.save[type='submit'][onclick]"))).get_attribute("innerHTML"))
Using XPATH:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//button[#class='save' and text()='save'][#type='submit' and #onclick]"))).text)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element

Related

How to write appropriate Xpath to locate text value

I am trying to use python, selenium and xpath to grab the text $0.180 in the following block of HTML,
<div class="wlp-values-div">
<span class="wlp-last-price-span">$0.180</span>
This code is deep into a websites HTML and I tried to use the following Xpath to locate it:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("https://smallcaps.com.au/stocks/?symbol=AWJ")
time.sleep(3)
webdriver.ActionChains(driver).send_keys(Keys.ESCAPE).perform()
ele = driver.find_element_by_xpath("//div[#class='wlp-values-div']/span[#class='wlp-last-price-span']")
But I receive the error:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#class='wlp-values-div']/span[#class='wlp-last-price-span']"}
Any help would be greatly appreciated
The last price field is within nested <iframe> elements so you have to:
Induce WebDriverWait for the parent frame to be available and switch to it.
Induce WebDriverWait for the child frame to be available and switch to it.
Induce WebDriverWait for the desired element to be visible.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get('https://smallcaps.com.au/stocks/?symbol=AWJ')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "svg[data-name='close']"))).click()
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#wl-summary")))
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe.wl-quote-frame")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span.wlp-last-price-span"))).get_attribute("innerHTML"))
driver.quit()
Using XPATH:
driver.get('https://smallcaps.com.au/stocks/?symbol=AWJ')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//*[name()='svg' and #data-name='close']"))).click()
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='wl-summary']")))
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#class='wl-quote-frame']")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[#class='wlp-last-price-span']"))).get_attribute("innerHTML"))
driver.quit()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Console Output:
$0.190
You can find a detailed relevant discussion in How To sign in to Applemusic With Python Using Chrome Driver With Selenium
Reference
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
<iframe class="wl-summary" id="wl-summary" src="https://cloud.weblink.com.au/clients/smallcaps/summary/summary.aspx?symbol=AWJ&" width="100%" height="25" frameborder="0" scrolling="no" allowtransparency="true" style="height: 1077px;"></iframe>
<iframe class="wl-quote-frame" src="quoteFrame.aspx?symbol=AWJ&" scrolling="no" frameborder="0" height="1" style="height: 177px;"></iframe>
It's in a double nested iframe so you have to switch to it. This also has the appropriate waits instead of using time.sleep() and hoping the elements load. It also clicks the popup to get out of the way and prints the spans text.
wait = WebDriverWait(driver, 10)
driver.get("https://smallcaps.com.au/stocks/?symbol=AWJ")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '#tve_editor > div'))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CLASS_NAME, 'wl-summary')))
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CLASS_NAME, 'wl-quote-frame')))
elem=wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'span.wlp-last-price-span')))
print(elem.text)
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Can't find element on webpage with Selenium

I'm trying to build a bot that will book a class for me when I run it.
But I'm confused about how to proceed.
I'm using Python and Selenium with Chromedriver to load in the page.
The next step is to click the login button, add my data, log in and then navigate the page to find the correct class.
I'm very new to this so it might be very obvious but I can't figure it out.
I'm running this code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
d = webdriver.Chrome()
d.get("http://www.360functionalfitness.se/boka-pass/")
elem = d.find_element_by_name("showLogin")
The following error:
python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[name="showLogin"]"}
(Session info: chrome=87.0.4280.141)
I've been reading a bunch and still can't figure out how to identify the dang button.
This is the block I think I'm supposed to look at to find the relevant info:
<li ng-if="!member" class="ng-scope">
<i class="fa fa-sign-in"><span class="icon-bg bg-darkgreen"></span></i><span class="ng-binding">Logga in</span>
</li>
Any pointers would be extremely helpful.
The element with the text as Logga in is within an <iframe> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("http://www.360functionalfitness.se/boka-pass/")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src^='https://crossfitmalmo.gymsystem.se/member#/schedule']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[ng-click^='showLogin'] span.ng-binding"))).click()
Using XPATH:
driver.get("http://www.360functionalfitness.se/boka-pass/")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[starts-with(#src, 'https://crossfitmalmo.gymsystem.se/member#/schedule')]")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[starts-with(#ng-click, 'showLogin')]//span[#class='ng-binding']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
Reference
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element

Python XPath selector not working with selenium

I tried to automate some inputs. For this I need to input some text after the tag.
To identify the place where to input I tried XPath for fowlloing HTML code.
<span data-offset-key="1dq3m-0-0">
<br data-text="true">
</span>
Here is what I wrote in python.
buf_comp_text = 'foobar'
el_xp_comp_text = '//*[#data-text]'
...
## create post in queue (comment)
print('create post in queue - text')
post_txt = driver.find_element_by_xpath(el_xp_comp_text).send_keys(buf_comp_text)
Unfortunately I alwas get error message:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#data-text]"}
Any hint is appreciated.
Instead of targetting the <br> tag you need to target the <span> tag you can use the following Locator Strategy:
Using css_selector:
buf_comp_text = 'foobar'
driver.find_element_by_css_selector("span[data-offset-key]").send_keys(buf_comp_text)
Using xpath:
buf_comp_text = 'foobar'
driver.find_element_by_xpath("//span[#data-offset-key][.//br[#data-text]]").send_keys(buf_comp_text)
Ideally, to send a character sequence to the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategy:
Using CSS_SELECTOR:
buf_comp_text = 'foobar'
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "span[data-offset-key]"))).send_keys(buf_comp_text)
Using XPATH:
buf_comp_text = 'foobar'
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[#data-offset-key][.//br[#data-text]]"))).send_keys(buf_comp_text)
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element

Selenium returning NoSuchElementException but the element exists

I am having this code
driver.implicitly_wait(300) # I have tried different timers
driver.find_element_by_xpath("(//button[#aria-pressed='false'])[1]").click()
But I get this error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"(//button[#aria-pressed='false'])[1]"}
However, this exists in my HTML content, like:
<div class="webix_el_box" style="width:90px; height:26px">
<button aria-pressed="false" type="button" class="webix_img_btn" style="line-height:22px;">
<span class="webix_icon_btn fa-lock red" style="max-width:22px;">
</span> Read Only
</button>
</div>
I have also tried to use the XPath expression:
//button[#class='webix_img_btn']//child::span[#class='webix_icon_btn fa-lock red']
Both XPath expressions worked fine in Google Chrome.
In the website I am able to find the button using Ctrl + F and 'Read Only', can I use it in Selenium?
It is not working because of Selenium Driver is one page behind ajax call after large json data download.
To click on the element with text as Read Only you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.webix_el_box > button.webix_img_btn"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='webix_el_box']/button[#class='webix_img_btn' and contains(., 'Read Only')]"))).click()
Note: You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on NoSuchElementException in:
"selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element" while clicking a 'Next' button with Selenium
Selenium in Python: "NoSuchElementException: Message: no such element: Unable to locate element"
Try doing just
driver.implicitly_wait(300)
driver.find_element_by_xpath("//button[#aria-pressed='false']").click()
This will click on the first element that satisfies the requirements.

How to identify the nested element using Selenium and Python

I have a nested element possibly within <svg> that I can't seems to access
I tried using
driver.find_element(By.CSS_SELECTOR, 'button.login-fake-btn')
and
driver.find_element(By.CSS_SELECTOR, 'login-fake-btn')
and a few others.
HTML structure of nested svg:
<svg class="1">
<div id="2">
<div>
<div class="3">
<div class="4">
<li>
<button class="5" type="button" id="login-fake-btn">
...closing tags
Snapshot of HTML:
I have no success with xpath either.
Error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"button.login-fake-btn"}
How do I get to a nested svg using a css selector (or xpath, but I understand css to be better)?
It's a <button> element and it's out of the <svg> tag and possibly moving forward you'd invoke click() on it. Hence to locate the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.login-btn.btn-shadow#login-fake-btn[data-testid='login-fake-btn']")))
Using XPATH:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#class='login-btn btn-shadow' and #id='login-fake-btn'][#data-testid='login-fake-btn']")))
Note : You have to add the following imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element

Categories