I wan't to get the value or the price of a stock from a trading website. The problem is, that when i'm using the .get attribute method like this:
.get_attribute('')
I can't seem to find anything to put in between the '' that will give me the value of the stock
Here is an image of the line when using inspect:
<span _ngcontent-c31="" class="price__value" style="" xpath="1"> 187.510 </span>
This is the code below that i've been making for this:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
browser = webdriver.Chrome('/Users/ludvighenriksen/downloads/chromedriver')
browser.get('https://www.forex.com/en-uk/account-login/')
username_elem = browser.find_element_by_name('Username')
username_elem.send_keys('kebababdulaziz#gmail.com')
password_elem = browser.find_element_by_name('Password')
password_elem.send_keys('KEbababdulaziz')
password_elem.send_keys(Keys.ENTER)
time.sleep(5)
search_elem = WebDriverWait(browser, 20).until(EC.element_to_be_clickable(
(By.CSS_SELECTOR, "input.market-search__search-input")))
search_elem.click()
search_elem.send_keys('FB')
search_click_elem = WebDriverWait(browser, 20).until(EC.element_to_be_clickable(
(By.XPATH, "//app-market-table[#class='search-results-element ng-star-inserted']//div[#class='price--buy clickable-price arrows-flashing']")))
browser.execute_script("arguments[0].click();", search_click_elem)
price_elem = browser.find_element_by_css_selector("div.mercury:nth-child(2) div.mercury__body:nth-child(4) div.mercury__body-content-container app-workspace.ng-star-inserted:nth-child(3) div.panel-container:nth-child(1) app-workspace-panel.active.ng-star-inserted div.workspace-panel-content.workspace-panel-content--no-scroll-vertical.workspace-panel-content--no-scroll-horizontal.workspace-panel-content--auto-size div.workspace-panel-content__component.workspace-panel-content__component--auto-size app-deal-ticket.ng-star-inserted form.ticket-form.ng-untouched.ng-pristine.ng-invalid.ng-star-inserted div.market-prices app-market-prices.main-prices.ng-untouched.ng-pristine.ng-valid div.market-prices div.market-prices__direction label.buy.selected span.price.ng-star-inserted:nth-child(2) > span.price__value")
price_value = price_elem.get_attribute('value')
print(price_value)
The ('value') isn't working which makes sense i guess, but I think i've tried all that i could think of - and it prints out none.
The log in to the website is included if you want to try it out, thanks in Advance
If you want to access the content of some tag, you could use the .text option.
Related
I'm having trouble trying to get the colour of a vehicle in Selenium using Python. I've checked YouTube, stackoverflow and all the usual resources but can't seem to find an answer that makes sense (I'm relatively new to Python and Selenium). I'm currently undertaking a project to automate the fetching of vehicle colour from the gov.uk website into an excel sheet based on the Vehicle Registration number already present on the spreadsheet. The code isn't finished yet, as I want to get over this Xpath hurdle first!
I need to fetch the 'Blue' value from this code:
<dl class="summary-no-action">
<div class="govuk-summary-list__row">
<dt>Registration number</dt>
<dd>
<div class="reg-mark-sm">WJ06 HYF</div>
</dd>
</div>
<div class="govuk-summary-list__row">
<dt>Make</dt>
<dd>VOLKSWAGEN</dd>
</div>
<div class="govuk-summary-list__row">
<dt>Colour</dt>
<dd>BLUE</dd>
</div>
</dl>
However, as you can see, they have made it very difficult as there is no specific ID, class, tag name, etc to work with so I'm assuming Xpath is my only option? could anyone help me as to the best implementation of this? My assumption is to find the first 'dd' tag underneath 'Colour' but I don't know how to write this!
Here is the code snippet I'm working on that I have so far:
try:
main = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, "summary-no-action"))
)
div = main.find_element(By.LINK_TEXT, "Colour")
for article in div:
header = article.find_element(By.TAG_NAME, "dd")
print(header.text)
finally:
driver.quit()
I'm aware the line 'div = main.find_element(By.LINK_TEXT, "Colour")' is incorrect, but I need to replace it with something so that I may fetch the colour present in the 'dd' tag underneath.
This is what I had originally, but it brings back all the values in the "summary-no-action" class name:
try:
main = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, "summary-no-action"))
)
div = main.find_elements(By.CLASS_NAME, "govuk-summary-list__row")
for article in div:
header = article.find_element(By.TAG_NAME, "dd")
print(header.text)
finally:
driver.quit()
Any help would be appreciated!
EDIT:
For reference, here is the whole code:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver =
webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get("https://www.google.com")
driver.get ("https://vehicleenquiry.service.gov.uk/")
time.sleep(5)
search = driver.find_element(By.ID ,
"wizard_vehicle_enquiry_capture_vrn_vrn")
search.send_keys("wj06hyf")
search.send_keys(Keys.RETURN)
try:
main = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, "summary-no-action"))
)
div = main.find_elements(By.CLASS_NAME, "govuk-summary-list__row")
for article in div:
header = article.find_element(By.TAG_NAME, "dd")
print(header.text)
finally:
driver.quit()
Use the following xpath to get the value BLUE. first identify the dt tag with text colour and following dd tag
//dt[text()='Colour']/following::dd[1]
code:
print(WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH, "//dt[text()='Colour']/following::dd[1]"))).text)
To fetch the text Blue you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:
Using CSS_SELECTOR and get_attribute("innerHTML"):
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "dl.summary-no-action div:last-child dd"))).get_attribute("innerHTML"))
Using XPATH and text attribute:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//dl[#class='summary-no-action']//div[#class='govuk-summary-list__row']/dt[text()='Colour']//following-sibling::dd[1]"))).text)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
As title. Couldn't find a solution online. You'll see by the print statements that the program will get the text property from the WebElement object, but it is always an empty string even though I am using WebDriverWait.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
PATH = "C:\\Users\\Anthony\\Desktop\\chromedriver.exe"
driver = webdriver.Chrome(PATH)
website = "https://www.magicspoiler.com"
driver.get(website)
el_name = '//div[#class="set-card-2 pad5"]'
try:
main = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, el_name)))
articles = main.find_elements(By.XPATH, el_name)
for article in articles:
print(type(article.text))
print(article.text)
finally:
driver.quit()
I checked the HTML DOM and I don't see any text in the element you are looking for. They are simply images with anchor links and therefore you are getting blank responses.
I have given an alternate try, to extract the links and they were successful.
So, after refactoring your code, it looks like this:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
PATH = "C:\\Users\\Anthony\\Desktop\\chromedriver.exe"
driver = webdriver.Chrome(PATH)
website = "https://www.magicspoiler.com"
driver.get(website)
el_name = '//div[#class="set-card-2 pad5"]/a'
try:
main = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, el_name)))
articles = main.find_elements(By.XPATH, el_name)
for article in articles:
# print(type(article.text))
print(article.get_attribute("href"))
finally:
pass
driver.quit()
Here is the response I got:
So, please check if you want to get the links or really the text. If you want the text, then there is none that I see in DOM as I said. You may have to traverse through each of the link by clicking on it, and find if any you need fro the navigated page.
I'm trying to scrape some data from a website and I can't seem to get the text in between two tags as it keeps returning None. This is my code, could somebody tell me what is wrong?
uq = "https://my.uq.edu.au/programs-courses/requirements/program/2451/2021"
driver = webdriver.Firefox(executable_path="/Users/Connor/Downloads/geckodriver")
driver.get(uq)
groups = driver.find_element_by_xpath("/html/body/div/div[2]/div/div/div[3]/div[1]/div[2]/div[4]/div[1]/div[2]/a[1]/span[1]")
print(groups)
print(groups.text) #This won't get the text between the span tags e.g. <span>hello</span>
To get that text you'll need to click the parent element first.
However, you can avoid it with .get_attribute("innerHTML") and waiting for the presence of the element. Also, make sure you are using stable locators.
from selenium import webdriver
from selenium.webdriver.firefox.options import Options as FirefoxOptions
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
options = FirefoxOptions()
options.add_argument('window-size=1920x1080')
driver = webdriver.Firefox(options=options)
uq = "https://my.uq.edu.au/programs-courses/requirements/program/2451/2021"
driver.get(uq)
wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "a[title='Theory of Computing']>span:nth-of-type(1)")))
group = driver.find_element_by_css_selector("a[title='Theory of Computing']>span:nth-of-type(1)").get_attribute("innerHTML")
print(group)
Prints: COMP2048
You have to set an action to get the value of the WebElement.
So you have to use this code instead:
print(groups.getText());
Yes, same for me. I have tried every way possible. Trying this:
print(speed.get_attribute('outerHTML'))
I get
<span data-download-status-value="NaN" class="result-data-large number result-data-value download-speed"> </span>
I am trying to extract the text "Margaret Osbon" from HTML below via Python with Selenium. But I keep getting blank values when I print. I have tried get_attribute
Still getting blank values when I print
<div class="author-info hidden-md">
By (author)
<span itemprop="author" itemtype="http://schema.org/Person" itemscope="Margareta Osborn">
<a href="/author/Margareta-Osborn" itemprop="url">
<span itemprop="name">
Margareta Osborn</span>
</a>
</span>
</div>
Below is my code for Python
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time"
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.bookdepository.com/")
keyword = "9781925324402"
Search = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '//*[#id="book-search-form"]/div[1]/input[1]'))
)
Search.clear()
Search.send_keys(keyword)
Search.send_keys(Keys.RETURN)
try:
authors = driver.find_element_by_xpath("//div[#class='author-info hidden-md']/span/a/span").text
print(authors)
driver.quit()
except:
authors = "Not Available"
print(authors)
driver.quit()
You need to call the .text method which is present in the Selenium Python binding.
.text is present for web element
authors = driver.find_element_by_xpath("//div[#class='author-info hidden-md']/span/a/span").text
print(authors)
or
authors = driver.find_element_by_xpath("//a[contains(#href,'/author/Margareta-Osborn')]").get_attribute('innerHTML')
print(authors)
Update 1 :
driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get("https://www.bookdepository.com/Rose-River-Margareta-Osborn/9781925324402")
authors = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.author-info.hidden-md span[itemprop='author'] span"))).text
print(authors)
You are missing ".text" to get the value and maybe because of that you are getting some junk value. I am thinking that you are receiving just a reference ID for that.
Using .text -
#Get Element using Xpath
element = //span[#itemprop='name']
#Fetch using the driver findElement
author = driver.find_element_by_xpath(element).text
#Print the text
print(author)
Using JavaScriptExecutor -
driver.execute_script('return arguments[0].innerText;', element)
Using Get Attribute -
driver.find_element_by_xpath(element).get_attribute('innerText')
To get the value from span. Use WebDriverWait() and wait for visibility_of_element_located() and following css selector.
and use either .text or .get_attribute("textContent"))
driver.get('https://www.bookdepository.com/Rose-River-Margareta-Osborn/9781925324402')
print(WebDriverWait(driver,5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.author-info.hidden-md [ itemprop="author"]'))).text)
print(WebDriverWait(driver,5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.author-info.hidden-md [ itemprop="author"]'))).get_attribute("textContent"))
you need to import below libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
<div _ngcontent-tpr-c123="" class="panel panel--bordered">hac-action: 1: /wae:wae/wae-ha:ha-config/ha-data-sync</div>
old_config_xpath = "//div[contains(text(),'hac-action')]"
element_value = wae_base_params[self.driver_key].find_element_by_xpath(old_config_xpath).text
element_value is empty.
note: the above HTML code is not a visible content on the webpage
The way any element is located is when it's on the DOM.
The way to be sure the element is located on the DOM is using WebDriverWait with expected_conditions:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(browser, 10)
element_value = wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(text(),'hac-action')]]"))).text
I didn't see the page source of the website, but please test the following code, and let me know the result. In the past, I've used get_attribute('innerHTML') instead of 'text' and it worked properly. I'd be very happy if you let me know the result.
old_config_xpath = "//div[contains(text(),'hac-action')]"
element_value = wae_base_params[self.driver_key].find_element_by_xpath(old_config_xpath).get_attribute('innerHTML')