Retrieving dynamic value with selenium webdriver, python - python

I am aware that there already exists similar threads about this. However, when trying previously suggested methods to retrieve my specific dynamic table value, all I am getting is either a nbsp value or something cryptic like "1a207feb-8080-4ff0-..."
What I am trying to do:
Get the current table value for euro/oz value for gold from here. I "inspected" the page and got the xpath (//*[#id="bullionPriceTable"]/div/table/tbody/tr[3]/td[3]/span)
My code:
driver = webdriver.Chrome("path/to/chromedriver")
driver.get("https://www.bullionvault.com/gold-price-chart.do")
xpath = '//*[#id="bullionPriceTable"]/div/table/tbody/tr[3]/td[3]/span'
select=driver.find_element_by_xpath(xpath)
print(select)
This prints:
<selenium.webdriver.remote.webelement.WebElement (session="3ade114e9f0907e4eb13deac6a264fc8", element="3a670af5-8594-4504-908a-a9bfcbac7342")>
which obviously is not the number I was looking for.
I've also experimented with using get_attribute('innerHtml') and .text on the webElement, but to no avail. What am I missing here? Am I just not encoding this value correctly, or am I extracting from the wrong source?

To extract the table value for euro/oz value for gold i.e. the text €1,452.47 you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:
Using XPATH and get_attribute():
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver.get('https://www.bullionvault.com/gold-price-chart.do#')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='cookies-warning-buttons']//a[text()='Accept']"))).click()
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//strong[text()='Live Gold Price']"))))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//th[text()='Gold Price per Ounce']//following-sibling::td[3]/span[#data-currency='EUR']"))).get_attribute("innerHTML"))
Console Output:
€1,456.30
Using XPATH and text attribute:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver.get('https://www.bullionvault.com/gold-price-chart.do#')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='cookies-warning-buttons']//a[text()='Accept']"))).click()
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//strong[text()='Live Gold Price']"))))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//th[text()='Gold Price per Ounce']//following-sibling::td[3]/span[#data-currency='EUR']"))).text)
Console Output:
€1,456.30

Wait for the page to load then try to get the innerHTML like the following example
import time
from selenium import webdriver
chrome_browser = webdriver.Chrome(
executable_path=r"chromedriver.exe")
chrome_browser.get("https://www.bullionvault.com/gold-price-chart.do")
time.sleep(2)
select = chrome_browser.find_element_by_xpath(
"//*[#id='bullionPriceTable']/div/table/tbody/tr[3]/td[3]/span"
).get_attribute("innerHTML")
print(select)
€1,450.98

Related

Unable to find element that looks like it's dynamically generated

I'm trying to find the email address text located in the below URL. You can clearly see the email address but I believe the text is generated dynamically through Javascript/React. When I copy the XPATH or CSS Selector and try to find the element like I would any other element I just get an error saying the element cannot be found.
I've tried time.sleep(30) to give the page time to fully load but that's not the issue.
I've tried:
driver.find_element(By.XPATH, '//*[#id="mount_0_0_D8"]/div/div[1]/div/div[5]/div/div/div[3]/div/div/div[1]/div[1]/div/div/div[4]/div[2]/div/div[1]/div[2]/div/div[1]/div/div/div/div/div[2]/div[2]/div/ul/div[2]/div[2]/div/div/span')
You can see from the snippet below that the email is visible but is between some ::before and ::after text I've not seen before.
https://www.facebook.com/homieestudio
Any ideas on how to consistently pull back the email address here? I'm using Chromedriver.
The clue is, the Email Address will always have the # sign.
Solution
To extract the text Info#homieestudio.co.uk ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:
Using XPATH and text attribute:
driver.get('https://www.facebook.com/homieestudio')
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[contains(., '#')]"))).text)
Using XPATH and get_attribute("textContent"):
driver.get('https://www.facebook.com/homieestudio')
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[contains(., '#')]"))).get_attribute("textContent"))
Console output:
Info#homieestudio.co.uk
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python
You can find the email address using the below
wait = WebDriverWait(driver, 20)
element = wait.until(EC.visibility_of_element_located((By.XPATH, "//span[#dir='auto'][contains(text(), 'info#homieestudio.co.uk')]")))
OR
wait = WebDriverWait(driver, 20)
element = wait.until(EC.presence_of_element_located((By.XPATH, "//span[#dir='auto'][contains(text(), 'info#homieestudio.co.uk')]")))
IMPORT
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

How to scrape in Python Selenium?

If you visit this site, https://www.premierleague.com/match/66686 and then press stats tab, you will see several information about the match. How am I supposed to scrape the Possession for both teams?
This did not work.
stats = driver.find_element(By.XPATH, '//*[#id="mainContent"]/div/section[2]/div[2]/div[2]/div[1]/div/div/ul/li[3]')
stats.click()
posHome = driver.find_element(By.XPATH,'//body[1]/main[1]/div[1]/section[2]/div[2]/div[2]/div[2]/section[3]/div[2]/div[2]/table[1]/tbody[1]/tr[1]/td[1]')
print(posHome.text)
posAway = driver.find_element(By.XPATH,'//*[#id="mainContent"]/div/section[2]/div[2]/div[2]/div[2]/section[3]/div[2]/div[2]/table/tbody/tr[1]/td[3]')
print(posAway.text)
Please let me know how to solve this issue and thanks!
To print the Possession for both teams you need to induce WebDriverWait for the visibility_of_element_located() and you can use the following locator strategies:
Code Block:
driver.get("https://www.premierleague.com/match/66686")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Accept All Cookies']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[text()='Stats']"))).click()
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tbody.matchCentreStatsContainer>tr>td>p"))).text)
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tbody.matchCentreStatsContainer>tr>td:nth-child(3)>p"))).text)
driver.quit()
Console Output:
33.9
66.1
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python

How can I adjust this line of selenium code to get the status info of this item?

I have this line of code which I am trying to use to obtain the status of an item. Here is the line of code:
item_status = driver.findElement(By.className("status-info")).getText();
I'm not sure how I can adjust this to retrieve the text seen here:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options=Options()
driver=webdriver.Chrome(options=options)
#Directing to site
driver.get("https://www.amazon.co.uk/Nintendo-Switch-OLED-Model-Neon/dp/B098TNW7NM/ref=sr_1_3?keywords=Nintendo+Switch&qid=1651147043&sr=8-3");
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[2]/span/form/div[3]/span[1]/span/input"))).click()
When you are doing this
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[1]/header/div/div[1]/div[2]/div/form/div[3]/div/span/input"))).click()
it will click on search icon, now on the result page, this xpath //span[#class='a-size-base a-color-success a-text-bold'] is not present hence nothing is getting printed on the console you are likely to face TimedoutException.
However looking at the screenshot that you've shared, I would say to use this xpath
//div[#id='availability']//span[contains(text(),'In stock.')]
If you want to print the text and tag
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[#id='availability']//span[contains(text(),'In stock.')]"))).get_attribute("innerHTML"))
If only text you want:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[#id='availability']//span[contains(text(),'In stock.')]"))).get_attribute("innerText"))
driver.findElement(By.className("status-info")) is the Java syntax and getText() is a Java method. Possibly you need Python syntax and method.
Solution
To print the text In stock. you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span.a-size-base.a-color-success.a-text-bold"))).text)
Using XPATH:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[#class='a-size-base a-color-success a-text-bold']"))).get_attribute("innerHTML"))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

How to find_element(By.XPATH) and sending keys in selenium?

Trying to update my code to use "driver.find_element(By.XPATH..." instead of "driver.find_elements_by_xpath(...", but I keep getting the following the error when I send keys:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
Here is my code:
driver = webdriver.Chrome(PATH)
link_login = "https://www.wyzant.com/tutor/jobs"
driver.get(link_login)
username_input = driver.find_element(By.XPATH, "//*[#id='Username']")[1]
username_input.send_keys("Test")
Use find_elements instead of find_element to select the element like you do in your example:
driver.get('https://www.wyzant.com/tutor/jobs')
username_input = driver.find_elements(By.XPATH, "//*[#id='Username']")[1]
username_input.send_keys("Test")
or change your xpath to select more specific '//form[#class="sso-login-form"]//*[#id="Username"]':
driver.get('https://www.wyzant.com/tutor/jobs')
username_input = driver.find_element(By.XPATH, '//form[#class="sso-login-form"]//*[#id="Username"]')
username_input.send_keys("Test")
Try using more precise XPath locator.
The entire XPath expression should be inside the (By.XPATH, "your_xpath_expression")
You should also use expected conditions explicit waits
This should work:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//form[#action='/sso/login']//input[#id='Username']"))).send_keys(your_user_name)
You will need to import these imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
To send a character sequence within the Username field you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://www.wyzant.com/tutor/jobs")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "form.sso-login-form input#Username"))).send_keys("rushi")
Using XPATH:
driver.get("https://www.wyzant.com/tutor/jobs")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//form[#class='sso-login-form']//input[#id='Username']"))).send_keys("rushi")
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Python selenium I can't acess to a linked page on a page with iframe

I want to go from this page to this https://resultats.ffbb.com/organisation/b5e6211d5970.html to this page https://resultats.ffbb.com/championnat/b5e6211f621a.html?r=200000002810394&d=200000002911791&p=2 by clicking on 'Régional féminin U15'.
I have tried many solutions but the best I had, is not working systematically.
Please coul you help me?
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
import selenium.webdriver.support.ui as ui
import time
driver = webdriver.Firefox()
driver.get("https://resultats.ffbb.com/organisation/b5e6211d5970.html")
driver.switch_to.frame("idIframeChampionnat")
#sign_in = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '/html/body/pre/span[93]'))).click();
button = driver.find_element_by_partial_link_text(u"minin U15")
button.click()```
To click() on the link with text as Régional féminin U15 as the elements are within an iframe so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://resultats.ffbb.com/organisation/b5e6211d5970.html")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#idIframeChampionnat")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Régional féminin U15"))).click()
Using XPATH:
driver.get("https://resultats.ffbb.com/organisation/b5e6211d5970.html")
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='idIframeChampionnat']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "Régional féminin U15"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
Reference
You can find a couple of relevant discussions in:
Switch to an iframe through Selenium and python
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
Try
driver.switch_to.frame("idIframeChampionnat")
button = driver.find_element_by_xpath("//a[contains(text(), 'minin U15')]")
button.click()
or
driver.switch_to.frame("idIframeChampionnat")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[contains(text(), 'minin U15')]"))).click()
This is my code example in groovy (sorry, haven't python env), pretty similar to yours, and it works perfectly 20 times in a loop even without timeouts.
#Test(invocationCount = 20)
void test() {
driver.get('https://resultats.ffbb.com/organisation/b5e6211d5970.html')
driver.switchTo().frame(
driver.findElement(By.xpath("//iframe[#id='idIframeChampionnat']"))
)
driver.findElement(By.xpath("//a[contains(text(), 'minin U15')]")).click()
driver.switchTo().defaultContent()
assert driver.getCurrentUrl() ==
'https://resultats.ffbb.com/championnat/b5e6211f621a.html?r=200000002810394&d=200000002911791&p=2'
}
So, I have no ideas. Maybe just add driver.switch_to.default_content() after click?

Categories