We Keep Coding

selenium scraper get empty list with Xpath

I am scraping price data on the website: https://fbx.freightos.com/ .
This is the code below:
from selenium import webdriver from selenium.webdriver.edge.service
import Service from selenium.webdriver.common.by import By
s = Service(e_driver_path)
driver = webdriver.Edge(service=s)
elements = driver.find_elements(By.XPATH,
/html/body/div[2]/div[1]/div[3]/div/div[1]/section/div[2]/div/div[2]/div/div[1]/div/div/div[1]/span)
content = "".join([element.text for element in elements])
print(content)
the problem is the result. It is an empty list.
As planned, it should be the "Current FBX" and the result looks like "$3,540".
Please help.
Thanks ahead.

Try to wait for required data
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
fbx = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//div[.='Current FBX']/following-sibling::span[normalize-space()]"))).text

Use webdriverwait() and wait for visibility of element located and following css selector
driver.get("https://fbx.freightos.com/")
print(WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.CSS_SELECTOR,".row.bottom-xs.between-xs div:nth-of-type(1)>div"))).text)
print(WebDriverWait(driver,20).until(EC.visibility_of_element_located((By.CSS_SELECTOR,".row.bottom-xs.between-xs div:nth-of-type(1)>span"))).text)
You need to import below libraries.
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By

This xpath will directly target the price web element.
Try fetching this after some wait as mentioned by #KunduK
//div[contains(text(),'Current FBX')]/parent::div/span

//span[#class='styled__Value-sc-1puuh0x-8 hIASbB']
You can also use xpath like this. simplicity is best

Web Scraping Identify and Extract Hyperlink

HI I have the following script that extracts the name and address of each site but I want to be able to also extract the href for each site so that I link to the individual sites. Any suggestions?
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://order.marstons.co.uk/")
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '//*[#id="app"]/div/div/div/div[2]/div'))
).find_elements_by_tag_name('a')
for el in element:
print("heading", el.find_element_by_tag_name('h3').text)
print("address", el.find_element_by_tag_name('p').text)
finally:
driver.quit()

You mean like this?
print(el.get_attribute("href"))
You can get attribute of a element from this.

Nontype Error when extracting text after using select_by_index() in Selenium Python

Code trials:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
browser = webdriver.Chrome()
url = "https://ssangmun.sen.es.kr/66769/subMenu.do"
browser.get(url)
browser.maximize_window()
select = Select(browser.find_element_by_id("srhMlsvYear"))
select.options[0].text #output "2017"
select.select_by_index(0).text #output Nonetype Error
When I use:
select.options[0].text
It works! But when I use:
select.select_by_dex(0)
it occur Nonetype Error
Why this error occur?

with id
#srhMlsvYear
there are two matches in HTMLDOM.
I would recommend you to use the below CSS with explicit wait:
select#srhMlsvYear
and use it like this:
wait = WebDriverWait(browser, 30)
ele = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "select#srhMlsvYear")))
select = Select(ele)
select.select_by_index(0)
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Presumably, the following option texts:
2017년
2018년
2019년 etc
are present within the DOM Tree as soon as the url is invoked. That's why when you:
select.options[0].text
You see the output:
2017
However, select_by_index(index) doesn't returns anything. So while you try to extract the text after selecting the desired option as:
select.select_by_index(0).text
the output is Nonetype Error
Solution
After selecting the desired option to get the option text you can use the attribute first_selected_option as follows:
Code Block:
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver.get("https://ssangmun.sen.es.kr/66769/subMenu.do")
select = Select(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "select#srhMlsvYear"))))
select.select_by_index(0)
print(select.first_selected_option.text)
Console Output:
2017년

element not visible exception python

I try to write a short script to search a key word on newspaper's website and an
'ElementNotVisibleException: Message: element not visible' is raised.
I'm not able to fixe it...
Thank you for help
code:
import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://www.tsa-algerie.com")
wait = WebDriverWait(driver, 8)
elem = driver.find_element_by_name("s")
wait.until(EC.visibility_of_element_located((By.name,"s")))
elem.send_keys("Algieria")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()
os.system('pause')

You just made a variable called wait:
wait = WebDriverWait(driver, 8)
But you didn't use it in your code. Try this:
visibility_of_element_located
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
WebDriverWait(driver, 8).until(EC.visibility_of_element_located((By.NAME, "s")))
elem = driver.find_element_by_name("s")
That way script will wait until 's' element appear, then u gonna find this element. Remember about the order. First u wait, next you can use it.

This code will help you:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://www.tsa-algerie.com")
driver.maximize_window()
#To click on Search Icon to open the search box
ele=driver.find_element_by_xpath('//div[#class="template-header__search search__open transition"]/img')
ele.click()
#wait till search text box appear and then enter the desired keyword you want to search
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.NAME, "s")))
elem = driver.find_element_by_name("s")
elem.send_keys("Algieria")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
driver.close()

python selenium click on button xpath error

I am trying to scrape an airbnb listing. I cant figure out a way to get the full list of amenities other than clicking on "more". I am using selenium to simulate the click, but it does nt seem to work.
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
url = 'https://www.airbnb.com/rooms/4660676'
driver = webdriver.Firefox()
driver.get(url)
elem = driver.find_element_by_xpath('//a[#class="expandable-trigger-more"]')
actions.click(elem).perform()

The XPath itself is correct, but you have not defined the actions:
from selenium.webdriver.common.action_chains import ActionChains
elem = driver.find_element_by_xpath('//a[#class="expandable-trigger-more"]')
actions = ActionChains(driver)
actions.click(elem).perform()
Note that you can simply use the WebElement's click() method instead:
elem = driver.find_element_by_xpath('//a[#class="expandable-trigger-more"]')
elem.click()
Works for me.
If you are getting NoSuchElementException error, you might need to wait for the link to be clickable via WebDriverWait and element_to_be_clickable.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, '//a[#class="expandable-trigger-more"]'))
)
element.click()

A very simple way of achieving this is given below
driver.find_element_by_xpath('//a[#class="expandable-trigger-more"]').click()
It works for me hope will work for you as well.