The code I want to extract the "ul" element from
After trying suggestion, I want to get "title" in the follwing "li" tag
I'm trying to extract the following "ul" element from a webpage using selenium. Using Python, I can't figure out what the X_PATH should be, I've tried everything I could think of. Also tried the css_selection. I'm getting back nothing.
I want to iterate over the "li" elements within that specific "ul" element. If anybody could help it would be appreciated, I've literally tried everything I can think/search.
//ul[#aria-label='Chat content']/li[1]/div[1]/div[1]
To me it looks like you could just grab the ul by it's aria label and then select the first li from there. You could also just inspect the element and copy the xpath from the developers console.
It also seems to be in an iframe.
wait = WebDriverWait(driver, 30)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[#class='embedded-electron-webview embedded-page-content']")))
Imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Check is that element are inside some iFrame, maybe is for that, if that dont work install selenium extension in your browser and try to recreat the flow, you will receive all xpath, is a way that I use when nothing is working to me.
Related
I'm using Selenium in a project that consists of opening a range of websites, that contains pretty much the same structure, collecting data in each site and storing it.
The problem I ran into is that some of the sites I wan't to access are unavailable, and when the program get to one of those it just stops.
What I want it to do, is to skip those and follow on with the next iterations, but so far my tries have been obsolete... In my latest try I used the method is_displayed(), but apparently it will only tell me if an element is visible or not, instead of telling me if it's present or not.
if driver.find_element_by_xpath('//*[#id="main-2"]/div[2]/div[1]/div[1]/div/div[1]/strong').is_displayed():
The example above doesn't work, because the driver needs to find the element before telling me if it visible or not, but the element is simply not there.
Have any of you dealt with something similar?
How one of the sites looks like normally
How it looks like when it is unavailable
You can use Selenium expected conditions waiting for element presence.
I'm just giving an example below.
I have defined the timeout for 5 seconds here, but you can use any timeout value.
Also, your element locator looks bad.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element_xpath_locator = '//*[#id="main-2"]/div[2]/div[1]/div[1]/div/div[1]/strong'
wait = WebDriverWait(browser, 5)
wait.until(EC.presence_of_element_located((By.XPATH, element_xpath_locator)))
I'm using selenium and I have a problem handling the accept cookies pop-up.
from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time
web = webdriver.Chrome("C:/chromedriver")
web.get('https://www.spaargids.be/sparen/schuldsaldo-berekenen.html')
time.sleep(2)
akkoord = web.find_element_by_xpath('/html/body/div/div[2]/div[3]/button[1]')
When I inspect the element, I can easily find it. But selenium is unable to do it.
Does anybody know how to fix this?
The problem is that the element is inside an iframe, you need to switch to the respective iframe.
Try this before akkoord variable:
web.switch_to.frame(web.find_element_by_xpath('//iframe[#title="SP Consent Message"]'))
That should make it work, then use the same approach but updating the XPATH to switch to the iframe where the content is.
I have been messing around with python and selenium and was trying to make a program that could automatically checkout on bestbuy.ca . I have gotten to nearly the last stage of the checkout process but am having difficulty getting the element/clicking the buttons to continue. The only button I have been able to get to work is the add to cart button on the product page.
Here is the site as well as the HTML:
Click here for image
Some things I have tried with no success include:
driver.find_element_by_class_name().click()
For this, I used the class in button and span. I have tried using each individual class in separately and together with little success. It would usually give me a "No Element Found" error or just not click anything afterwards.
driver.driver.find_element_by_xpath().click()
For this, I tried using the element locator extension for the xpath as well as "//*[#type='submit']" with no success.
driver.driver.find_element_by_css_selector().click()
For this, I tried the element locator extension again as well as copying it from the firefox inspect with no success.
Not really sure if there are any other ways to go about doing this but any help would be appreciated.
Did you try to copy the entire class attribute from the html and use :
wait.until(EC.presence_of_element_located((By.XPATH, "//button[#class='asd asd sad asd']"))).click()
copy the class and paste it.
Try either of the following to click on the button tag.
wait=WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Continue']/parent::button"))).click()
Or by attribute data automation
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[#data-automation='review-order']"))).click()
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I'm trying to make a web scraper for a webpage using selenium.
the webpage is https://www.supersport.hr/sport/dan/0/sport/1
I am able to get most of the web elements I'm trying to get but my script doesn't return all of them even though they are the same class.
page example:
In this case my script returns all of the above divs, but cuts off at "SRL GRČKA 1." and doesn't get any leagues below.
the HTML markup for each of these leagues are the same:
I'm getting a list of these elements in python like this:
football_leagues_elements = driver.find_elements_by_css_selector("div.sportska-liga.nogomet")
I've also tried with this code but it returns the same result:
football_leagues_elements = driver.find_elements_by_xpath("//div[contains(#class, 'sportska-liga-wrap')]//div[contains(#class, 'nogomet')]")
I think all the leagues are loaded to the page at the same time.
My question is why are some divs not included in the webelement list?
Any help is welcome.
The Website is NOT accessible here.However try Induce Explicit wait and wait for all element visible.
football_leagues_elements=WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.sportska-liga.nogomet")))
You need to import following libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
If above doesn't work then try to scroll the page first and then check for elements.
#Scroll bottom of the page.
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")
#Get the elements.
football_leagues_elements=WebDriverWait(driver,20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"div.sportska-liga.nogomet")))
First of all, I'm completely new in web scraping, html and selenium, so my question might not seem meaningful to you.
What I'm trying to do: automated click on an image on a webpage. Manual click on this image allows to display new information on a part of the webpage.
How I tried to do it: I found something specific to this image, for example:
<img src="images/btn_qmj.gif" border="0">
So I just entered in python:
xpath = '//img[#src="images/btn_qmj.gif"]'
driver.find_elements_by_xpath(xpath)
Problem: this returns me an empty list. I partially understood why, but I don't have any solution.
image of the html code inspector
I included here an image of the html tree I obtain on the web inspector. As one can see, the tree to reach my line of interest gives "/html/frameset/frame1/#document/html/body/center/table/tbody/tr[2]/td/a/img".
The issue is that I cannot access -by using an xpath- anything that comes after the #document. If I try to copy the xpath of my line of interest, it tracks it back only up to the 2nd "/html". It is like the page is subdivided, compartmentalized, with inner parts I cannot access. I suppose there is a way to access the #document content, maybe it refers to another webpage, but I'm stuck at that point.
I would be very grateful if anyone could help me on this.
Have you tried switching to the frame first?
driver.switch_to.frame('gauche')
You need to switch control to frame first to handle image on it. Please see below code for your reference
driver.switch_to_frame(frame)
WebDriverWait(driver, 20).until(
EC.visibility_of_element_located((By.XPATH, "'//img[#src="images/btn_qmj.gif"]'"))).click()
driver.switch_to_default_content()
Note :: You need to add below imports
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
The element is present inside an iframe.In order to access the element you need to switch it first.
Induce WebDriverWait() and frame_to_be_available_and_switch_to_it()
Induce WebDriverWait() and visibility_of_element_located()
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"gauche")))
WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH,"//img[#src='images/btn_qmj.gif']"))).click()
You need to import following libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By