I'm trying to use selenium & python to auto get part of the html content.
I use find xpath like below ways to sort out the price from specified flight number, but always get failed result " unable to locate "
Anyone could shed some lights on it ?
element_price = driver.find_element_by_xpath("//div[#id='flight_MU5401']")
element_price.find_element_by_xpath(".//span[#class='base_price02']")
It's the html
If I were to guess, I would say that you are probably getting that error because the element isn't loaded when your code runs. It's probably a slower loading page. You should try adding a wait and see if that helps.
You can also simplify your locator by using a CSS selector and only scraping the page once.
price = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#flight_MU5401 span.base_price02")).text
Related
My goal is to be able to scrape definitions of words in python.
To begin with, I am trying to get just the first definition of the word "assist" which should be "to help". I am using dictionary.cambridge.org
//web driver goes to page
driver.get("https://dictionary.cambridge.org/dictionary/english/assist")
//to give time for the page to load
time.sleep(4)
//click "accept cookies"
driver.find_element_by_xpath("/html[#class='i-amphtml-singledoc i-amphtml-standalone']/body[#class='break default_layout amp-mode-mouse']/div[#id='onetrust-consent-sdk']/div[#id='onetrust-banner-sdk']/div[#class='ot-sdk-container']/div[#class='ot-sdk-row']/div[#id='onetrust-button-group-parent']/div[#id='onetrust-button-group']/div[#class='banner-actions-container']/button[#id='onetrust-accept-btn-handler']").click()
Up this point, everything is working correctly. However, when I try to print the first definition using "find element by xpath", I get a NoSuchElementException. I'm pretty familiar with selenium and have scraped web stuff hundreds of times before but on this webpage, I don't know what I'm doing wrong. Here's the code I am using:
print(driver.find_element_by_xpath("/html[#class='i-amphtml-singledoc i-amphtml-standalone']/body[#class='break default_layout amp-mode-mouse']/div[#class='cc fon']/div[#class='pr cc_pgwn']/div[#class='x lpl-10 lpr-10 lpt-10 lpb-25 lmax lp-m_l-20 lp-m_r-20']/div[#class='hfr-m ltab lp-m_l-15']/article[#id='page-content']/div[#class='page']/div[#class='pr dictionary'][1]/div[#class='link']/div[#class='pr di superentry']/div[#class='di-body']/div[#class='entry']/div[#class='entry-body']/div[#class='pr entry-body__el'][1]/div[#class='pos-body']/div[#class='pr dsense dsense-noh']/div[#class='sense-body dsense_b']/div[#class='def-block ddef_block ']/div[#class='ddef_h']/div[#class='def ddef_d db']").text())
To print the scrape definitions of words you can use either of the following Locator Strategies:
Using xpath and text attribute:
print(driver.find_element_by_xpath("//span[contains(#class, 'epp-xref dxref')]//following::div[1]").text)
Using xpath and innerText:
print(driver.find_element_by_xpath("//span[contains(#class, 'epp-xref dxref')]//following::div[1]").get_attribute("innerText"))
Console Output:
to help:
Instead of Absolute xpath, opt for Relative xpaths. You can refer this link
Tried with below code and it retrieved the data.
driver.get("https://dictionary.cambridge.org/dictionary/english/assist")
print(driver.find_element_by_xpath("(//div[#class='ddef_h'])[1]/div").get_attribute("innerText"))
to help:
Currently I'm attemping to switch from my default content, to the only iframe in the website. I don't know if it's how the site is coded, but I can't access via DOM any element.
This is the HTML structure of the site:
XPATH of site is //*[#id="iframeBody"] (When I paste this in the element inspector, I get the correct iframe). So, if I try to switch using frame_to_be_available_and_switch_to_it, this is the output:
try:
WebDriverWait(self.driver, 10).until(ec.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='iframeBody' and #name='body']")))
except Exception as e:
print(e)
>>> Message: javascript error: Failed to execute 'getComputedStyle' on 'Window': parameter 1 is not of type 'Element'.
>>>(Session info: chrome=79.0.3945.88)
I also tried creating an iframe element variable, finding it via ID and XPATH and then I've used switch_to(element). Getting the same result. When I print this variable, the element is actually found:
# Also tried finding with id
element = self.driver.find_element_by_xpath('//iframe[#id='iframeBody')
print(element)
<selenium.webdriver.remote.webelement.WebElement (session="9184691b1fdcccc15dd36bbcb914ac8b", element="1ef77729-8a6e-4d3c-98bd-c95878437585")>
But when I try to switch to this iframe, I get the same result as above.
For some reason, this site is not letting me use the DOM data, actually, when I try to click a button I need to use action chains, because I get the same error.
Anybody can help me?
Did you put enough time before switching ?
also, you could directly switch as well:
driver.switch_to.default_content()
driver.switch_to.frame(driver.find_element_by_id("iframeBody"))
I'm trying to scrape every item on a site that's displayed in a grid format with infinite scrolling. However, I'm stuck on even getting the second item using xpath because it's saying:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#class='el-card advertisement card is-always-shadow'][2]"}
x = 1
while x < 5:
time.sleep(5)
target = driver.find_element_by_xpath(f"//div[#class='el-card advertisement card is-always-shadow'][{x}]")
target.click()
wait.until(EC.visibility_of_element_located((By.ID, "details")))
print(driver.current_url)
time.sleep(5)
driver.back()
time.sleep(5)
WebDriverWait(driver, 3).until(EC.title_contains("Just Sold"))
time.sleep(5)
x += 1
With my f-string xpath it's able to find the first div with that class and print the URL, but the moment it completes one iteration of the while loop, it fails to find the 2nd div with that class (so 2).
I've tried monitoring it with all the time.sleep() to see exactly where it was failing because I thought maybe it was running before the page loaded and therefore it couldn't be found, but I gave it ample time to finish loading every page and yet I can't find the issue.
This is the structure of the HTML on that page:
There is a class of that name (as well as "el-card__body" which I have also tried using) within each div, one for each item being displayed.
(This is what each div looks like)
Thank you for the help in advance!
(disclaimer: this is for a research paper, I do not plan on selling/benefiting off of the information being collected)
Storing each item in a list using find_elements_by_xpath, then iterating through them did the trick, as suggested by tdelaney.
I'm using Selenium to fill out this HTML form, but when it comes to inputting the data it says 'element not interactable'. I am able to click on the element however actually sending a string produces an error. How can I fix this?
driver.get('https://www.masmovil.es/cobertura-fibra-optica-campos/')
prov = Select(driver.find_element_by_xpath('//*[#id="province"]'))
prov.select_by_index(32)
driver.find_element_by_xpath('//*[#id="town"]').send_keys('1')
Thank you!
In the page you are accessing there are 2 elements that are returned with the selector by_xpath('//*[#id="town"]'), one is a "mm-ui-autocomplete", the other one is an "input".
the "mm-ui-autocomplete" is not visible nor interactable to a real user, that's probably what's throwing the exception you're having, and selenium always takes the first match when there's more than one element returned by the selector, so, assuming you want to type something on the "Localidad" field, it is selecting the wrong element.
Try changing your selector to by_xpath('//input[#id="town"]') and see if it works.
Hope it helps.
Can you try with this css selector :
input[id='town']
code :
driver.find_element_by_css_selector("input[id='town']").send_keys('1')
The xpath (//*[#id="town"]) you have used has two entries :
one with mm-ui-autocomplete tag and one with input tag.
Always give preference to css selector over xpath. It's more stable then xpath.
In case you would not want to use css selector, then you can use xpath like this :
//input[#id='town']
Code :
driver.find_element_by_xpath("//input[#id='town']").send_keys('1')
In my case, it happens that the find_element was not working before the frontend finished loading.
I solved this by adding sleep(2) before the find_element_by_xpath. You will need to import the function by from time import sleep.
I have the following code when I inspect on Chrome.
<span id="button-1111-btnInnerEl" class="x-btn-inner x-btn-inner-center" unselectable="on" style="">New Email</span>
I need to click on the label "New Email", but how should invoke it in Selenium (I'm using Python).
def CreateMail():
EmailButton="//*[contains(text(),'New Email')]"
driver.find_elements_by_xpath(EmailButton) // there is no method to enable click.
You can use execute_script
driver.execute_script("document.getElementById('button-1111-btnInnerEl').click()")
driver.find_element_by_id("button-1111-btnInnerEl").click()
Thanks all for your help. Finally i found the answer to my question.I had to add a wait statement, before finding the key. key wasn't present when the page loads, so had to wait a little bit to find the correct key.
def CreateMail():
try:
element = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.ID, "button-1143-btnInnerEl")))
driver.find_element_by_id("button-1143-btnInnerEl").click()
except TimeoutException:
print ("Loading took too much time!")
Hope This XPath will work for you.If you want to validate xpath using your chrome browser just paste this text on your chrome console $x("//*[text()='New Email']") and check how many elements found using this XPath
driver.find_elements_by_xpath("//span[text()='New Email']")
Your statement : "driver.find_elements_by_xpath(EmailButton)" . Click does not work on group of elements. It is actionable only on single element. So you use a singular finder.
driver.find_**element**_by_id(EmailButton).click()
As per the HTML you have shared, the id attribute looks dynamic to me. So we must construct a dynamic xpath or css. Additionally instead of find_elements we have to use find_element so a single WebElement is returned and we can invoke the click() method. Finally, if you look at the node properly, the unselectable attribute is on so we will take help of JavascriptExecutor as follows :
myElement = driver.find_element_by_xpath("//span[starts-with(#id, 'button-')][#class='x-btn-inner x-btn-inner-center']")
driver.execute_script("arguments[0].click();", myElement);
Actually there is different way for locating elements
but in your case there is a ID,
so you can prefer id if its exists
find solution below :
driver.find_element_by_id("button-1111-btnInnerEl").click()