I am trying to locate the "Confirm Booking" button in selenium. My driver is on the current webpage and we can see in the html that the button is of class "menubutton". Yet when I iterate through all elements of that class it only locates the "Return to timetable" button. As you can see in the terminal output.
My code is as follows:
#confirms booking
elems = driver.find_elements(By.CLASS_NAME, 'menubutton')
print(len(elems))
for e in elems:
print(e.text)
I have also tried locating by XPATH but it only finds the "Return to Class Timetable" button.
Any help would be appreciated!
Related
Trying to scrape a website, I created a loop and was able to locate all the elements. My problem is, that the next button id changes on every page. So I can not use the id as a locator.
This is the next button on page 1:
<a rel="nofollow" id="f_c7" href="#" class="nextLink jasty-link"></a>
And this is the next button on page 2:
<a rel="nofollow" id="f_c9" href="#" class="nextLink jasty-link"></a>
Idea:
next_button = browser.find_elements_by_class_name("nextLink jasty-link")
next_button.click
I get this error message:
Message: no such element: Unable to locate element
The problem here might be that there are two next buttons on the page.
So I tried to create a list but the list is empty.
next_buttons = browser.find_elements_by_class_name("nextLink jasty-link")
print(next_buttons)
Any idea on how to solve my problem? Would really appreciate it.
This is the website:
https://fazarchiv.faz.net/faz-portal/faz-archiv?q=Kryptow%C3%A4hrungen&source=&max=10&sort=&offset=0&_ts=1657629187558#hitlist
There are two issues in my opinion:
Depending from where you try to access the site there is a cookie banner that will get the click, so you may have to accept it first:
browser.find_element_by_class_name('cb-enable').click()
To locate a single element, one of the both next buttons, it doeas not matter, use browser.find_element() instead of browser.find_elements().
Selecting your element by multiple class names use xpath:
next_button = browser.find_element(By.XPATH, '//a[contains(#class, "nextLink jasty-link")]')
or css selectors:
next_button = browser.find_element(By.CSS_SELECTOR, '.nextLink.jasty-link')
Note: To avoid DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() import in addition from selenium.webdriver.common.by import By
You can't get elements by multiple class names. So, you can use find_elements_by_css_selector instead.
next_buttons = browser.find_elements_by_css_selector(".nextLink.jasty-link")
print(next_buttons)
You can then loop through the list and click the buttons:
next_buttons = browser.find_elements_by_css_selector(".nextLink.jasty-link")
for button in next_buttons:
button.click()
Try below xPath
//a[contains(#class, 'step jasty-link')]/following-sibling::a
On this website, I'm trying to find an element based on its XPATH, but the XPATH keeps changing. What's the next best alternative?
Snippet from website
<button class="KnkXXg vHcWfw T1alpA kiPMng AvEAGQ vM2UTA DM1_6g _-kwXsw Mqe1NA SDIrVw edrpZg" type="button" aria-expanded="true"><span class="nW7nAQ"><div class="VpIG5Q"></div></span></button>
XPATH:
//*[#id="__id15"]/div/div/div[1]/div[2]/div
#Sometimes id is a number between 15-18
//*[#id="__id23"]/div/div/div[1]/div[2]/div
#Sometimes id is a number between 13-23
Here's how I use the code:
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, """//*[#id="__id3"]/div/div/div[1]/div[2]/div/div/div/button"""))).click()
I've tried clicking the element by finding the button class, but for whatever reason it won't do anything.
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "KnkXXg vHcWfw T1alpA kiPMng AvEAGQ vM2UTA DM1_6g _-kwXsw Mqe1NA SDIrVw edrpZg"))).click()
If Part of the text is keep changing you can use contains in the xpath.
//*[contains(#id,"__id"]/div/div/div[1]/div[2]/div
I have been searching across the site in the hope of finding an answer, however, every question I view doesn't have heavily nested HTML code like the page I am trying to scrape.
I am really hoping someone will spot my obvious error. I have the following code which is pulling the category headers and but annoyingly not the href that goes with each one. When run, the code currently returns 'None' for all the href's but I cannot decipher why. I think it may be because I am targeting the wrong element, tag or class in the HTML but cannot correctly identify which one it should be.
from selenium import webdriver
import time
# The website to scrape
url = "https://www.jtinsight.com/JTIRA/JTIRA.aspx#!/full-category-list"
# Creating the WebDriver object using the ChromeDriver
driver = webdriver.Chrome()
# Directing the driver to the defined url
driver.get(url)
# driver.implicitly_wait(5)
time.sleep(1)
# Locate the categories
categories = driver.find_elements_by_xpath('//div[#class="subCatEntry ng-scope"]')
# Print out all categories on current page
num_page_items = len(categories)
print(num_page_items)
for headers in range(num_page_items):
print(categories[headers].text)
for elem in categories:
print(elem.get_attribute("a.divLink[href='*']"))
# Clean up (close browser once task is completed)
time.sleep(1)
driver.close()
I would really appreciate if anyone can point out my error.
Try this below code.
for elem in categories:
print(elem.find_element_by_css_selector("a.divLink").get_attribute('href'))
You are passing the CSS selector for the get_attribute method. That wouldn't work. You have to provide the attribute name only. If the web element elem has an attribute named href then it would print the value of that attribute.
First, get the anchor <a> element. All the subcategory anchors have class divLink. For getting anchor elements try this,
categories = driver.find_elements_by_class_name('divLink')
Second, Print the attribute value by passing the attribute name in the get_ttribute. Try this,
print(elem.get_attribute("href"))
This way you'll be able to print all the href values.
I am trying to scrape the list of followings for a given instagram user. This requires using Selenium to navigate to the user's Instagram page and then clicking "following". However, I cannot seem to click the "following" button with Selenium.
driver = webdriver.Chrome()
url = 'https://www.instagram.com/beforeeesunrise/'
driver.get(url)
driver.find_element_by_xpath('//*[#id="react-root"]/section/main/article/header/div[2]/ul/li[3]/a').click()
However, this results in a NoSuchElementException. I copied the xpath from the html, tried using the class name, partial link and full link and cannot seem to get this to work! I've also made sure that the above xpath include the element with a "click" event listener.
UPDATE: By logging in I was able to get the above information. However (!), now I cannot get the resulting list of "followings". When I click on the button with the driver, the html does not include the information in the pop up dialog that you see on Instagram. My goal is to get all of the users that the given username is following.
Make sure you are using the correct X Path.
Use the following link to get perfect X Paths to access web elements and then try.
Selenium Command
Hope this helps to solve the problem!
Try a different XPath. I've verified this is unique on the page.
driver.find_element_by_xpath("//a[contains(.,'following')]")
It's not the main goal of selenium to provide rich functionalities, from a web-scraping perspective, to find elements on the page, so the better option is to delegate this task to a specific tool, like BeautifulSoup. After we find what we're looking for, then, we can ask for selenium to interact with the element.
The bridge between selenium and BeautifulSoup will be this amazing function below that I found here. The function gets a single BeautifulSoup element and generates a unique XPATH that we can use on selenium.
import os
import re
from selenium import webdriver
from bs4 import BeautifulSoup as bs
import itertools
def xpath_soup(element):
"""
Generate xpath of soup element
:param element: bs4 text or node
:return: xpath as string
"""
components = []
child = element if element.name else element.parent
for parent in child.parents:
"""
#type parent: bs4.element.Tag
"""
previous = itertools.islice(parent.children, 0, parent.contents.index(child))
xpath_tag = child.name
xpath_index = sum(1 for i in previous if i.name == xpath_tag) + 1
components.append(xpath_tag if xpath_index == 1 else '%s[%d]' % (xpath_tag, xpath_index))
child = parent
components.reverse()
return '/%s' % '/'.join(components)
driver = webdriver.Chrome(executable_path=YOUR_CHROMEDRIVER_PATH)
driver.get(url = 'https://www.instagram.com/beforeeesunrise/')
source = driver.page_source
soup = bs(source, 'html.parser')
button = soup.find('button', text=re.compile(r'Follow'))
xpath_for_the_button = xpath_soup(button)
elm = driver.find_element_by_xpath(xpath_for_the_button)
elm.click()
...and works!
( but you need writing some code to log in with an account)
I get "element is no longer attached to the dom" exception even though the element is there and is clickable, I am trying to click the "next" arrow on ryanair website, the html for the next button is:
<li class="newer" ng-class="{'loadingsmall':loading}">
<a ng-disabled="loading" ng-click="loadMore(0, 'SelectInput$LinkButtonNext1', 1)"
title="Next Week" href="">
</a>
</li>
I located and clicked it in several methods:
elem = WebDriverWait(browser, 15).until(EC.element_to_be_clickable((By.XPATH,
"//a[#title='Next Week']")))
elem = browser.find_element_by_xpath("//a[#title='Next Week']")
elem.click()
and:
area = browser.find_element_by_xpath("//a[#title='Next Week']")
action = webdriver.ActionChains(browser)
action.move_to_element(area) action.click(area) action.perform()
and:
elem = browser.find_element_by_link_text('>')
elem.click()
all work fine if I have no action in between, but once I tell selenium to click on other elements on the page (I do not move to other pages, I stay on the same page and show some dynamic content) the "next" link only works the first time around, and then gives me the exceptions, help would be so greatly appreciated! :)
You're still on the same page but the DOM's elements have changed due to some AJAX actions & calls, so you're forced to re-detect your object using:
elem = browser.find_element_by_link_text('>')
If you don't do that, it is more than likely that you'll get a
StaleReferenceException
For more info