Scraping dynamic data with Selenium & Python 'Unable to locate element' - python

I'm trying to use Selenium & Python to scrape a website (http://epl.squawka.com/english-premier-league/06-03-2017/west-ham-vs-chelsea/matches). I am using the webdriver to click a heading and then wait for the new information to load before clicking on an object before scraping the resulting data (which loads from the clicking). My problem is that I keep on getting an 'Unable to locate element error.
I've taken a screenshot at this point and can physically see the element and I've also printed the entire source code and can see that the element is there.
driver.find_element_by_id("mc-stat-shot").click()
time.sleep(3)
driver.save_screenshot('test.png')
try:
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID,"svg")))
finally:
driver.find_element_by_xpath("//g[3]/circle").click()
time.sleep(1)
goalSource = driver.page_source
goalBsObj = BeautifulSoup(goalSource, "html.parser")
#print(goalBsObj)
print(goalBsObj.find(id="tt-mins").get_text())
print(goalBsObj.find(id="tt-event").get_text())
print(goalBsObj.find(id="tt-playerA").get_text())
and the result is an error:
"selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: //g[3]/circle"

Related

How to click on <a> tag using selenium in python?

I am new to web scraping and I am trying to scrape reviews off amazon.
After going on a particular product's page on amazon I want to click on the 'see all reviews' button. I did inspect element on the page, I found that the see all reviews button has this structure
structure
So I tried to find this element using the class name a-link-emphasis a-text-bold.
This is the code I wrote
service = webdriver.chrome.service.Service('C:\\coding\\chromedriver.exe')
service.start()
options = webdriver.ChromeOptions()
#options.add_argument('--headless')
options = options.to_capabilities()
driver = webdriver.Remote(service.service_url, options)
driver.get(url)
sleep(5)
driver.find_element_by_class_name('a-link-emphasis a-text-bold').click()
sleep(5)
driver.implicitly_wait(10)
But this returns me the following error
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".a-link-emphasis a-text-bold"}
What am I doing wrong here?
driver.find_element_by_class_name('a-link-emphasis.a-text-bold').click()
By class expects single class not multiple but you can use the above syntax , remove space with . as it uses css under the hood, or use :
driver.find_element_by_css_selector('.a-link-emphasis.a-text-bold').click()
driver.find_element_by_css_selector('[class="a-link-emphasis a-text-bold"]').click()

Selenium can't locate xpath of facebook pages

I am trying to download public images of some facebook pages using xpath. I got the xpath from google chrome dev mode (right click and copy xpath).
The xpath I got is: /html/body/div[1]/div/div[1]/div[1]/div[3]/div/div/div[1]/div[1]/div[4]/div/div/div[3]/div/div/div/div[2]
When I try to find it in gogole chrome, it finds the xapth just fine as shown in the image.
But Selenium throws an exception.
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element
The code snippet I am using is as follows-
driver.get(page)
sleep(10)
allimgdiv = driver.find_element_by_xpath(
'/html/body/div[1]/div/div[1]/div[1]/div[3]/div/div/div[1]/div[1]/div[4]/div/div/div[3]/div/div/div/div[2]')
This expression returns the link of only posted images in an facebook page type: https://www.facebook.com/username/photos
//h2[.//a[contains(#href,"/photos")]]//following::div//img
Try
images = driver.find_elements_by_xpath('//h2[.//a[contains(#href,"/photos")]]//following::div//img')
for image in images:
linkimage = image.find_element_by_xpath('./#src').text

location of search input term element identified for selenium in python

I'm using Python and Selenium in PyCharm to go to the SEC website to download a 10-K CSV file. Ideally, the program should ask for user input for a "ticker symbol", then go to the SEC's website, input the ticker symbol provided and download the 10-K and 10-Q CSV files from the page. I was using Microsoft's ticker symbol (MSFT) as an example test. The SEC's Edgar search website is this:
https://www.sec.gov/edgar/searchedgar/companysearch.html
and I am using the 'Fast Search' search engine. I created a function 'get_edgar_results' to perform this download. It might be that I'm new to web scraping, but I thought I identified the HTML tags correctly on where to put my search term. Previous problems suggested that I might need to have the program wait before searching for the HTML element, so I added code for the program to wait. I continue getting this error:
line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [id="Find"]
My code is below:
import selenium.webdriver.support.ui as ui
from pathlib import Path
import selenium.webdriver as webdriver
ticker_symbol = input("please provide a ticker symbol: ")
def get_edgar_results(ticker_symbol):
url = "https://www.sec.gov/edgar/searchedgar/companysearch.html"
driver = webdriver.Firefox(executable_path=r"C:\Program Files\JetBrains\geckodriver.exe")
wait = ui.WebDriverWait(driver,30)
driver.set_page_load_timeout(30)
driver.get(url)
search_box = driver.find_element_by_id("Find")
search_box.send_keys(ticker_symbol)
search_box.submit()
annual_links = driver.find_elements_by_class_name("10-K")
quarterly_links = driver.find_elements_by_class_name("10-Q")
results = []
driver.close()
driver.quit()
return results
get_edgar_results(ticker_symbol)
Any help would be greatly appreciated.
Consider using a waitUntil or Implicit/Explicit waits method to wait until an element is loaded. This way you can circumvent the error shown above, code below for wait until method.
browser = webdriver.Firefox()
browser.get("url")
delay = 3 # seconds
try:
myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
print "Page is ready!"
except TimeoutException:
print "Loading took too much time!"
You also seem to have an error in the following code
search_box = driver.find_element_by_id("Find")
search_box.send_keys(ticker_symbol)
search_box.submit()
The id=find locates the Search Box and not the input element and therefore sending keys value to a button is incorrect. I would recommend you to use the xpath to uniquely locate the element of your choice.
The following will send a value to the input box and will make a button click on the SEARCH button.
driver.findElement(By.xpath("//*[#id="lesscompany"])).sendKeys("your value");
driver.findElement(By.xpath("//*[#id="search_button_1"]")).click();

NoSuchElementException on all elements on page with Python Selenium

Set-up
I'm trying to log in to a website using Python + Selenium.
My code to load the website is,
browser = webdriver.Firefox(
executable_path='/mypath/to/geckodriver')
url = 'https://secure6.e-boekhouden.nl/bh/'
browser.get(url)
Problem
Selenium cannot locate the element containing the account and password fields.
For example, for the field 'Gebruikersnaam',
browser.find_element_by_id('txtEmail')
browser.find_element_by_xpath('//*[#name="txtEmail"]')
browser.find_element_by_class_name('INPUTBOX')
all give NoSuchElementException: Unable to locate element.
Even worse, Selenium cannot find the body element on the page,
browser.find_element_by_xpath('/html/body')
gives NoSuchElementException: Unable to locate element: /html/body.
I'm guessing something on the page is either blocking Selenium (maybe the 'secure6' in the url) or is written in a language/form Selenium cannot handle.
Any suggestions?
All elements are inside the frame. So that, it is throwing No Such Element exception. Please try to switch to the frame before all actions as given below.
browser = webdriver.Firefox(
executable_path='/mypath/to/geckodriver')
url = 'https://secure6.e-boekhouden.nl/bh/'
browser.get(url)
browser.switch_to.frame(browser.find_element_by_id("mainframe"))
browser.find_element_by_id('txtEmail')
browser.find_element_by_xpath('//*[#name="txtEmail"]')
browser.find_element_by_class_name('INPUTBOX')

Python Selenium StaleElementReferenceException Finding Element by xpath

So I have been using selenium to open a webpage and wait for a specific element to be loaded. Once that's loaded, I'm finding an element within the first element that's already loaded and getting a value from that element. But, every time I run the code I get a StaleElementReferenceException on the line that says price = float(...). This is weird to me because it didn't crash on the line before which has the same xpath search. I'm very new to this, any help would be appreciated!
browser.get(url + page)
element = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.ID, "searchResultsTable")))
price_element = element.find_element_by_xpath("//div[#class='market_listing_row market_recent_listing_row market_listing_searchresult']")
price = float(price_element.find_element_by_xpath("//span[#style='color:white']").text[:1])

Categories