We Keep Coding

How to locate the pagination select button using Python Selenium

Code trials:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import pandas as pd
driver = webdriver.Firefox()
url = r"https://www.nba.com/stats/players/advanced"
driver.get(url)
select = Select(driver.find_element(By.XPATH, r"/html/body/div[2]/div[2]/div[2]/div[3]/section[2]/div/div[2]/div[2]/div[1]/div[3]/div/label/div/select"))
select.select_by_index(0)
No matter everything I try I cannot find this full Xpath. I just want the code to recognise the little button that goes from page 1 to all to view all player stat on single page.
I've looked into similar questions but unable to get it solved.
Snapshot:

When it seems that the path is not working, the better way to start solving the problem is to gradually remove tags from left to right while in the inspector tool. By removing /html/body/div[2] from your xpath I was able to find the element in the HTML
xpath = "//div[2]/div[2]/div[3]/section[2]/div/div[2]/div[2]/div[1]/div[3]/div/label/div/select"
select = Select(driver.find_element(By.XPATH, xpath))
which if I understood correctly is this one

To recognise the little button that goes from page 1 to all to view all player stat on single page and select the an option within the website you can use the following locator strategies:
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver.get('https://www.nba.com/stats/players/advanced')
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
Select(driver.find_element(By.XPATH, "//div[starts-with(#class, 'Pagination')]//div[contains(., 'Page')]//following::div[1]//select")).select_by_index(0)

Use Python Selenium to extract span text

Hi i'm new at selenium and webscraping and i need some help.
i try to scrape one site and i need and i dont know how to get span class.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
PATCH = "/Users/bobo/Downloads/chromedriver"
driver = webdriver.Chrome(PATCH)
driver.get("https://neonet.pl")
print(driver.title)
search = driver.find_element_by_class_name("inputCss-input__label-263")
search.send_keys(Keys.RETURN)
time.sleep(5)
i try to extract this span
<span class="inputCss-input__label-263">Szukaj produktu</span>

I can see that you are trying to search something in the search bar.
First I recommend you to use the xpath instead of the class name, here is a simple technique to get the xpath of every element on a webpage:
right-click/ inspect element/ select the mouse in a box element on the upper left/ click on the element on the webpage/ it will directly show you the corresponding html/ then right click on the selected html/ copy options and then xpath.
Here is a code example that searches an element on the webpage, I also included the 'Webdriver-wait' option because sometimes the code runs to fast and can't find the next element so this function make the code wait till the element is visible:
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path="/Users/bobo/Downloads/chromedriver")
driver.get("https://neonet.pl") #loading page
wait = WebDriverWait(driver, 20) #defining webdriver wait
search_word = 'iphone\n' # \n is going to act as an enter key
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/main/div[1]/div[4]/div/div/div[2]/div/button[1]'))).click() #clicking on cookies popup
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/main/header/div[2]/button'))).click() #clicking on search button
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/aside[2]/section/form/label/input'))).send_keys(search_word) #searching on input button
print('done!')
sleep(10)
Hope this helped you!

wait=WebDriverWait(driver,10)
driver.get('https://neonet.pl')
elem=wait.until(EC.visibility_of_element_located((By.XPATH, "//span[contains(#class,'inputCss-input')]"))).text
print(elem)
To output the value of the search bar you use .text on the Selenium Webelement.
Imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Python selenium find_element_by_xpath not finding existing element in page

I am using Python 3.6+ and Selenium 3.141
I am trying to get an element from a page, and although I'm using the correct Xpath expression (confirmed in the brower console), the same Xpath expression raises a 'NotFound' error in selenium chrome driver.
myscript.py
from selenium import webdriver
url = 'https://live.euronext.com/en/product/stock-options/AH1-DPAR'
browser = webdriver.Chrome(executable_path='./chromedriver')
browser.get(url)
try:
checkbox = browser.find_element_by_xpath('//*[#id="form-options-index"]/div/div[2]')
except:
pass
The script throws an exception where the find_element_by_xpath() method is invoked - even though when using a browser, the same Xpath expression will result in the element being identified/selected correctly.
Why is the Xpath expression not working with selenium? How do I fix this?

Required content absent in page source - it's loaded dynamically, so you need to wait until it appear in DOM:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://live.euronext.com/en/product/stock-options/AH1-DPAR'
browser = webdriver.Chrome(executable_path='./chromedriver')
browser.get(url)
checkbox = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH, '//*[#id="form-options-index"]/div/div[2]')))

To click on Select all CheckBox. Use WebDriverWait() and wait for element_to_be_clickable() and following Xpath.
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,'//*[#id="form-options-index"]//label[#for="select_all_dates"]'))).click()
You need to import below libraries.
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

Selenium and interacting with searchbar

Hi I'm new to selenium but trying to automate searching on angel.co without success! The search box element changes when clicked on and I'm having difficuly with selenium being able to pick up the elements to do things on. Essentially I want to input growth hacker into the box and do a search. You have to first click on the box, type in the search term and then press enter.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.get("https://angel.co/jobs")
time.sleep(5)
WebDriverWait(driver,20).until(EC.visibility_of_element_located
((By.XPATH,"//[#id='main']/div/div[5]/div[2]/div/div[2]/div[1]/div[1]/
button/div")))
element = driver.find_element_by_xpath('//*[#id="main"]/div/div[5]/div[2]/div/div[2]/div[1]/div[1]/button/div')
driver.execute_script("arguments[0].innerText = 'Growth Hacker';",element)
time.sleep(5)
element.send_keys(Keys.RETURN')
I get as far as being able to input the Growth Hacker into the box, but can't make it become a tag as all search terms do in this search box.
Any help would be really appreciated.
UPDATE
I've managed to insert text into the box but for some reason I can't seem to get it to turn into the tag with the correct text on the search box.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.get("https://angel.co/jobs")
WebDriverWait(driver,60).until(EC.element_to_be_clickable((By.XPATH,'//span[#class="label_82eab"]')))
element4 = driver.find_element_by_xpath('//span[#class="label_82eab"]')
time.sleep(5)
driver.execute_script("arguments[0].innerText='Edinburgh';",element4)
time.sleep(5)
WebDriverWait(driver,60).until(EC.element_to_be_clickable((By.XPATH,'//span[#class="label_82eab"]')))
driver.find_element_by_xpath('//span[#class="label_82eab"]').click()
Before Click
After click
Text to Change

So i tried my own way and was able to search inputted result, you can try the code and see if its what you want, because your question wanst clear enough. From my understanding your finding difficulties in searching the text field which works fine in my code. Try it out and let me know if i can help !
You can always write the code without the classes if your not familiar with selenium and classes!
# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
import unittest, time, re
class Angel(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Chrome("change to ur chrome driver")
self.driver.implicitly_wait(30)
def test_angel(self):
driver = self.driver
driver.get("https://angel.co/")
driver.find_element_by_id("search").click()
time.sleep(3)
driver.find_element_by_id("search").clear()
time.sleep(3)
driver.find_element_by_id("search").send_keys("Growth Hacker")
time.sleep(3)
driver.find_element_by_id("search").send_keys(Keys.ENTER)
time.sleep(3)
def tearDown(self):
self.driver.quit()
if __name__ == "__main__":
unittest.main()

Python 2.7 Selenium No Such Element on Website

I'm trying to do some webscraping from a betting website:
As part of the process, I have to click on the different buttons under the "Favourites" section on the left side to select different competitions.
Let's take the ENG Premier League button as example. I identified the button as:
(source: 666kb.com)
The XPath is: //*[#id="SportMenuF"]/div[3] and the ID is 91.
My code for clicking on the button is as follows:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_path = "C:\Python27\Scripts\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("URL Removed")
content = driver.find_element_by_xpath('//*[#id="SportMenuF"]/div[3]')
content.click()
Unfortunately, I always get this error message when I run the script:
"no such element: Unable to locate element:
{"method":"xpath","selector":"//*[#id="SportMenuF"]/div[3]"}"
I have tried different identifiers such as CCS Selector, ID and, as shown in the example above, the Xpath. I tried using waits and explicit conditions, too. None of this has worked.
I also attempted scraping some values from the website without any success:
from selenium import webdriver
from selenium.webdriver.common.by import By
chrome_path = "C:\Python27\Scripts\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("URL removed")
content = driver.find_elements_by_class_name('price-val')
for entry in content:
print entry.text
Same problem, nothing shows up.
The website embeddes an iframe from a different website. Could this be the cause of my problems? I tried scraping directly from the iframe URL, too, which didn't work, either.
I would appreciate any suggestions.

Sometimes elements are either hiding behind an iframe, or they haven't loaded yet
For the iframe check, try:
driver.switch_to.frame(0)
For the wait check, try:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, '-put the x-path here-')))