Here is the website I am trying to scrape: https://www.horrycounty.org/bookings
I have developed a for-loop to cycle through the dates, all I need is to write a simple block of code to click on the "search" button, but I am having some difficulty.
My thoughts are
search = driver.find_element_by_class_name('btn btn-primary')
search.click()
This is my first time using selenium, any help would be welcome.
Try to use JavaScript for the click element (that worked for me):
search = driver.find_element_by_css_selector("span.btn-primary")
driver.execute_script("arguments[0].click();", search)
Try this:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
search = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, 'btn btn-primary')))
search.click()
Try using a CSS Selector span.btn-primary to click on the button:
from selenium import webdriver
URL = "https://www.horrycounty.org/bookings"
driver = webdriver.Chrome()
driver.get(URL)
search = driver.find_element_by_css_selector("span.btn-primary")
search.click()
Related
Can not find the element
The code was writen using python with visual studio code
from time import time
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
driver = webdriver.Chrome()
paginaHit = 'https://hit.com.do/solicitud-de-verificacion/'
driver.get(paginaHit)
driver.maximize_window()
time.sleep(5)
bl = 'SMLU7318830A'
elementoBL = driver.find_element(By.XPATH, '//*[#id="billoflanding"]').send_keys(bl)
# WebDriverWait(driver,2).until(EC.element_to_be_clickable((By.NAME, "bl"))).Click()
The code is OK, but can not find the element in the webpage.
The portion of the page you are trying to access is inside an EMBED tag. It looks similar to an IFRAME so I would start by switching the context to the EMBED tag and then try searching for the element.
driver = webdriver.Chrome()
paginaHit = 'https://hit.com.do/solicitud-de-verificacion/'
driver.get(paginaHit)
driver.maximize_window()
embed = driver.find_element(By.CSS_SELECTOR, "embed")
driver.switch_to.frame(embed)
bl = 'SMLU7318830A'
wait =WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.ID, "billoflanding"))).send_keys(bl)
Couple of additional points:
Don't use sleeps... sleeps are a bad practice. Instead use WebDriverWait when you need to wait for something to happen.
If you are using an ID to find an element, use By.ID and not XPath. ID should be preferred, when available. Next should be a CSS selector and then finally, XPATH only when needed, e.g. to locate elements by contained text or to do complicated DOM traversal.
I'm using selenium 4 with python to automate webpage.
Click() method is not working.
Able to click a button using submit() method only if the path has type=submit, but couldnt able to select a link using click method. Please help if anyone knows alternative option for click() method to click a link.
The following is the code:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
s = Service("C:/Program Files/webdriver/Chrome Driver/chromedriver.exe")
driver = webdriver.Chrome(service=s)
driver.get("https://www.sololearn.com/users/login")
time.sleep(2)
#driver.find_element(By.CLASS_NAME, "sl-login-login-form").send_keys(Keys.ENTER)
#---click here fails--------
driver.find_element(By.CLASS_NAME, "sl-login-login-form").click()
#---click here fails
#driver.find_element(By.CLASS_NAME, "sl-login-login-form").submit()
#login.click()
time.sleep(5)
driver.quit()
**Also i have tried with the below code but still it is not working:**
driver.find_element(By.CLASS_NAME, "sl-login-login-form").send_keys(Keys.ENTER)
driver.find_element(By.CLASS_NAME, "sl-login-login-form").submit()
login.click()
Try the below
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CLASS_NAME, "CLASSNAME")))
element.click();
OR
element = driver.find_element(By.CLASS_NAME, "className")
driver.execute_script("arguments[0].click();", element)
Do not forget to import
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import org.openqa.selenium.JavascriptExecutor
Hi i'm new at selenium and webscraping and i need some help.
i try to scrape one site and i need and i dont know how to get span class.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
PATCH = "/Users/bobo/Downloads/chromedriver"
driver = webdriver.Chrome(PATCH)
driver.get("https://neonet.pl")
print(driver.title)
search = driver.find_element_by_class_name("inputCss-input__label-263")
search.send_keys(Keys.RETURN)
time.sleep(5)
i try to extract this span
<span class="inputCss-input__label-263">Szukaj produktu</span>
I can see that you are trying to search something in the search bar.
First I recommend you to use the xpath instead of the class name, here is a simple technique to get the xpath of every element on a webpage:
right-click/ inspect element/ select the mouse in a box element on the upper left/ click on the element on the webpage/ it will directly show you the corresponding html/ then right click on the selected html/ copy options and then xpath.
Here is a code example that searches an element on the webpage, I also included the 'Webdriver-wait' option because sometimes the code runs to fast and can't find the next element so this function make the code wait till the element is visible:
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path="/Users/bobo/Downloads/chromedriver")
driver.get("https://neonet.pl") #loading page
wait = WebDriverWait(driver, 20) #defining webdriver wait
search_word = 'iphone\n' # \n is going to act as an enter key
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/main/div[1]/div[4]/div/div/div[2]/div/button[1]'))).click() #clicking on cookies popup
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/main/header/div[2]/button'))).click() #clicking on search button
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/aside[2]/section/form/label/input'))).send_keys(search_word) #searching on input button
print('done!')
sleep(10)
Hope this helped you!
wait=WebDriverWait(driver,10)
driver.get('https://neonet.pl')
elem=wait.until(EC.visibility_of_element_located((By.XPATH, "//span[contains(#class,'inputCss-input')]"))).text
print(elem)
To output the value of the search bar you use .text on the Selenium Webelement.
Imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I am trying to scrape some tennis statistics starting from 01-01-2019.
For this I try to scrape the following webpage with selenium: https://www.sofascore.com/de/tennis/2019-01-01
When I click on the first match manually the container on the right side changes and shows the statistics.
This is what I want to access automatically.
When I try to click on the element with selenium it redirects me to another page.
Can anyone tell me why it is not just showing the same content as by manually clicking and how I can solve this issue?
Here is my code:
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
import time
options = Options()
options.binary_location = "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe"
browser = webdriver.Chrome(chrome_options = options)
url = 'https://www.sofascore.com/de/tennis/2019-01-01'
browser.get(url)
browser.maximize_window()
xpath = '/html/body/div[1]/main/div/div[2]/div/div[3]/div[2]/div/div/div/div/div[2]/a/div'
browser.find_element_by_xpath(xpath).click()
time.sleep(2)
browser.close()`
You can use the below xpath :
//div[contains(#class, 'Col-pm5mcz-')]//descendant::div[contains(#class, 'styles__StyledWidget-')]
and get the innerHTML of that using get_attribute method
Code :
url = "https://www.sofascore.com/de/tennis/2019-01-01"
driver.get(url)
xpath = '/html/body/div[1]/main/div/div[2]/div/div[3]/div[2]/div/div/div/div/div[2]/a/div'
driver.find_element_by_xpath(xpath).click()
sleep(2)
details = driver.find_element_by_xpath("//div[contains(#class, 'Col-pm5mcz-')]//descendant::div[contains(#class, 'styles__StyledWidget-')]").get_attribute('innerHTML')
print(details)
The xpath that you are using is absolute xpath /html/body/div[1]/main/div/div[2]/div/div[3]/div[2]/div/div/div/div/div[2]/a/div
try to replace that with Relative xpath.
See if this works
tableRows = driver.find_elements_by_xpath(".//div[#class='ReactVirtualized__Grid ReactVirtualized__List']//following::div/a[contains(#class,'EventCellstyles__Link')]")
for e in tableRows:
e.click()
//You can add implicit wait here for the statics section to load
driver.find_element_by_xpath(".//a[text()='Statistiken']").click()
I am trying to scrape some google data but I first want to click the 'I agree' button that google pops up. This is the script I use to do that:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
search_question = input("Ask a question: ")
driver = webdriver.Chrome("*Your Webdriver location*")
driver.wait = WebDriverWait(driver, 5)
driver.get("https://google.com")
time.sleep(1)
agree = driver.wait.until(EC.presence_of_element_located((By.XPATH, '//*[#id="introAgreeButton"]/span/span')))
agree.click()
# time.sleep(0.2)
search = driver.find_element_by_class_name("gLFyf")
search.send_keys(search_question)
search.send_keys(Keys.ENTER)
The problem is selenium doesn't seem to locate the button and therefore I get a timeout error. (I have tried also with find_element_by_xpath and still not working).
If you scroll up in the devtools inspector you'll notice that your element is within an iframe:
You need to switch to that frame first, click your button then switch back to the default content (the main page)
driver.get("https://google.com")
#active the iframe and click the agree button
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe")))
agree = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="introAgreeButton"]/span/span')))
agree.click()
#back to the main page
driver.switch_to_default_content()
That works for me.
FYI - There's only 1 iframe on the page, that's why the xpath //iframe works. If there were multiple you'd need to identify it with higher accuracy.
If you already have been agreed,then agree button would not appear. That why it is not be able to find given XPath.
Try this:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
search_question = input("Ask a question: ")
driver = webdriver.Chrome(".\chromedriver.exe")
driver.wait = WebDriverWait(driver, 5)
driver.get("https://google.com")
time.sleep(3)
# agree = driver.wait.until(EC.presence_of_element_located((By.XPATH, '//*[#id="introAgreeButton"]/span/span')))
# agree.click()
# time.sleep(0.2)
search = driver.find_element_by_class_name("gLFyf")
search.send_keys(search_question)
search.send_keys(Keys.ENTER)
Managed to get it working.
Seems like one of the few elements that are interactive in the popup when you first load the site is the language dropdown, which I found by class name.
Then the Agree button is detectable and you can find it by class too.
driver.get("https://www.google.com")
driver.maximize_window()
time.sleep(2)
#finds+clicks language dropdown, interaction unhides the rest of the popup html
driver.find_element_by_class_name('tHlp8d').click()
time.sleep(2)
#finds+clicks the agree button now that it has become visible
driver.find_element_by_id('L2AGLb').click()
you should now have the standard searchbar etc
i had some problems with clicking pop-ups and the problem turned out to be with that click().
im not sure that it might be the same issue with urs or not but try changing ur click to this :
agree = driver.find_element_by_xpath('//*[#id="introAgreeButton"]/span/span')
driver.execute_script("arguments[0].click();", agree)
The problem was that it didn't change frames here is the soulution code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome('C:\\Users\\gassp\\OneDrive\\Namizje\\Python.projects\\chromedriver.exe')
url = 'https://www.google.com/maps/'
driver.get(url)
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, '//*[#id="consent-bump"]/div/div[1]/iframe')))
agree = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="introAgreeButton"]/span/span')))
agree.click()
#back to the main page
driver.switch_to_default_content()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="searchboxinput"]'))).send_keys('gostilne')
driver.submit()