I'm new in Webscraping, I want to click this button "Afficher plus" which has the HTML code below.
<button class="more-results js-more-results" id="search-more-results">Afficher plus de résultats</button>
I tried the code below in selenium but it doesn't click the button.
driver = webdriver.Safari()
driver.get(carte)
bt = driver.find_element_by_id("search-more-results")
bt.click()
"carte" is the link the of web page I want to scrape.
1 Check if this button is not inside an iframe.
2 If not, try waiting until this button is clickable
3 '"carte" is the link the of web page I wanna to scrape.' You are using get() function incorrectly. It is used to open a page, not to get a link from a page.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Safari()
driver.get("YOUR WEB PAGE")
wait = WebDriverWait(driver, 30)
wait.until(EC.element_to_be_clickable((By.ID, "search-more-results")))
bt = driver.find_element_by_id("search-more-results")
bt.click()
If this won't work, try CSS selector or XPath. Example for CSS:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#search-more-results")))
email = driver.find_element_by_css_selector('#search-more-results').click()
Related
When I run the code, it opens the browser, but I want to click the reject all button when the cookie banner comes from YouTube. I tried using class and a link text. It did not work.
I will appreciate any help.
from selenium import webdriver
from selenium.webdriver.common.by import By
url = 'https://www.youtube.com/'
driver = webdriver.Chrome(executable_path="C:\\Users\\donner\\Downloads\\chromedriver_win32\\chromedriver.exe")
driver.get(url)
driver.implicitly_wait(100)
continue_link = driver.find_element(By.LINK_TEXT, "Reject all")
continue_link.click()
driver.implicitly_wait(100)
content = driver.find_element(By.CLASS_NAME, '.style-scope.ytd-button-renderer.style-primary size-default')
content.click()
Try the below xpath with the given code
//*[normalize-space()='Reject all']
OR
//*[contains(text(),'Reject all')]
The Code:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "myXpath")))
element.click();
The above soultion work if there is no iframe
In case there is iframe, than you need to follow below
WebDriverWait(driver,20).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"iframe")))
driver.find_element(By.XPATH, "yourXpath").click()
I want to scrape links to news articles using scrapy + selenium. The website I am using uses a 'Load more' button, so I obviously want selenium to click on this button to load all articles.
I have looked for similar questions and tried various options already such as
element = driver.find_element(By.XPATH, value='//*[#id="fusion-app"]/main/div/div/div/div/div[4]/div/div/button')
driver.execute_script("arguments[0].click();", element)
and
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ais-InfiniteHits-loadMore")))
ActionChains(driver).move_to_element(element).click().perform()
All to no result. I've also inserted some print statements in between to check whether it does run the code, and that seems to work fine; I think it's just a matter of the button not being located/clicked on.
This is the html of the button btw:
<button class="ais-InfiniteHits-loadMore">Load more </button>
And when I print element, this is what I get: <selenium.webdriver.remote.webelement.WebElement (session="545716eef622a12bdbeddef99e02bdef", element="551741ec-4616-4bd4-b8fd-57c2f4bffb00")>
Is someone able to help me out? Thank you in advance.
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get('https://www.businessoffashion.com/search/?q=Louis+Vuitton&f=Articles%2CFashion+Shows%2CNews')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'button.ab-close-button'))).click()
elem=wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ais-InfiniteHits-loadMore")))
driver.execute_script("arguments[0].click()", elem)
You hit two different errors with a pop up and an element click intereception when you can just use javascript to click that element.
Import:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I've been trying to scrape this website Link I'm interested in clicking the first DONWLOAD button. However whenever I try to find any element that is a button, I can't find any.
Here is the code :
url = 'https://ember-climate.org/data/carbon-price-viewer/'
webdriver = create_driver()
with webdriver as driver:
driver.maximize_window()
driver.get(url)
wait = WebDriverWait(driver, 30)
try:
wait.until(EC.element_to_be_clickable((By.XPATH, '//button')))
except:
pass
ids = driver.find_elements_by_xpath('//button')
for ii in ids:
print (ii.tag_name, ii.get_attribute('class'))
Is there anything wrong with the XPath or is it an issue with the website itself?
Your xpath is wrong, Download button is wrapped inside an span tag not button tag.
try this instead :
//span[contains(text(),'DOWNLOAD')]
also, I see it's in iframe, which can be located via
iframe[name='ETS']
CSS_SELECTOR, and we need to switch also to this iframe.
so in sequence the explanation would be :
You would have to click on cookies button.
Download button is in an iframe, we need to switch to iframe first and then we can interact with download button.
download button is a part of span tag not button tag.
Use Explicit waits.
Prefer id, css over xpath. (if they are unique in nature)
Launch browser in full screen mode.
Code :
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(30)
wait = WebDriverWait(driver, 50)
driver.get("https://ember-climate.org/data/carbon-price-viewer/")
wait.until(EC.element_to_be_clickable((By.ID, "cn-accept-cookie"))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[name='ETS']")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[contains(text(),'DOWNLOAD')]"))).click()
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I am trying to scrape some google data but I first want to click the 'I agree' button that google pops up. This is the script I use to do that:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
search_question = input("Ask a question: ")
driver = webdriver.Chrome("*Your Webdriver location*")
driver.wait = WebDriverWait(driver, 5)
driver.get("https://google.com")
time.sleep(1)
agree = driver.wait.until(EC.presence_of_element_located((By.XPATH, '//*[#id="introAgreeButton"]/span/span')))
agree.click()
# time.sleep(0.2)
search = driver.find_element_by_class_name("gLFyf")
search.send_keys(search_question)
search.send_keys(Keys.ENTER)
The problem is selenium doesn't seem to locate the button and therefore I get a timeout error. (I have tried also with find_element_by_xpath and still not working).
If you scroll up in the devtools inspector you'll notice that your element is within an iframe:
You need to switch to that frame first, click your button then switch back to the default content (the main page)
driver.get("https://google.com")
#active the iframe and click the agree button
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe")))
agree = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="introAgreeButton"]/span/span')))
agree.click()
#back to the main page
driver.switch_to_default_content()
That works for me.
FYI - There's only 1 iframe on the page, that's why the xpath //iframe works. If there were multiple you'd need to identify it with higher accuracy.
If you already have been agreed,then agree button would not appear. That why it is not be able to find given XPath.
Try this:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
search_question = input("Ask a question: ")
driver = webdriver.Chrome(".\chromedriver.exe")
driver.wait = WebDriverWait(driver, 5)
driver.get("https://google.com")
time.sleep(3)
# agree = driver.wait.until(EC.presence_of_element_located((By.XPATH, '//*[#id="introAgreeButton"]/span/span')))
# agree.click()
# time.sleep(0.2)
search = driver.find_element_by_class_name("gLFyf")
search.send_keys(search_question)
search.send_keys(Keys.ENTER)
Managed to get it working.
Seems like one of the few elements that are interactive in the popup when you first load the site is the language dropdown, which I found by class name.
Then the Agree button is detectable and you can find it by class too.
driver.get("https://www.google.com")
driver.maximize_window()
time.sleep(2)
#finds+clicks language dropdown, interaction unhides the rest of the popup html
driver.find_element_by_class_name('tHlp8d').click()
time.sleep(2)
#finds+clicks the agree button now that it has become visible
driver.find_element_by_id('L2AGLb').click()
you should now have the standard searchbar etc
i had some problems with clicking pop-ups and the problem turned out to be with that click().
im not sure that it might be the same issue with urs or not but try changing ur click to this :
agree = driver.find_element_by_xpath('//*[#id="introAgreeButton"]/span/span')
driver.execute_script("arguments[0].click();", agree)
The problem was that it didn't change frames here is the soulution code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome('C:\\Users\\gassp\\OneDrive\\Namizje\\Python.projects\\chromedriver.exe')
url = 'https://www.google.com/maps/'
driver.get(url)
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, '//*[#id="consent-bump"]/div/div[1]/iframe')))
agree = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="introAgreeButton"]/span/span')))
agree.click()
#back to the main page
driver.switch_to_default_content()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="searchboxinput"]'))).send_keys('gostilne')
driver.submit()
I am trying to click on a non-button element (checkbox) inside an iframe but am not able to click on it using the selenium webdriver. The URL is https://www.nissanoflithiasprings.com/schedule-service and the box that I want to click is shown in the screenshot below:
NOTE: Select the following options to reach the screen shown in the screenshot below: Click on New Customer "MAKE. MODEL. YEAR". Select Make as "NISSAN" -> Year as "2018" -> Model as "ALTIMA" -> Trim as "SL" -> Engine Type as "I4" -> Enter Mileage as "35000" -> Click on "CONTINUE" at the bottom. On the following page, the goal is to click on the checkbox Maintenance Package Standard / Schedule 1 (the checkbox needs to be clicked but I am not able to find the correct code line for Selenium to be able to click on it)
Below is the working Selenium Python script that I wrote to successfully navigate until the page that shows the checkbox that I wish to click:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
chrome_path = r"C:\Users\gh456\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.maximize_window()
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
wait = WebDriverWait(driver, 10)
# first frame - by css selector
wait.until(ec.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, '[src^="https://consumer.xtime.com"]')))
# second frame - by ID
wait.until(ec.frame_to_be_available_and_switch_to_it('xt01'))
driver.find_element_by_id("new_customer_button").click()
driver.find_element_by_id("NISSAN").click()
wait.until(ec.visibility_of_element_located((By.ID, "2018"))).click()
wait.until(ec.visibility_of_element_located((By.ID, "ALTIMA"))).click()
wait.until(ec.visibility_of_element_located((By.ID, "SL"))).click()
wait.until(ec.visibility_of_element_located((By.ID, "I4"))).click()
wait.until(ec.visibility_of_element_located((By.NAME, "mileage_input"))).send_keys("35000")
wait.until(ec.visibility_of_element_located((By.ID, "continue_button"))).click()
# Click on the checkbox
# ---??????????????????
Can someone help me with the correct code to make selenium webdriver to click that checkbox?
The element is covered by another element...
Also, I changed the expected_conditions as element_to_be_clickable().
So you can use ActionChains:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException, NoSuchElementException
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.common.action_chains import ActionChains
chrome_path = r"C:\Users\gh456\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.maximize_window()
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
wait = WebDriverWait(driver, 10)
# first frame - by css selector
wait.until(ec.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, '[src^="https://consumer.xtime.com"]')))
# second frame - by ID
wait.until(ec.frame_to_be_available_and_switch_to_it('xt01'))
driver.find_element_by_id("new_customer_button").click()
driver.find_element_by_id("NISSAN").click()
wait.until(ec.element_to_be_clickable((By.ID, "2018"))).click()
wait.until(ec.element_to_be_clickable((By.ID, "ALTIMA"))).click()
wait.until(ec.element_to_be_clickable((By.ID, "SL"))).click()
wait.until(ec.element_to_be_clickable((By.ID, "I4"))).click()
wait.until(ec.visibility_of_element_located((By.NAME, "mileage_input"))).send_keys("35000")
wait.until(ec.element_to_be_clickable((By.ID, "continue_button"))).click()
check_box_el = wait.until(ec.visibility_of_element_located((By.XPATH, '//div[#id="maintenance_package_section"]//label[#class="checkbox"]')))
ActionChains(driver).move_to_element(check_box_el).click().perform()
Screenshot:
You might want to use CSS_SELECTOR and not XPATH:
check_box_el = wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, '#maintenance_package_section > div label')))
ActionChains(driver).move_to_element(check_box_el).click().perform()
Try this locator (//*[#class="checkbox"])[1] and use ActionChains.
chk = wait.until(ec.element_to_be_clickable((By.XPATH, '(//*[#class="checkbox"])[1]')))
ActionChains(driver).move_to_element(chk).click().perform()
You can change [1] to [2] or [3] if you want
It seems to be your check box is not enabled. Please, have some conditions following is_selected()...And check whether it is enabled now.....
http://selenium-interview-questions.blogspot.com/2014/03/working-with-checkboxes-in-selenium.html