I want to push a button on a Google web page, but selenium can't locate it.
Here's how the page appears:
Here's the html:
https://search.google.com/search-console/about
<span class="RveJvd snByac">Start now</span>
Here's the code:
def show_webpage(judge_url):
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(SITE)
button_element = driver.find_element_by_class_name('RveJvd snByac')
button_element[1].click()
html_source = driver.page_source
driver.close()
return html_source
And this is the error:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such
element: Unable to locate element: {"method":"css
selector","selector":".RveJvd snByac"}
As Micheal said, the find_element_by_class_name takes only one class name at a time as argument. you are passing two. if you want to use two class name then you can use css selector instead as given below.
def show_webpage(judge_url):
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(SITE)
button_element = driver.find_element_by_css_selector('.RveJvd.snByac')
button_element[1].click()
html_source = driver.page_source
driver.close()
return html_source
Passing multiple classNames within find_element_by_class_name() will result in Invalid selector: Compound class names not permitted using find_element_by_class_name
Moreover, the classnames e.g. RveJvd, snByac, etc looks dynamic.
However, to click on the button with text as Start now on the Google web page https://search.google.com/search-console/about you can use the following Locator Strategy:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("start-maximized")
# chrome_options.add_argument('disable-infobars')
driver = webdriver.Chrome(options=chrome_options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://search.google.com/search-console/about")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='Improve your performance on Google Search']//following::div[1]//span[text()='Start now']"))).click()
Most probably these class names are changing each time the page is loaded, you should rather stick to Start now text of the span tag
There is no guarantee that the element will be immediately available in DOM so consider using Explicit Wait to ensure that the document is there
Suggested code change:
driver.get("https://search.google.com/search-console/about")
start_now = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//span[text()='Start now']")))
driver.execute_script("arguments[0].click()", start_now)
More information: How to use Selenium to test web applications using AJAX technology
Related
I am web-scraping reviews from Goodreads for a project. Here's an example of a page I've been trying: https://www.goodreads.com/book/show/2767052-the-hunger-games/reviews?
The reviews page initially shows 30 reviews with a 'Show More' button at the bottom. Selenium seems unable to click the button.
Here is the code I'm using:
showmore_button = driver.find_element(By.XPATH, '/html/body/div[1]/div/main/div[1]/div[2]/div[4]/div[4]/div/button/span[1]')
driver.execute_script("arguments[0].click();", showmore_button)
I have also tried
showmore_button.click()
but that leads to an exception stating that the element is not clickable
For more context my driver is set up like this:
def createdriver():
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument("start-maximized")
options.add_argument('--window-size=1920,1080')
options.add_argument("--incognito")
driver = webdriver.Chrome(options=options)
return driver
and then I use:
driver = createdriver()
driver.get(url)
Where the URL is the reviews page I'm trying to scrape
To click on the element Show more reviews at the bottom of the page you need to scrollIntoView() inducing WebDriverWait for the visibility_of_element_located() and you can use the following locator strategies:
Code block:
driver.get('https://www.goodreads.com/book/show/2767052-the-hunger-games/reviews?')
time.sleep(5)
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[#class='ReviewsList__listContext ReviewsList__listContext--centered']//span[contains(., 'Displaying 1 -')]"))))
driver.execute_script("arguments[0].click();", driver.find_element(By.XPATH, "//span[text()='Show more reviews']"))
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser snapshot:
As the page loads for few milli-seconds before the element is displayed, you need to apply selenium waits. Try using implicit wait after creating the driver instance, see code below:
driver = createdriver()
driver.implicitly_wait(10)
driver.get(url)
Above code waits for 10 seconds searching for the element before throwing error
Also another suggestion:
Instead of using an absolute XPath, as a best practice use relative XPath. This is because relative XPath is more consistent compared to absolute XPath. Absolute XPath may stop working, If the DOM structure changes in the future. Try the below relative XPath:
showmore_button = driver.find_element(By.XPATH, '//span[contains(text(),"Show more reviews")]')
driver.execute_script("arguments[0].click();", showmore_button)
So im Trying to make a discord bot that sends news from selected news services, and Ive got it working on every website, except Bild.de. They have got a banner you have to accept before accesing the website, and I cant get past that. Like I said, I had no problems on any other website, but this one.
My code (python):
import time
from selenium import webdriver
# selenium part
url = 'https://www.bild.de/home/newsticker/news/alle-news-54190636.bild.html'
browser = webdriver.Chrome()
browser.get(url)
time.sleep(5)
#trying to accept cookie banner
browser.find_element_by_xpath(
"/html/body/div/div[2]/div[3]/div[2]/button").click()
Error Message
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div/div[2]/div[3]/div[2]/button"}
Things to be noted down here.
Cookies button is in iframe, so first we have to switch to iframe in Selenium.
I am using execute script to click on it.
Remember to switch back to default content when you are done with the iframe.
Code :
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(50)
driver.get("https://www.bild.de/home/newsticker/news/alle-news-54190636.bild.html")
wait = WebDriverWait(driver, 20)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[title='SP Consent Message']")))
button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.message-component.message-row.mobile-reverse>div:nth-child(2)>button")))
driver.execute_script("arguments[0].click();", button)
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
My first test with selenium is to click a button on a website. The first button that I need to click is this "yes you can use cookies"-buttons in the popup of a website. But it seems that selenium doesn't find that button even though I added a wait line. I tried other buttons in the popup as well, but none of them can be found by my element_to_be_clickable. The element is in an iframe, so I guess I have to change to it, but it seems that I'm doing something wrong.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver_path = "D:/Python/learning_webclicker/firefox_driver/geckodriver.exe"
firefox_path = "C:/Program Files/Mozilla Firefox/firefox.exe"
option = webdriver.FirefoxOptions()
option.binary_location = firefox_path
driver = webdriver.Firefox(executable_path=driver_path, options=option)
url = "https://web.de/"
driver.get(url)
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element_by_xpath("/html/body/div[2]/iframe")))
#I tried to find the "save-all-conditionally"-element with lots of different methods:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "save-all-conditionally"))).click()
#WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, """//*[#id="save-all-conditionally"]"""))).click()
#WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.ID, "save-all-conditionally")))
# ...
This raises the error
selenium.common.exceptions.TimeoutException: Message:
And if I try to click the button directly after changing to iframe (or without checking for iframe), then I get
driver.implicitly_wait(10)
element=driver.find_element_by_xpath("""//*[#id="save-all-conditionally"]""")
element.click()
>>> selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [id="save-all-conditionally"]
I guess, that I'm not really in the iframe (although frame_to_be_available_and_switch_to_it doesn't return an error), but I'm not sure how/what/why.
The element you are looking after is inside nested iframe. You need to switch both the
iframes.
Use following css selector to identify the iframe.
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[name='landingpage']")))
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src*='plus.web.de']")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "save-all-conditionally"))).click()
Or Use below xpath to identify the iframe.
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#name='landingpage']")))
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#src,'plus.web.de')]")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "save-all-conditionally"))).click()
First of all if a similar topic occurred earlier I'm sorry but I couldn't find any problem like mine.
I would like to create a simple script which enters an e-mail website, log into my account and finds the amount of unread messages.
This is the part with logging in
from selenium import webdriver
from time import sleep
class sMailBot():
def __init__(self):
self.driver = webdriver.Chrome()
def login(self):
self.driver.get('website.com')
sleep(2)
btn_login = self.driver.find_element_by_xpath('//*[#id="username"]')
btn_login.send_keys('my_username')
btn_password = self.driver.find_element_by_xpath('//*[#id="password"]')
btn_password.send_keys('my_password')
btn_logintoaccount = self.driver.find_element_by_xpath('//*[#id="button"]')
btn_logintoaccount.click()
sleep(5)
It works really well. After logging into my mail account comments like driver.title or driver.current_url work.
Now I would like to scrape this part of html code:
<b>some_important_string_which_stores_the_amount_of_unread_mails</b>
I tried to do this using it's path
driver.find_element_by_xpath('//*[#id="MS_act1"]/span)
However it does not work. Moreover I can't find any other elements from this side.
I would like to highlight that I waiting even more than 10 seconds for the page to load.
The error which occurred
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="MS_act1"]/span/b"}
(Session info: chrome=80.0.3987.87)
As you asked I add some surrounding HTML code
<span style="float: right">
<b>some_important_string_which_stores_the_amount_of_unread_mails</b>
</span>
Please, don't use sleep, it's not a good choice for selenium.
Instead, use selenium waits:
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
finally:
driver.quit()
https://selenium-python.readthedocs.io/waits.html
First of all I will avoid using sleep. You may try using WebDriverWait instead. This will pause the browser until a given condition is satisfied.
e.g. as follows
WebDriverWait(self.driver, 60).until(EC.presence_of_element_located((By.XPATH, "//button[text()='Login']")))
This will wait for 60 sec maximum for the element (button with text Login) to occur in the page.
After logging into your mail account commands like driver.title and driver.current_url works but they are not part of the DOM Tree.
The relevant HTML would have helped us to construct a canonical answer. However to extract the desired text, you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR and get_attribute("innerHTML"):
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "[id^='MS_act'] span>b"))).get_attribute("innerHTML"))
Using XPATH and text attribute:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//*[starts-with(#id, 'MS_act')]//span/b"))).text)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
There is site, that streams youtube videos. I want to get playlist with them. So I use selenium webdriver to get the needed element div with class-name ytp-title-text where youtube link is located.
It is located here for example, when I use browser console to find element:
<div class="ytp-title-text"><a class="ytp-title-link yt-uix-sessionlink" target="_blank" data-sessionlink="feature=player-title" href="https://www.youtube.com/watch?v=VyCY62ElJ3g">Fears - Jono McCleery</a><div class="ytp-title-subtext"><a class="ytp-title-channel-name" target="_blank" href=""></a></div></div>
I wrote simple script for testing:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
driver = webdriver.Firefox()
driver.get('http://awsmtv.com')
try:
element = WebDriverWait(driver, 10).until(
EC.visibility_of_element_located((By.CLASS_NAME, "ytp-title-text"))
)
finally:
driver.quit()
But no element is found and timeout exception is thrown. I cannot understand, what actions selenium needs to perform to get the full page source.
Required link is hidden and also located inside an iframe. Try below to locate it:
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it("tvPlayer_1"))
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, "ytp-title-link")))
print(element.get_attribute('href'))
finally:
driver.quit()
Just saw this element is inside iframe... You need to switch to the iframe first -> find it by ClassName -> ifame = ...(By.CLASS_NAME, "player") then switch to it driver.switch_to_frame(iframe) and you should be able now to get the wanted element :)
The XPath locator like this one will work (or your locator) -> "//a[#class='ytp-title-link yt-uix-sessionlink']".
You then need via the element to get the property href for the youtube video url or the text of the element for the song title.
If still not working I can suggest to get the page source - html = driver.page_source which will give you the source of the page and via some regex to get the info you want eventually.