I'm trying to scrape some information on prices from a website and get find the different prices for different states and cities. However, an "Accept Cookies" pop up shows up, and I need to click it for my code to be successful, but I am struggling to get Selenium to locate anything to click at all
here is the website www.kettlebellkings.com
here is the html for the button:
<button role="button" data-testid="uc-accept-all-button" class="sc-eDvSVe hSwgIR" style="margin: 0px 6px;">Accept All</button>
I've been using Expected conditions but it just times out every time. Here is my code:
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
path = "C:\Program Files (x86)\msedgedriver.exe"
driver = webdriver.Edge('www.kettlebellkings.com')
driver.get(url)
element_present = EC.element_to_be_clickable((By.CSS_SELECTOR, 'button[data-testid="uc-accept-all-button"]'))
WebDriverWait(driver, 30).until(element_present)
try:
driver.find_element(By.CSS_SELECTOR, 'button[data-testid="uc-accept-all-button"]').click()
except:
clicker = driver.find_element(By.CSS_SELECTOR, 'button[data-testid="uc-accept-all-button"]')
driver.execute_script("arguments[0].click();", clicker)
But it just times out. If I leave out the expected conditions:
path = "C:\Program Files (x86)\msedgedriver.exe"
driver = webdriver.Edge('www.kettlebellkings.com')
driver.get(url)
driver.find_element(By.CSS_SELECTOR, 'button[data-testid="uc-accept-all-button"]').click()
Returns:
Message: no such element: Unable to locate element: {"method":"css selector","selector":"button[data-testid="uc-accept-all-button"]"}
It doesn't appear to be in an iframe either - what am I missing here?
'Accept All' cookies button is inside the Shadow Dom, try the below code:
shadow_root = driver.find_element(By.CSS_SELECTOR, "#usercentrics-root").shadow_root
shadow_root.find_element(By.CSS_SELECTOR, "button[data-testid='uc-accept-all-button']").click()
You can refer how to handle shadow dom elements from the below links:
https://www.lambdatest.com/blog/shadow-dom-in-selenium/
https://www.seleniumeasy.com/selenium-tutorials/accessing-shadow-dom-elements-with-webdriver
you could try setting a cookie that would stop the pop up
just accept cookies on your normal browser, set them in the program and the website will recognize it as you/your account
driver.add_cookie({"name": "key", "value": "value"})
Related
I've been trying to scrape this website Link I'm interested in clicking the first DONWLOAD button. However whenever I try to find any element that is a button, I can't find any.
Here is the code :
url = 'https://ember-climate.org/data/carbon-price-viewer/'
webdriver = create_driver()
with webdriver as driver:
driver.maximize_window()
driver.get(url)
wait = WebDriverWait(driver, 30)
try:
wait.until(EC.element_to_be_clickable((By.XPATH, '//button')))
except:
pass
ids = driver.find_elements_by_xpath('//button')
for ii in ids:
print (ii.tag_name, ii.get_attribute('class'))
Is there anything wrong with the XPath or is it an issue with the website itself?
Your xpath is wrong, Download button is wrapped inside an span tag not button tag.
try this instead :
//span[contains(text(),'DOWNLOAD')]
also, I see it's in iframe, which can be located via
iframe[name='ETS']
CSS_SELECTOR, and we need to switch also to this iframe.
so in sequence the explanation would be :
You would have to click on cookies button.
Download button is in an iframe, we need to switch to iframe first and then we can interact with download button.
download button is a part of span tag not button tag.
Use Explicit waits.
Prefer id, css over xpath. (if they are unique in nature)
Launch browser in full screen mode.
Code :
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(30)
wait = WebDriverWait(driver, 50)
driver.get("https://ember-climate.org/data/carbon-price-viewer/")
wait.until(EC.element_to_be_clickable((By.ID, "cn-accept-cookie"))).click()
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[name='ETS']")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[contains(text(),'DOWNLOAD')]"))).click()
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
So im Trying to make a discord bot that sends news from selected news services, and Ive got it working on every website, except Bild.de. They have got a banner you have to accept before accesing the website, and I cant get past that. Like I said, I had no problems on any other website, but this one.
My code (python):
import time
from selenium import webdriver
# selenium part
url = 'https://www.bild.de/home/newsticker/news/alle-news-54190636.bild.html'
browser = webdriver.Chrome()
browser.get(url)
time.sleep(5)
#trying to accept cookie banner
browser.find_element_by_xpath(
"/html/body/div/div[2]/div[3]/div[2]/button").click()
Error Message
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div/div[2]/div[3]/div[2]/button"}
Things to be noted down here.
Cookies button is in iframe, so first we have to switch to iframe in Selenium.
I am using execute script to click on it.
Remember to switch back to default content when you are done with the iframe.
Code :
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(50)
driver.get("https://www.bild.de/home/newsticker/news/alle-news-54190636.bild.html")
wait = WebDriverWait(driver, 20)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[title='SP Consent Message']")))
button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.message-component.message-row.mobile-reverse>div:nth-child(2)>button")))
driver.execute_script("arguments[0].click();", button)
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
My first test with selenium is to click a button on a website. The first button that I need to click is this "yes you can use cookies"-buttons in the popup of a website. But it seems that selenium doesn't find that button even though I added a wait line. I tried other buttons in the popup as well, but none of them can be found by my element_to_be_clickable. The element is in an iframe, so I guess I have to change to it, but it seems that I'm doing something wrong.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver_path = "D:/Python/learning_webclicker/firefox_driver/geckodriver.exe"
firefox_path = "C:/Program Files/Mozilla Firefox/firefox.exe"
option = webdriver.FirefoxOptions()
option.binary_location = firefox_path
driver = webdriver.Firefox(executable_path=driver_path, options=option)
url = "https://web.de/"
driver.get(url)
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element_by_xpath("/html/body/div[2]/iframe")))
#I tried to find the "save-all-conditionally"-element with lots of different methods:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "save-all-conditionally"))).click()
#WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, """//*[#id="save-all-conditionally"]"""))).click()
#WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.ID, "save-all-conditionally")))
# ...
This raises the error
selenium.common.exceptions.TimeoutException: Message:
And if I try to click the button directly after changing to iframe (or without checking for iframe), then I get
driver.implicitly_wait(10)
element=driver.find_element_by_xpath("""//*[#id="save-all-conditionally"]""")
element.click()
>>> selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [id="save-all-conditionally"]
I guess, that I'm not really in the iframe (although frame_to_be_available_and_switch_to_it doesn't return an error), but I'm not sure how/what/why.
The element you are looking after is inside nested iframe. You need to switch both the
iframes.
Use following css selector to identify the iframe.
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[name='landingpage']")))
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src*='plus.web.de']")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "save-all-conditionally"))).click()
Or Use below xpath to identify the iframe.
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#name='landingpage']")))
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#src,'plus.web.de')]")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "save-all-conditionally"))).click()
First of all if a similar topic occurred earlier I'm sorry but I couldn't find any problem like mine.
I would like to create a simple script which enters an e-mail website, log into my account and finds the amount of unread messages.
This is the part with logging in
from selenium import webdriver
from time import sleep
class sMailBot():
def __init__(self):
self.driver = webdriver.Chrome()
def login(self):
self.driver.get('website.com')
sleep(2)
btn_login = self.driver.find_element_by_xpath('//*[#id="username"]')
btn_login.send_keys('my_username')
btn_password = self.driver.find_element_by_xpath('//*[#id="password"]')
btn_password.send_keys('my_password')
btn_logintoaccount = self.driver.find_element_by_xpath('//*[#id="button"]')
btn_logintoaccount.click()
sleep(5)
It works really well. After logging into my mail account comments like driver.title or driver.current_url work.
Now I would like to scrape this part of html code:
<b>some_important_string_which_stores_the_amount_of_unread_mails</b>
I tried to do this using it's path
driver.find_element_by_xpath('//*[#id="MS_act1"]/span)
However it does not work. Moreover I can't find any other elements from this side.
I would like to highlight that I waiting even more than 10 seconds for the page to load.
The error which occurred
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="MS_act1"]/span/b"}
(Session info: chrome=80.0.3987.87)
As you asked I add some surrounding HTML code
<span style="float: right">
<b>some_important_string_which_stores_the_amount_of_unread_mails</b>
</span>
Please, don't use sleep, it's not a good choice for selenium.
Instead, use selenium waits:
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
finally:
driver.quit()
https://selenium-python.readthedocs.io/waits.html
First of all I will avoid using sleep. You may try using WebDriverWait instead. This will pause the browser until a given condition is satisfied.
e.g. as follows
WebDriverWait(self.driver, 60).until(EC.presence_of_element_located((By.XPATH, "//button[text()='Login']")))
This will wait for 60 sec maximum for the element (button with text Login) to occur in the page.
After logging into your mail account commands like driver.title and driver.current_url works but they are not part of the DOM Tree.
The relevant HTML would have helped us to construct a canonical answer. However to extract the desired text, you have to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR and get_attribute("innerHTML"):
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "[id^='MS_act'] span>b"))).get_attribute("innerHTML"))
Using XPATH and text attribute:
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//*[starts-with(#id, 'MS_act')]//span/b"))).text)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
There is site, that streams youtube videos. I want to get playlist with them. So I use selenium webdriver to get the needed element div with class-name ytp-title-text where youtube link is located.
It is located here for example, when I use browser console to find element:
<div class="ytp-title-text"><a class="ytp-title-link yt-uix-sessionlink" target="_blank" data-sessionlink="feature=player-title" href="https://www.youtube.com/watch?v=VyCY62ElJ3g">Fears - Jono McCleery</a><div class="ytp-title-subtext"><a class="ytp-title-channel-name" target="_blank" href=""></a></div></div>
I wrote simple script for testing:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
driver = webdriver.Firefox()
driver.get('http://awsmtv.com')
try:
element = WebDriverWait(driver, 10).until(
EC.visibility_of_element_located((By.CLASS_NAME, "ytp-title-text"))
)
finally:
driver.quit()
But no element is found and timeout exception is thrown. I cannot understand, what actions selenium needs to perform to get the full page source.
Required link is hidden and also located inside an iframe. Try below to locate it:
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it("tvPlayer_1"))
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASS_NAME, "ytp-title-link")))
print(element.get_attribute('href'))
finally:
driver.quit()
Just saw this element is inside iframe... You need to switch to the iframe first -> find it by ClassName -> ifame = ...(By.CLASS_NAME, "player") then switch to it driver.switch_to_frame(iframe) and you should be able now to get the wanted element :)
The XPath locator like this one will work (or your locator) -> "//a[#class='ytp-title-link yt-uix-sessionlink']".
You then need via the element to get the property href for the youtube video url or the text of the element for the song title.
If still not working I can suggest to get the page source - html = driver.page_source which will give you the source of the page and via some regex to get the info you want eventually.