I'm trying to put together a small scraper for public trademark data. I have a database available that i'm using selenium and python to access.
I can do just about anything I need to be able to, but for some reason i can't actually click the "accept cookies" button on the website. The following code i use highlights the button, but it does not get rid of the popup.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get('https://data.inpi.fr/recherche_avancee/marques')
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "tarteaucitronPersonalize2"))
).click()
I have looked up similar threads on this forum, and I have tried multiple things :
- adding a waiting period, that ended up highlighting the button, so at least i know it does something
- using javascript code to do the actual click, did not work any better
- i tried calling the button via its ID, its XPATH, its CSS selector, anything i could find really
I even downloaded Selenium IDE to record my clicks to see exactly how I could replicate it, but it still only recorded a click.
I tried my best, does anyone know where my mistake lie ? I am open to using other languages, or another platform
Well it looks like I managed to solve it just minutes after posting my question !
For some reason you need to resize the window. I just added the following line of code after opening the URL and it worked first time.
driver.maximize_window()
I added this answer in case anyone stumbles upon this post and wants to avoid pulling their hair out over this !
Related
I have gone through existing questions and google results of a similar nature, every solution posed has not worked for me within the particular website I am currently scraping.
https://dutchie.com/embedded-menu/revolutionary-clinics-somerville/menu
I am sending page down keys to the body element, which loads each item to be scraped. I have two issues with this, first I am unable to detect when the scrolling has stopped. Second, I have to manually click the browser window as it opens to allow the keys to be sent. I am not sure how to mimic this same focus giving behavior via code.
elem = driver.find_element_by_tag_name("body")
elem.send_keys(Keys.PAGE_DOWN)
I have tried the following, in many different iterations and the number printed never charged regardless of how far down the page I am or if I used innerHeight, or body instead of documentElement.
height = driver.execute_script("return document.documentElement.scrollHeight")
If I attempt to scroll down the page using a similar approach, this page does not move.
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
I am unsure if this has to do with iframes or if I am simply misunderstanding the best approach.
Still have been unable to find a way to reliably detect the end of the page.
Thank you!
After importing the required imports
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
you can validate the page button is reached when the following element is visible:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.XPATH, "//p[contains(text(),'License')]"))
As about the second issue, try clicking on the following element with Selenium:
driver.find_element_by_id("products-container").click
I have no environment to debug this, but I guess this will work
Newbie here... 2 days into learning this.
In a learning management system, there is an element (a plus mark icon) to click which adds a form field upon each click. The goal is to click the icon, which generates a new field, and then put text into the new field. This field does NOT exist when the page loads... it's added dynamically based on the clicking of the icon.
When I try to use "driver.find_element_by_*" (have tried ID, Name and xpath), I get an error that it can't be found. I'm assuming it's because it wasn't there when the page loaded. Any way to resolve this?
By the way, I've been successful in scripting the login process and navigating through the site to get to this point. So, I have actually learned how to find other elements that are static.
Let me know if I need to provide more info or a better description.
Thanks,
Bill
Apparently I needed to have patience and let something catch up...
I added:
import time
and then:
time.sleep(3)
after the click on the icon to add the field. It's working!
You can use time.sleep(3) but that would force you to wait for the entire 3 seconds before using that element. In Selenium we use webdriver waits that polls the DOM to allow us to immediately use that element as quick as possible when it is useable.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,""))).click()
I have been messing around with python and selenium and was trying to make a program that could automatically checkout on bestbuy.ca . I have gotten to nearly the last stage of the checkout process but am having difficulty getting the element/clicking the buttons to continue. The only button I have been able to get to work is the add to cart button on the product page.
Here is the site as well as the HTML:
Click here for image
Some things I have tried with no success include:
driver.find_element_by_class_name().click()
For this, I used the class in button and span. I have tried using each individual class in separately and together with little success. It would usually give me a "No Element Found" error or just not click anything afterwards.
driver.driver.find_element_by_xpath().click()
For this, I tried using the element locator extension for the xpath as well as "//*[#type='submit']" with no success.
driver.driver.find_element_by_css_selector().click()
For this, I tried the element locator extension again as well as copying it from the firefox inspect with no success.
Not really sure if there are any other ways to go about doing this but any help would be appreciated.
Did you try to copy the entire class attribute from the html and use :
wait.until(EC.presence_of_element_located((By.XPATH, "//button[#class='asd asd sad asd']"))).click()
copy the class and paste it.
Try either of the following to click on the button tag.
wait=WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Continue']/parent::button"))).click()
Or by attribute data automation
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[#data-automation='review-order']"))).click()
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I am new to python and i am trying to automate click of load more button present in comment section of instagram using selenium and python. I am able to click only once. After first click the button disappears, I have waited for 10 minutes but it did not appear again. And also the status of the request changed to 302 when click in performed using automation. The status code remain 200 when clicked manually. Please help me guys how to perform click until all comment have been loaded. Any help will be appreciated. here is my code
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time
options = webdriver.FirefoxOptions()
options.add_argument('start-maximized')
browser = webdriver.Firefox(executable_path='./drivers/geckodriver')
browser.maximize_window()
url = 'https://www.instagram.com/p/CKCVIu2gDgn'
browser.get(url)
browser.implicitly_wait(10) #wait for 10 sec
load_more = browser.find_element(By.XPATH,'/html/body/div[1]/section/main/div/div[1]/article/div[3]/div[1]/ul/li/div/button/span').click()
For something like this, you might want to use Selenium's excute_script feature, which basically lets you execute Javascript on the webpage.
The following Javascript code will find the button in the comments and click it.
document.querySelector("button.dCJp8.afkep").click()
For such a feat, you would not want to use Selenium to click it because Selenium tries to mimic human behaviour as much as possible. In this case, Selenium would try to find the button, and if it is not visible on the page (ie. you need to scroll to find it), it would raise an Exception.
The following is a line of Python code you can use in your program to click the load more button.
driver.execute_script("document.querySelector('button.dCJp8.afkep').click()")
By the way, I am assuming that this round circular plus sign button is the click target.
Use instaloader library, it handles such stuff very easily.
here is the full documentation.
I am trying to access PenFed in order to get my current outstanding amount. I have done quite a bit of research and unfortunately, I am still stumped. I am using Python Selenium and I am trying to click on the initial login button on the side in order to see the username field. This is the element's HTML code:
Login
When I try to run the following code:
driver.find_element_by_id("mobile-login").click()
I get the following error:
selenium.common.exceptions.ElementNotVisibleException: Message: element not interactable
Even when I try to use WebDriver Wait functions such as these:
try:
WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.ID, "mobile-login"))).click()
except ElementNotVisibleException:
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.ID, "mobile-login"))).click()
No matter how long I make them wait, I get a timeout message:
raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message:
All of my research says that invoking a wait function should fix it but it doesn't work for me. I also read that there might be an image overlay on top of the element that I would have to invoke before clicking on the button, but I didn't see anything in the website code as well. If I am testing it out, the only way that I am able to click on the button through code is if I physically click on it first, so I am unaware of anything else I can use. Thank you in advance for the help!
UPDATE: I have discovered that the following code works for me:
element = driver.find_element_by_id("mobile-login")
driver.execute_script("$(arguments[0]).click();", element)
But I do not know what the execute_script actually does. Can someone explain that piece of code works or if any other alternatives work for them?
The code you specified is JQuery. execute_script(p1, p2) runs a js script, where p1 is the script (in your case a JQuery line that clicks the element) and p2 is the desired element. It seems like you shouldn't need p2 if arguments[0] is equal to "element," but I'm not totally sure.
One potential fix is to use a counter for the number of times you clicked the element. If the counter reaches a certain number and the page doesn't change (you can check by finding a unique element/value on your current page), then you know it's not clickable.
Good luck!
The desired element is a dynamic element so to locate the element you have to induce WebDriverWait for the element to be clickable and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument('start-maximized')
options.add_argument('--disable-extensions')
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.penfed.org/')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.pfui-button.login-slide-button.pfui-button-login.dtm-global-nav[data-id='Open Log In Drawer']"))).click()
Browser Snapshot:
The link you are attempting to click is for the mobile site and is not visible if you are viewing the site at desktop resolutions. If you shrink your browser down until it changes layout, you will see the LOGIN button appear that corresponds to that link. That's why you are getting the ElementNotVisibleException.
To your second question, the reason that using .execute_script() works is that it executes JS directly and can click on anything... hidden or not. Selenium was designed to interact with the page as a user would so it won't let you click invisible elements, etc.
If you are intending for your script to act like a user, you will want to avoid using .execute_script() because it allows you to do things on the page that a user cannot do.
If you want to log in like a desktop user would, you need to click the "LOG IN" button using the CSS selector below
button[data-id='Open Log In Drawer']
That will open a side panel where you can enter your username, etc. and log in. FYI... you will likely need a wait to give the panel a chance to open before continuing the log in process.