We Keep Coding

Cannot use WebDriverWait library properly in Python Selenium

Hello everyone I'm learning selenium recently and as I bet is a classic newbie mistake I filled my code with time.sleep which made everything really slow. So I started to learn about webdriverWait. I took some sample code and I have tried multiple things but I still get the error of "what you clicked is blocked by this other thing" meaning that the library did not crash but it also did not do anything. Which must mean I am doing something wrong. This is an error I can avoid if use the time.sleep function.
I'm using expected conditions, BY and action chains alongside WebDriverWait, though in my last test I tried to not use Acton chains to lower the possibilities of why its failing.
I'm using a company site that is probably under NDA so I don't have a public example to show this on, I tried searching for "pages with cover openings" but I couldn't find any, so if you guys know of any that I can use to illustrate this I would also find that really helpful. What is happening is that the site loads with a "cover" animation and after a few seconds the animation goes away to reveal the Button I'm looking for.
Environment info:
Using Pycharm community latest version
Using Selenium 4.0.0b2.post1 (I also tried in 3.9 to no results)
Using ChromeDriver 89 as my google chrome version is 89 as well [Version 89.0.4389.114 for Chrome]
Here are my code snippets:
Attempt #1:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
import selenium.webdriver.support.ui as ui
def test(driver, actions, delay):
cart = "//header/div[1]/div[1]/a[2]/*[1]"
cartxp = WebDriverWait(driver, delay).until(EC.visibility_of_element_located((By.XPATH, cart)))
actions.move_to_element(cartxp).perform()
cartclk = driver.find_element_by_xpath(cart).click()
Attempt #2: Using UI library
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
import selenium.webdriver.support.ui as ui
def test2(driver, delay, actions):
cart = "//header/div[1]/div[1]/a[2]/*[1]"
cartxp = ui.WebDriverWait(driver, delay).until(EC.visibility_of_element_located((By.XPATH, cart)))
actions.move_to_element(cartxp).perform()
cartclk = driver.find_element_by_xpath(cart).click()
Attempt #3: Not using Action chains
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
import selenium.webdriver.support.expected_conditions as EC
from selenium.webdriver.common.by import By
def test3(driver, delay):
wait = WebDriverWait(driver, delay)
cart = "//header/div[1]/div[1]/a[2]/*[1]"
button = wait.until(EC.element_to_be_clickable((By.XPATH, cart)))
cartclk = driver.find_element_by_xpath(cart).click()
All of these have a main method that just has:
driver = webdriver.Chrome()
test3(driver, 30)
driver.get('Site that I cant reveal cuz of NDA')
action = ActionChains(driver)
The error I get is:
Message: element click intercepted: Element ... is not clickable at point (1183, 27). Other element would receive the click
Thanks in advance for any help!
Edit: After some comments changed my xpath to //a[#class='header__cart'] both the previous and this one leads to a single result If I use it on chrome inspect to look for the button, additionally it works as intended if I use it to actually click the button after using time.sleep() to wait out the animation
Additionally just in case tried using the try catch as they did on the suggested question. It also failed
Attempt 4: surrounding in a try-catch
def test4(driver, delay):
cart = "//a[#class='header__cart']"
try:
my_elem = WebDriverWait(driver, delay).until(EC.visibility_of_element_located((By.XPATH, cart)))
print
"Page is ready!"
except TimeoutException:
print
"Loading took too much time!"
cart_btn = driver.find_element_by_xpath(cart)
cart_btn.click()
Error: selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <a href="/cart" class="header__cart" data-items-in-cart="0" )="">... is not clickable at point (1200, 27). Other element would receive the click
To clarify what I meant earlier this works:
cart = "//a[#class='header__cart']"
time.sleep(20)
cartclk = driver.find_element_by_xpath(cart).click()

I'm trying to automate a process on a website. How do I close a pop up window with selenium, when it is not clear when the popup window will appear?

I am trying to close a pop up window with selenium in python, which is not allowing my code to execute further. But I am unable to do that. There is a a pop up window which asks me if i want to sign up but it keeps popping up at inconsistent times. Is there a method to check wether or not the pop-up window is active?
My code so far:
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
PATH = "C:\Program Files (x86)\chromedriver.exe";
driver = webdriver.Chrome(PATH);
driver.get("https://www.investing.com/news/")
time.sleep(3)
accept_cookies = driver.find_element_by_xpath('//*[#id="onetrust-accept-btn-handler"]');
accept_cookies.click();

So your problem is that you don't want the signup popup to display.
I was just poking the site's script, I found a really nice way for you to work around it: set user agent to gene. The script to display popup will be disabled if user agent matches some mobile browsers.
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
opts = Options()
opts.add_argument("user-agent=gene")
driver = webdriver.Chrome(options=opts)
# Adds the cookie into current browser context
driver.get("https://www.investing.com/news/")
time.sleep(60) # wait for a minute to see if the popup happens
driver.quit()
Run this code, observe the page, it will not display the signup popup.
Alternative work around:
Use a Chrome profile that already turned off the Signup popup, will also not display the popup
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Path") #Path to your chrome profile
If you really need to close the popup without these methods, define a check function with try/except. Before doing any actions, call this function to check if there's popup, then close the popup, save the state to some variable (so you don't check it anymore). Since the function has try/except, it will not raise Exception and your code will run.

You can do this with 2 quick methods that I can think of immediately.
1:
You will use this often
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
then when you want to get an element that needs time to load and you want to do actions with it, you implement it with WebDriverWait :
wait = WebDriverWait(driver, 10)
try:
accept_cookies = wait.until(EC.presence_of_element_located((By.XPATH, "'//*
[#id=\"onetrust-accept-btn-handler\"]'")))
catch:
# probably you have a bad locator and the element doesn't exist or it needs more than 10 sec to load.
else:
# logic here is processed after successfully obtaining the pop up
accept_cookies.click()
I would recommend using ActionChains
from selenium.webdriver.common.action_chains import ActionChains
then after you obtain the pop up element make the click like this:
actions = ActionChains(driver)
actions.move_to_element(accept_cookies).click().perform()
This is the equivalent of moving the mouse pointer to the middle of the pop up close btn and clicking it

Why does sometimes selenium locate objects and sometimes no?

Good morning,
I am trying to automatize simple online procedure: go to this site http://nvidia-research-mingyuliu.com/gaugan/
Check the little box, update a picture, render it and download it clicking the download button.
I have written my code, which is extremely intuitive and very short, but for some reasons sometimes it works and sometimes it doesn't.
Sure enough, sometimes python returns the following error:
no such element: Unable to locate element: {"method":"css selector","selector":"[id="myCheck"]"}
My code is:
import selenium
import time
from selenium import webdriver
# Using Chrome to access web
driver = webdriver.Chrome(executable_path='/Users/Marco/Downloads/chromedriver')
# Open the website
driver.get('http://nvidia-research-mingyuliu.com/gaugan/')
# Select the checkbox
time.sleep(5)
check_box = driver.find_element_by_id('myCheck')
check_box.click()
# Upload File button
element = driver.find_element_by_id("myCheck")
driver.execute_script("arguments[0].click();", element)
choose_file = driver.find_element_by_id('segmapfile')
# Send the file location to the button
choose_file.send_keys('/Users/Marco/Desktop/Foto upload/Schermata 2020-10-31 alle 00.07.03.png')
#Locate submit button and click
submit_assignment = driver.find_element_by_id('btnSegmapLoad')
submit_assignment.click()
#render
render_button = driver.find_element_by_id('render')
render_button.click()
#download
save_box = driver.find_element_by_id('save_render')
time.sleep(5)
save_box.click()
Why is this happening? How should i solve it?
Thank you

Use explicit waits which wait for something to happen, as opposed to 'time.sleep(5)' which just waits for 5 seconds:
import selenium
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
# Using Chrome to access web
driver = webdriver.Chrome(executable_path='/Users/Marco/Downloads/chromedriver')
# Open the website
driver.get('http://nvidia-research-mingyuliu.com/gaugan/')
# Wait for checkbox to be located
check_box_wait = EC.presence_of_element_located((By.ID, 'myCheck'))
WebDriverWait(driver, 10).until(check_box_wait)
# Select the checkbox
check_box = driver.find_element_by_id('myCheck')
check_box.click()

Unable to input values fully in a text field using send keys. Only partial values entered and no error shown

Unable to input values fully in a textbox field using send keys. Only partial values entered and no error shown.
Attempted sending keys with a wait.
Attempted sending keys slowly.
[Edit] - This is the entire script as requested in the comments below
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.wait import WebDriverWait
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
import time
from time import sleep
from selenium.webdriver.common.keys import
Keysdriver=webdriver.Chrome(executable_path="C:\Program Files (x86)\Drivers\chromedriver.exe")
driver.get("https://www.volunteers.ae/register.aspx")
driver.find_element_by_xpath("//*[#id='topnav_btnLangShift']").click()
driver.find_element(By.ID,'body_txtFName').send_keys("RADHIKA")
driver.find_element(By.ID,'body_txtLName').send_keys("PORANKI")
driver.find_element_by_id("body_txtEmail").send_keys("radhika.po#gmail.com")
element=driver.find_element_by_id("body_ddGender")
dropdown=Select(element)dropdown.select_by_value("Female")
driver.find_element_by_id("ctl00_body_txtDOB_dateInput").send_keys("08/04/1986")
element=driver.find_element_by_id("body_ddNationality")
dropdown=Select(element)
dropdown.select_by_value("India")
element=driver.find_element_by_id("body_ddCountryOfResidence")
dropdown=Select(element)
dropdown.select_by_value("+971")
element=driver.find_element_by_id("body_ddEmirate")
dropdown=Select(element)
dropdown.select_by_value("DUBAI")
time.sleep(10)
element=driver.find_element_by_id("body_ddCity")
dropdown=Select(element)
dropdown.select_by_value("DUBAI CITY")
def ClickAndSlowType(element, text):
element.click()
sleep(1) # let scripts run
for t in list(text):
print(t)
element.send_keys(t)
sleep (0.1)
mobile = WebDriverWait(driver, 8).until(ec.element_to_be_clickable((By.XPATH, "//*[#id='ctl00_body_txtEtisalat']")))
mobile.click()
mobile.send_keys("68862632")

What's the problem when you run your script?
I enhanced your script with a wait action and it works great:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('https://www.volunteers.ae/register.aspx')
mobile = WebDriverWait(driver, 8).until(EC.element_to_be_clickable((By.XPATH, "//*[#id='ctl00_body_txtEtisalat']")))
mobile.click()
mobile.send_keys("68862632")
Output looks like this:
There does seem to be some scripting at work in the page. Evertyime you click and type it resents the field. However, the above code still seems to work.
If there's a problem, let me know what's happening (with any relevant errors that occur) and i'll look again.
[Update!]
Based on the comments below, try typing with a delay.
Add this import:
from time import sleep
Add this function:
def ClickAndSlowType(element, text):
element.click()
sleep(1) # let scripts run
for t in list(text):
print(t)
element.send_keys(t)
sleep (0.1)
Run this:
mobile = WebDriverWait(driver, 8).until(EC.element_to_be_clickable((By.XPATH, "//*[#id='ctl00_body_txtEtisalat']")))
ClickAndSlowType(mobile, "68862632")
I can't recreate your issue but this also works for me so worth a try. It just types a bit slower in case of the funky in-page script issues.
Let me know how it goes.
If you can update your question with your entire i code i might be able to recreate and provide a tried and tested answer. (no one likes "it worked for me")
As it works locally but not on #Paul 's machine : we need to debug it . Try this at the end of the script and please tell exactly what you see happen to the field and what the output from the prints on the console:
def WaitForDocumentReadyState():
print("Ready State Before: " + driver.execute_script('return document.readyState'))
WebDriverWait(driver, 10).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')
def ClickAndSlowType(element, text):
element.click()
WaitForDocumentReadyState()
sleep(1) # let scripts run
for t in list(text):
element.send_keys(t)
WaitForDocumentReadyState()
sleep (0.5)
sleep(1) # let scripts run
WaitForDocumentReadyState()
mobile = WebDriverWait(driver, 8).until(ec.element_to_be_clickable((By.XPATH, "//*[#id='ctl00_body_txtEtisalat']")))
#mobile.click()## REMOVED
#mobile.send_keys("68862632") ## REMOVED
ClickAndSlowType(mobile, "68862632") ## using the function

StaleElementReferenceException in Python

I am trying to scrape data from the Sunshine List website (http://www.sunshinelist.ca/) using the Selenium package but I get the following error mentioned below. From several other related posts I understand that I need to use the WebDriverWait to explicitly ask the driver to wait/refresh but I am unable to identify where and how I should call the function.
Screenshot of Error
StaleElementReferenceException: Message: The element reference
of (tr class="even") stale: either the element is no longer attached to the DOM or the
page has been refreshed
import numpy as np
import pandas as pd
import requests
import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import StaleElementReferenceException
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
ffx_bin = FirefoxBinary(r'C:\Users\BhagatM\AppData\Local\Mozilla Firefox\firefox.exe')
ffx_caps = DesiredCapabilities.FIREFOX
ffx_caps['marionette'] = True
driver = webdriver.Firefox(capabilities=ffx_caps,firefox_binary=ffx_bin)
driver.get("http://www.sunshinelist.ca/")
driver.maximize_window()
tablewotags1=[]
while True:
divs = driver.find_element_by_id('datatable-disclosures')
divs1=divs.find_elements_by_tag_name('tbody')
for d1 in divs1:
div2=d1.find_elements_by_tag_name('tr')
for d2 in div2:
tablewotags1.append(d2.text)
try:
driver.find_element_by_link_text('Next →').click()
except NoSuchElementException:
break
year1=tablewotags1[0::10]
name1=tablewotags1[3::10]
position1=tablewotags1[4::10]
employer1=tablewotags1[1::10]
df1=pd.DataFrame({'Year':year1,'Name':name1,'Position':position1,'Employer':employer1})
df1.to_csv('Sunshine List-1.csv', index=False)

If your problem is to click the "Next" button, you can do that with the xpath:
driver = webdriver.Firefox(executable_path=r'/pathTo/geckodriver')
driver.get("http://www.sunshinelist.ca/")
wait = WebDriverWait(driver, 20)
el=wait.until(EC.presence_of_element_located((By.XPATH,"//ul[#class='pagination']/li[#class='next']/a[#href='#' and text()='Next → ']")))
el.click()

For each click on the "Next" button -- you should find that button and click on it.
Or do something like this:
max_attemps = 10
while True:
next = self.driver.find_element_by_css_selector(".next>a")
if next is not None:
break
else:
time.sleep(0.5)
max_attemps -= 1
if max_attemps == 0:
self.fail("Cannot find element.")
And after this code does click action.
PS: Also try to add just time.sleep(x) after fiding element and then do click action.

Try this code below.
When the element is no longer attached to the DOM and the StaleElementReferenceException is invoked, search for the element again to reference the element.
Please do note I checked with Chrome:
try:
driver.find_element_by_css_selector('div[id="datatable-disclosures_wrapper"] li[class="next"]>a').click()
except StaleElementReferenceException:
driver.find_element_by_css_selector('div[id="datatable-disclosures_wrapper"] li[class="next"]>a').click()
except NoSuchElementException:
break

>>>Stale Exceptions can be handled using **StaleElementReferenceException** to continue to execute the for loop. When you try to get the element by any find_element method in a for loop.
from selenium.common import exceptions
and customize your code of for loop as:
for loop starts:
try:
driver.find_elements_by_id("data") //method to find element
//your code
except exceptions.StaleElementReferenceException:
pass

When you raise the StaleElementException that means that somthing changed in the site, but not in the list you have. So the trick is to refresh that list every time, inside the loop like this:
while True:
driver.implicitly_wait(4)
for d1 in driver.find_element_by_id('datatable-disclosures').find_element_by_tag_name('tbody').find_elements_by_tag_name('tr'):
tablewotags1.append(d1.text)
try:
driver.switch_to.default_content()
driver.find_element_by_xpath('//*[#id="datatable-disclosures_wrapper"]/div[2]/div[2]/div/ul/li[7]/a').click()
except NoSuchElementException:
print('Don\'t be so cryptic about error messages, they are good\n
...Script broke clicking next') #jk aside put some info there
break
Hope this help you, cheers.
Edit:
So I went to the said website, the layout is pretty straight forward, but the structure repeats itself like four times. So when you go about crawling the site like that something is bound to change.
So I’ve edited the code to only scrap one tbody tree. This tree comes from the first datatable-disclousure. And added some waits.