We Keep Coding

How to loop code until it works in Python?

I have a long code, here is a small excerpt from it. You constantly need to find elements, insert data and click buttons. Sometimes something fails to load and errors pop up. Is it possible to have Python try these commands until it succeeds? I can't use
time.sleep()
with a long delay as it will greatly increase the execution time and even that doesn't always help(
start = driver.find_element(By.CSS_SELECTOR, "SELECTOR")
start.click()
time.sleep(1)
start2 = driver.find_element(By.CSS_SELECTOR, "SELECTOR")
start2.click()
time.sleep(1)

If you use a method that waits for an element to be clickable first, then you can set a default timeout, and then use that, as shown below with this partial code example that uses WebDriverWait:
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
def wait_for_element_clickable(
driver, selector, by="css selector", timeout=10
):
try:
return WebDriverWait(driver, timeout).until(
EC.element_to_be_clickable((by, selector))
)
except Exception:
raise Exception(
"Element {%s} was not visible/clickable after %s seconds!"
% (selector, timeout)
)
# ...
element = wait_for_element_clickable(driver, "button#id")
element.click()

Why does my second explicit wait not work if an action was performed on the previous one?

I would like to switch up to two toggles on a website that expand or hide parts of a CMS tree (expanded picture for context) that does not have an API. They are on the same hierarchy level of the tree. The toggles are randomly set on the "hidden" or "expanded" view and I need the first one to be set to hidden and the second to be expanded.
Generally, my code works:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
try:
iframe_cms_subpages = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.XPATH, '/html/body/table/tbody/tr[2]/td[1]/iframe'))) # (unfortunately, I have to go with an abs xpath here)
print("iframe found!")
except TimeoutException:
print("Loading iframe took too long")
driver.switch_to.frame(iframe_cms_subpages)
try:
main_menu1 = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.XPATH, "//a[#class='npsLogic_nodeViewEntry_c_/main_menu1_npsLogic']/img[#alt='-']")))
print("main_menu1 open! Close it first")
main_menu1.click()
print("closed!")
except TimeoutException:
print("main_menu1 was either already closed or something went wrong.")
try:
main_menu2 = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.XPATH, "//a[#class='npsLogic_nodeViewEntry_e_/main_menu2_npsLogic']/img[#alt='+']")))
print("main_menu2 closed! Needs to be opened first")
main_menu2.click()
print("main_menu2 opened!")
except TimeoutException:
print("main_menu2 was either already open or something went wrong."
The "+" in the xpath is present if it is closed (hidden), for the "-" it is vice versa (expanded).
My issue: I have tested this in all four scenarios and the code only works if up to one of the two toggles has to be switched to have both be set correctly, i.e. if either both of them are open, both of them are closed or the first one is closed and second one open (correct settings right away). If both of them need to be switched, i.e. if the first one is open and the second one is closed, the second try statement throws a TimeOutException. Why does the second explicit wait not work if an action was performed based on the previous one?
Notes: I included the iframe because I thought it might be relevant; that explicit wait works. I noticed that there is a "c" in the first xpath and an "e" in the second, but that is part of the path for some reason, not a typo.
Do I perhaps have an incorrect understanding of how explicit waits work?

Every time one of the elements is clicked, the elements were not recognized by the webdriver anymore. This can be fixed by just switching back to parent frame and then back into the iframe again! I have also changed the explicit wait type to element_to_be_clickable in line with the recommendation of Greg:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
try:
iframe_cms_subpages = WebDriverWait(driver, delay).until(EC.element_to_be_clickable((By.XPATH, '/html/body/table/tbody/tr[2]/td[1]/iframe'))) # (unfortunately, I have to go with an abs xpath here)
print("iframe found!")
except TimeoutException:
print("Loading iframe took too long")
driver.switch_to.frame(iframe_cms_subpages)
try:
main_menu1 = WebDriverWait(driver, delay).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='npsLogic_nodeViewEntry_c_/main_menu1_npsLogic']/img[#alt='-']")))
print("main_menu1 open! Close it first")
main_menu1.click()
print("closed!")
except TimeoutException:
print("main_menu1 was either already closed or something went wrong.")
driver.switch_to.parent_frame()
try:
iframe_cms_subpages = WebDriverWait(driver, delay).until(EC.element_to_be_clickable((By.XPATH, '/html/body/table/tbody/tr[2]/td[1]/iframe')))
print("iframe found!")
except TimeoutException:
print("Loading took too much time or the window was open.")
driver.switch_to.frame(iframe_cms_subpages)
try:
main_menu2 = WebDriverWait(driver, delay).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='npsLogic_nodeViewEntry_e_/main_menu2_npsLogic']/img[#alt='+']")))
print("main_menu2 closed! Needs to be opened first")
main_menu2.click()
print("main_menu2 opened!")
except TimeoutException:
print("main_menu2 was either already open or something went wrong."

Python Selenium TimeoutException error even with wait until clickable and except TimeoutException:

My selenium webdriver been constantly crashing due to
TimeoutException: Message: timeout: Timed out receiving message from
renderer: 298.972
The cookies pop up opens up but the script doesn't click on it,
in like 20 driver.get(url), 19 times it will accept cookies but the 20th will fail to accept cookies, although the window has opened up, I tried to use the code below but still fails.
retries = 1
while retries <= 5:
try:
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#class="coi-banner__accept"]'))) #wait until cookies clickable
element.click()
break
except TimeoutException:
driver.refresh()
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#class="coi-banner__accept"]'))) #wait until cookies clickable
element.click()
retries += 1

I ran the below script more than 20 times and still it was able to click on the desired button every single time.
All I had to do was basically to change the locator to CSS from XPath:
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get("https://www.novasol.com/")
try:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[onclick='CookieInformation.submitAllCategories();']"))).click()
print('Clicked it')
except:
print('Either element was not found, or Bot could not click on it.')
pass
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Updated:
driver.get("https://www.novasol.com/")
def retry_click(number_of_retries, wait_before_performing_click):
while number_of_retries > 0:
time.sleep(wait_before_performing_click)
try:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[onclick='CookieInformation.submitAllCategories();']"))).click()
break
except:
pass
number_of_retries = number_of_retries - 1
try:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[onclick='CookieInformation.submitAllCategories();']"))).click()
print('Clicked it')
except:
print('Either element was not found, or Bot could not click on it.')
driver.refresh()
retry_click(20, 10)
pass

Try to use driver.execute_script() instead of element.click()
htmlElement = driver.find_element_by_xpath('//*[#class="coi-banner__accept"]')
driver.execute_script("arguments[0].click();", htmlElement)

Generally,
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#class="coi-banner__accept"]'))) #wait until cookies clickable
element.click()
should work. From the code you shared I can't see why it works in 95% cases but fails in 5% case.
What I do see:
You are trying to find exactly the same condition in except block that caused the TimeoutException.
So, if somehow Selenium could not find element matching this locator //*[#class="coi-banner__accept"] to be clickable and thrown an TimeoutException waiting for the same condition in except will give you the same result....
Also, I see no sense to put this in a loop of 5 attempts.
In case element was found, clicked and closed - no sense to try again.
And in case you could not do that in the first attempt - this will throw you TimeoutException exception on the first attempt, you will never continue to the second attempt here...

How to check the element is displayed on the next page or not?

I want to check whether the Google submit button is still displayed or not on the next page. If it is not displayed then extract some text or else close the browser.
I have tried using the WebDriverWait element with less seconds so that I can get the result in a faster way. But it still takes 3–4 minutes to throw the exception.
driver.get("https://www.google.com/")
driver.maximize_window()
driver.find_element_by_name("q").send_keys("New york")
driver.find_element_by_name("q").send_keys(Keys.ENTER)
sleep(2)
print("System in 2 seconds sleep mode.")
before_Btn_Check_time = datetime.now()
print(before_Btn_Check_time)
wait = WebDriverWait(driver, 0)
try:
ui.WebDriverWait(driver, 0).until(ec.presence_of_element_located(
driver.find_element_by_xpath(
"//*[#id='tsf']/div[2]/div/div[3]/center/input[1]"
)
))
sleep(1)
print("System in 1 seconds sleep mode.")
print("Still on the same page..!")
except NoSuchElementException as e:
print("No Google Search button was found" + str(e))
The expected output is to get an error message in less than 10 seconds if the button is not displayed on the next page.

Using implicitly_wait and WebDriverWait can result in unexpected behavior.
You set WebDriverWait timeout to 0, but driver.find_element_by_xpath uses the implicitly_wait timeout. The element will be searched up to this timeout.
Actually, presence_of_element_located() should receive By as parameter, not WebElement. The correct use is
WebDriverWait(driver, 0).until(ec.presence_of_element_located((By.XPATH, "//*[#id='tsf']/div[2]/div/div[3]/center/input[1]")))
But the same reason apply, it will search for the element internally.
As a side note, implicitly_wait won't effect driver.get(), use driver.set_page_load_timeout().

Telling selenium to stop blocking after expected condition

I am interested in one element, let's call it
<div class="ofInterest" some-attr="dataIReallyWant"></div>
When I switch off js in firefox, this element does not exist. With javascript it does. I could not tell how it was being generated but my guess is that there is an ajax call which returns a js file which executes this javascript.
I am using selenium but it is very slow. I want to tell Selenium this:
Wait for this element to load, i.e something like EC.visibility_of_element_located((By.CSS, '.ofInterest'))
once you detect said element, stop blocking the code and don't download any further so don't waste my bandwidth

Simply wait for the element to exist in the DOM, then either quit/close the browser or execute some JavaScript to stop the page from loading:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
element = WebDriverWait(driver, 10).until( #10 second timeout.
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
finally:
driver.quit()
# OR
#driver.execute_script("window.stop();")
More information can be found here.

Explicit waits were made exactly for what you are describing:
An explicit waits is code you define to wait for a certain condition
to occur before proceeding further in the code. The worst case of this
is time.sleep(), which sets the condition to an exact time period to
wait.
In the worst case scenario, you would wait X amount of seconds that you've passed to the WebDriverWait, 10 seconds in this case:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
But, if the element is found earlier, it would give you the element and stop blocking the execution. By default, it checks for the expected condition every 500ms.
FYI, under-the-hood, it is just a while True: loop:
def until(self, method, message=''):
"""Calls the method provided with the driver as an argument until the \
return value is not False."""
screen = None
stacktrace = None
end_time = time.time() + self._timeout
while True:
try:
value = method(self._driver)
if value:
return value
except self._ignored_exceptions as exc:
screen = getattr(exc, 'screen', None)
stacktrace = getattr(exc, 'stacktrace', None)
time.sleep(self._poll)
if time.time() > end_time:
break
raise TimeoutException(message, screen, stacktrace)