I realize this is a relatively simple question but I haven't found the answer yet.
I'm using driver.get() in a for loop that iterates through some urls. To help avoid my IP address getting blocked, I've implemented time.sleep(5) before the driver.get statement in the for loop.
Basically, I just want a wait period to make my scraping seem more natural.
I think time.sleep may be causing page crashes. What is the equivalent of time.sleep in selenium? From what I understand, implicitly_wait just sets the amount of time before throwing an exception, but I'm not sure that's what I want here? I want a specific amount of time for the driver to wait.
time.sleep()
The sleep() function is from the time module which suspends execution of the current thread for a given number of seconds.
Now, WebDriver being a out-of-process library which instructs the browser what to perform and at the same time the web browser being asynchronous in nature, WebDriver can't track the active, real-time state of the HTML DOM. This gives rise to some intermittent issues that arise from usage of Selenium and WebDriver those are subjected to race conditions that occur between the browser and the user’s instructions.
As of now Selenium doesn't have any identical method to time.sleep(), however there are two equavalent methods at your disposal and can be used as per the prevailing condition of your automated tests.
Implicit wait: In this case, WebDriver polls the DOM for a certain duration when trying to find any element. This can be useful when certain elements on the webpage are not available immediately and need some time to load.
def implicitly_wait(self, time_to_wait) -> None:
"""
Sets a sticky timeout to implicitly wait for an element to be found,
or a command to complete. This method only needs to be called one
time per session. To set the timeout for calls to
execute_async_script, see set_script_timeout.
:Args:
- time_to_wait: Amount of time to wait (in seconds)
:Usage:
::
driver.implicitly_wait(30)
"""
self.execute(Command.SET_TIMEOUTS, {
'implicit': int(float(time_to_wait) * 1000)})
Explicit wait: This type of wait allows your code to halt program execution, or freeze the thread, until the condition you pass it resolves. As an example:
presence_of_element_located()
visibility_of_element_located()
element_to_be_clickable()
There is no specific method in Selenium for hardcoded pauses like a time.sleep() general Python method.
As you mentioned there is an implicitly_wait and Expected Conditions explicit WebDriverWait waits but both these are NOT a hardcoded pauses.
Both the implicitly_wait WebDriverWait are used for setting the timeout - how long time to poll for some element presence or condition, so if that condition is fulfilled or the element is presented the program flow will immediately continue to the next code line.
So, if you want to put a pause you have to use some general Python method that will suspend the program / thread run like the time.sleep().
Related
This feature is available in Puppeteer which allows passing 0 as timeout which is infinite. But the same is not available in Selenium. Is there any custom function that you guys know?
P.S. I need this function because I want to test something on my server and the timeout can be variable like sometimes it can take the 30s or sometimes 30m.
There is nothing infinite wait kind of concept in Selenium. Because this is not the ideal case to wait something for infinite time (1Minute, 1Hour, 1Day, 1Month, 1Year... and so on).
The certain condition should be fulfill within specific time. So one thing you can do is provide maximum threshold wait time. Selenium will continuously check whether your element be clickable until maximum threshold value reached.
As per Selenium documentation:
ExplicitWait allow your code to halt program execution, or freeze the
thread, until the condition you pass it resolves. The condition is
called with a certain frequency until the timeout of the wait is
elapsed. This means that for as long as the condition returns a falsy
value, it will keep trying and waiting.
Yes, I know both are used to wait for some specified time.
Selenium:
driver.implicitly_wait(10)
Python:
import time
time.sleep(10)
Is there any difference between these two?
time.sleep(secs)
time.sleep(secs) suspends the execution of the current thread for the given number of seconds. The argument may be a floating point number to indicate a more precise sleep time. The actual suspension time may be less than that requested because any caught signal will terminate the sleep() following execution of that signal’s catching routine. Also, the suspension time may be longer than requested by an arbitrary amount because of the scheduling of other activity in the system.
You can find a detailed discussion in How to sleep webdriver in python for milliseconds
implicitly_wait(time_to_wait)
implicitly_wait(time_to_wait) is to specify the amount of time the WebDriver instance i.e. the driver should wait when searching for an element if it is not immediately present in the HTML DOM in-terms of SECONDS when trying to find an element or elements if they are not immediately available. The default setting is 0 which means the driver when finds an instruction to find an element or elements, the search starts and results are available on immediate basis.
In this case, after a fresh loading of a webpage an element or elements may be / may not be found on an immediate search. So your Automation Script may be facing any of these exceptions:
NoSuchElementException
TimeoutException
ElementNotVisibleException
ElementNotSelectableException
ElementClickInterceptedException
ElementNotInteractableException
Hence we introduce ImplicitWait. By introducing ImplicitWait the driver will poll the DOM Tree until the element has been found for the configured amount of time looking out for the element or elements before throwing a NoSuchElementException. By that time the element or elements for which you had been looking for may be available in the HTML DOM. As in your code you have already set ImplicitWait to a value of 10 seconds, the driver will poll the HTML DOM for 10 seconds.
You can find a detailed discussion in Using implicit wait in selenium
time.sleep(10) pauses code execution exactly 10 seconds.
driver.implicitly_wait(10) waits maximum 10 seconds for element's presence. If it is found after 2 seconds then code execution will be continued without wait for more 8 seconds.
When we use implicit wait in test script it is declared globally and it will automatically get applied to all the elements on that script and for example in java if you use implicit wait. --> driver. manage().timeouts().implictwait(10,timeunit.seconds);. this code will wait for the element to be present in DOM until then it will wait once element gets visible execution will get continue. during the time of hold script execution is stopped.
In thread.sleep(1000) in this case script will get hold for 1000ms no matter if element gets visible on dom at 500ms it will stay at this point till 1000 ms.
Thread.sleep() is a static wait which holds script duration for fixed number of time. Where as implicit wait will hld the script execution until element gets visible in DOM.
Hope this helps!
I run selenium in python to do web testing, and I have noticed that, when I have added a longer wait in python, my selenium session is logged-out after a certain point of time.
Below Line of code I use to wait in the code
time.sleep(420)
I have tried to do some fake click during the wait period, but still, I have seen the security logout. is there any approach I can solve this issue?
I use python 3.5 and firefox web driver for testing.
What if you try to insert also a wait to Selenium.
In Java it looks like: SeleniumUtils.sleepQuietly(420);
There should also be a method for Python how to tell Selenium to wait (sleep) for certain time.
In your case, you have to say Selenium to wait, after you have called "the function" or click the button that needs to long (6 minutes) to execute.
when you use sleep() function you cannot do anything in that time. that is very useful between two lines of code where you need some time for processing like you are downloading file so, wait 5 seconds after 5 seconds add new parameters and download another file something like that.
you can surely do clicks without having sleep() function. between to click use sleep of 5 to 10 seconds like this your session will not expire it will run until you do driver.quit().
Yes I know the question has been asked quite often but I still don't get it. I want to make Selenium wait, no matter what. I tried these methods
driver.set_page_load_timeout(30)
driver.implicitly_wait(90)
WebDriverWait(driver, 10)
driver.set_script_timeout(30)
and other things but it does not work. I need selenium to wait 10 seconds. NO not until some element is loaded or whatever, just wait 10 seconds. I know there is this
try:
element_present = EC.presence_of_element_located((By.ID, 'whatever'))
WebDriverWait(driver, timeout).until(element_present)
except TimeoutException:
print "Timed out waiting for page to load"
I do not want that.
If waiting for some seconds is to much (not achievable) for selenium, what other (python) library's/programs would be capable to achieve this task? With Javas Selenium it does not seem to be a problem...
All the APIs you have mentioned is basically a timeout, so it's gonna wait until either some event happens or maximum time reached.
set_page_load_timeout - Sets the amount of time to wait for a page load to complete before throwing an error. If the timeout is negative, page loads can be indefinite.
implicitly_wait - Specifies the amount of time the driver should wait when searching for an element if it is not immediately present.
set_script_timeout - Sets the amount of time to wait for an asynchronous script to finish execution before throwing an error. If the timeout is negative, then the script will be allowed to run indefinitely.
For more information please visit this page. (documention is for JAVA binding, but functionality should be same for all the bindings)
So, if you want to wait selenium (or any script) 10 seconds, or whatever time. Then the best thing is to put that thread to sleep.
In python it would be
import time
time.sleep(10)
The simplest way to do this in Java is using
try {
Thread.sleep(10*1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
I have the following code using Selenium in Python 3:
profile = webdriver.FirefoxProfile()
profile.set_preference('webdriver.load.strategy', 'unstable')
browser = webdriver.Firefox(profile)
browser.set_page_load_timeout(10)
url = 'my_url'
while True:
try:
st = time.time()
browser.get(url)
print('Finished get!')
time.sleep(2)
wait = WebDriverWait(browser, 10)
element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'div[my_attr="my_attr"]')))
print('Success after {} seconds.'.format(round(time.time()-st)))
break
except:
print('Timed out after {} seconds.'.format(round(time.time()-st)))
print('Reloading')
continue
From my understanding, using the explicit wait here (even with the unstable load strategy and page load timeout), what should happen is that the page should load, it should look for the element specified, and if either the page doesn't load within 10 seconds or the element is not found within 10ish seconds, it should time out and reload again (because of the try/except clause with the while loop).
However, what I'm finding is that it's not timing out consistently. For example, I've had instances where the loading times out after 10ish seconds the first time around, but once it reloads, it doesn't time out and instead "succeeds" after like 140 seconds. Or sometimes it doesn't time out at all and just keeps running until it succeeds. Because of the unstable load strategy, I don't think the page load itself is ever timing out (more specifically, the 'Finished get!' message always prints). But the explicit wait here that I specified also does not seem to be consistent. Is there something in my code that is overriding the timeouts? I want the timeouts to be consistent such that if either the page doesn't load or the element isn't located within 10ish seconds, I want it to timeout and reload. I don't ever want it to go on for 100+ seconds, even if it succeeds.
Note that I'm using the unstable webdriver load strategy here because the page I'm going to takes forever to completely load so I want to go straight through the code once the elements I need are found without needing the entire page to finish loading.
After some more testing, I have located the source of the problem. It's not that the waits are not working. The problem is that all the time is being taken up by the locator. I discovered this by essentially writing my own wait function and using the .find_element_by_css_selector() method, which is where all the runtime is occurring when it takes 100+ seconds. Because of the nature of my locator and the complexity of the page source, it's taking 100+ seconds sometimes for the locator to find the element when the page is nearly fully loaded. The locator time is not factored into the wait time. I presume that the only "solution" to this is to write a more efficient locator.