I am trying to build a service where users can insert their Javascript code and it gets executed on a website of their choice. I use webdriver from python's selenium lib and chromedriver. The problem is that the python script gets stuck if user submits Javascript code with infinite loop.
The python script needs to process many tasks like: go to a website and execute some Javascript code. So, I can't afford to let it get stuck. Infinite loop in Javascript is known to cause a browser to freeze. But isn't there some way to set a timeout for webdriver's execute_script method? I would like to get back to python after a timeout and continue to run code after the execute_script command. Is this possible?
from selenium import webdriver
chromedriver = "C:\chromedriver\chromedriver.exe"
driver = webdriver.Chrome(chromedriver)
driver.get("http://www.bulletproofpasswords.org/") # Or any other website
driver.execute_script("while (1); // Javascript infinite loop causing freeze")
You could set a timeout for your driver.execute_script("while (1);") call.
I have found another post that could solve this issue.
Basically, if you are on a Unix system, you could use signal to set a timeout for your driver.execute_script("while (1); call.
Or if you could run it in a separate process and then end the process if it takes too long using multiprocessing.Process. I'm including the example that was given in the other post:
import multiprocessing
import time
# bar
def bar():
for i in range(100):
print "Tick"
time.sleep(1)
if __name__ == '__main__':
# Start bar as a process
p = multiprocessing.Process(target=bar)
p.start()
# Wait for 10 seconds or until process finishes
p.join(10)
# If thread is still active
if p.is_alive():
print "running... let's kill it..."
# Terminate
p.terminate()
p.join()
Related
What I am currently trying to do is the following. There are a number of changing values (js driven) in a website that I am monitoring and saving to a database using Selenium. The values are read through infinite loops, from elements found with selenium's find_element.
This works as intended with one process. However, when I try to multiprocess this (to monitor multiple values at the same time), there seems to be no way to do it without opening one separate browser for each process (unfeasible, since we are talking about close to 60 different elements).
The browser I open before multiprocessing seems to not be available from within the various processes. Even if I find the elements before the multiprocessing step, I cannot pass them to the process since the webelements can't be pickled.
Am I doing something wrong, is selenium not the tool for the job, or is there another way?
The code below doesn't actually work, it's just meant to show the structure of what I currently have as a "working version". What I need to get away from is opening the browser from within the function and have all my processes relying on a single browser.
import time
import datetime
import os
from selenium import webdriver
from multiprocessing import Pool
def sampling(value_ID):
dir = os.path.dirname(__file__)
driver = webdriver.Firefox(dir)
driver.get("https:\\website.org")
monitored_value = driver.find_element_by_xpath('value_ID')
while(1):
print(monitored_value.text)
time.sleep(0.1)
value_array = [1,2,3,4,5,6]
if __name__ == '__main__':
with Pool(6) as p:
p.map(getSampleRT, value_array)
You can checkout selenium abstract listeners if you want to capture the changes in elements. By implementing a listener you can get rid of infinite loops. Here is an example that i think it can work for you.
class EventListeners(AbstractEventListener):
def before_change_value_of(self, element, driver):
# check if this is the element you are looking for
# do your stuff
print("element changed!")
driver_with_listeners = EventFiringWebDriver(driver, EventListeners()
# wait as much as you like while your listeners are working
driver_with_listeners.implicitly_wait(20000)
Also you can checkout this post for more complete implementation.
I have a script that uses a selenium webdriver (geckodriver) and loads various webpages.
The scripts works fine at the beginning, but than at a random point it stops working without raising any error (the program sorts of hangs without really doing anything).
I added some logging statement to check when it hangs, and this is caused by the WebDriverWait statement (see below).
The last thing that is printed in the log is "get_records - Loaded".
The expected behavior to me would be to either print "get_records - Acquired pager", or to raise a TimeoutException after 10 seconds.
[...]
logging.info("get_records - Getting url: {}".format(url))
driver.get(url)
logging.info("get_records - Loaded")
# Get records number and result pages
elem = WebDriverWait(driver, 10).until(ec.element_to_be_clickable(
(By.XPATH, "//td[#align='right']/span[#class='pager']"))
)
logging.info("get_records - Acquired pager")
[...]
Python version: 3.7.3
Selenium version: 3.141.0
Firefox version: 70.0.1
It seems like a similar bug happened with previous version (Selenium WebDriver (2.25) Timeout Not Working), but that bug was closed.
Is anyone having the same issue?
Update:
It seems like adding time.sleep(0.5) before elem prevents the script from freezing (either "get_records - Acquired pager" is printed, or the timeoutException is raised).
Even though this is a turnaround for the issue, I would rather not put any forced wait.
I actually have the exactly same experience when the script works fine at first but hangs forever after some time. The '10 seconds' timeout is that webdriver/browser tries to open a page in 10 seconds. But the timeout that python script sends request to webdriver/browser is not defined. And it's none by default meaning request will wait infinitely.
Short answer:
driver.command_executor.set_timeout(10)
driver.get(url)
Explain:
Chromedriver as example. Whenever you run a selenium script. A process named 'chromedriver' starts as well. Let's call it 'control process'. It opens the browser and controls it. And it also acts as a http server which you can get the address and port by driver.command_executor._url. It receives http request, processes it, tells the browser to do something(maybe open a url) and returns. Details here.
When you call
elem = WebDriverWait(driver, 10).until(ec.element_to_be_clickable(
(By.XPATH, "//td[#align='right']/span[#class='pager']"))
)
you are actually sending a request to the 'control process' which is a http server and tell it to do something(find some elements in current page). The timeout '10' means that 'control process' tells browser to open a page in 10 seconds before it cancels and returns timeout status to the python script.
But what really happens here is the 'control process' is receiving request but not responding. I don't really know what's happening in the 'control process'.
Python selenium package is using urllib3.request to send request and socket._GLOBAL_DEFAULT_TIMEOUT as timeout. It is none by default that makes a request wait infinitely. So you can set it by using driver.command_executor.set_timeout(10). Now if 'control process' brokes you will get a timeout exception and maybe recreate webdriver.
I've made a bot that uses the Selenium Webdriver Module to navigate a website. Unfortunately I've noticed that the script will sometime stop when it tries to click a button. The code is simple. I try to click the button, if I can't we wait a second and try again. It works, but at seemingly random times (sometimes after 10 minutes, sometimes after several hours) it just stops after clicking the button.
while 1:
try:
#Try to click the button
confirmButton = driver.find_element_by_name("confirm")
confirmButton.click()
#If we can't, wait a second and try again
except:
time.sleep(1)
I've been thinking about creating some way to detect this, thus being able to timeout the current attempt to click, but I can't seem to figure out how. The script of single threaded and I can't use the simple datetime technique, as it would never run that check, since it's still waiting for the button to have finished being clicked on.
Edit: Someone asked me how I knew it was hanging and not just retrying indefinitely. I did a test where I printed a number for each line that was executed and when I the hang occurred it would not execute any of he lines below confirmButton.click(). I think that is proof that it's hanging and not retrying indefinitely. Or maybe not?
Your problem can be solved with a timeout: launch a function in a separate thread and stops after a certain time limit if the function is not finished.
Here is the example
from threading import Thread
from time import sleep
def threaded_function():
confirmButton = driver.find_element_by_name("confirm")
confirmButton.click()
if __name__ == "__main__":
thread = Thread(target = threaded_function)
thread.start()
thread.join(1)# this means the thread stops after 1 second, even if it is not finished yet
print ("thread finished...exiting")
Hope that helps, and tell me if it does not solve the prob.
I am working on selenium with python for downloading file from a url.
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference('browser.download.folderList', 2) # custom location
profile.set_preference('browser.download.manager.showWhenStarting', False)
profile.set_preference('browser.download.dir', '/tmp')
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'text/csv')
browser = webdriver.Firefox(profile)
try:
browser.get("http://www.drugcite.com/?q=ACTIMMUNE")
browser.find_element
browser.find_element_by_id('exportpt').click()
browser.find_element_by_id('exporthlgt').click()
except:
pass
I want to set timeout for this program. Means, If within 60 seconds if this url is not loaded due to net issue, it should retry after each 60 seconds and after 3 tries, it should go ahead.
How can I achieve such in this code?
Thanks
You could use browser.implicitly_wait(60)
WebDriver.implicitly_wait
There is nothing built in to do this. However, I wouldn't have said it would be too hard.
Just use an explicit wait to find a particular element that should be there when the page loads. Set the timeout to be 60 seconds on this explicit wait.
Wrap this in a loop that executes up to three times. To avoid it running three times unnecessarily, put in a break statement when the explicit wait actually runs without any issue.
That means it'll run up to three times, waiting 60 seconds a time, and once it's successful it'll exit the loop. If it isn't successful after all of that, then it'll crash.
Note: I've not actually tried this but it's just a logical solution!
Updating with more context: Selenium 1 had a command called "setSpeed." This allowed the execution of each command to be slowed down by X milliseconds. The team behind Selenium 2 (Webdriver) decided to deprecate this command and now there is no way to slow down the tests to run at speeds where it's easy to visually monitor the App during execution. I've read the developers' explanation as to why they deprecated it, as well as the suggested workarounds like using implicit_waits, but that doesn't solve the issue for me (or the other people complaining about the deprecation). That said, I was hoping to work around this by setting a global execution speed that would be applicable to either each method in unittest, or the entire suite of tests.
Original Question: I have different unit tests that I'd like to execute using different delays between commands. I know that I can keep copying and pasting time.sleep between the commands, but surely there is a way to just set a universal sleep that will be run before each command in the specified method?
def test_x_test(self):
driver = self.driver
time.sleep(2)
print("running the First selenium command such as click button")
time.sleep(2)
print("running another Selenium command such as click link ")
time.sleep(2)
self.driver.quit()
if __name__ == '__main__':
unittest.main()
Ahh now the answer is so obvious.
Create a helper method that controls webdriver actions and before it executes the action put in a pause:
The below is going to be pseudocode-ish as I no longer have access to a Python IDE at work
#passing in Webdriver instance and the command we want to execute into our helper method
webdriverHelper(driver, command):
#this 2 second sleep will get run each time
time.sleep(2)
if command == "click":
driver.getElement.click()
elif command== "getText":
driver.getElement.getText()
etc...............