I am working on Crawl project using Selenium and Webdriver. Since that data I need to crawl while big, I want to split it to 2 threads and run at the same time. But when I start 2 Webdrivers at the same time, my code can not recognize which driver belong to which thread to fill the information.
This is the code that I setup 2 threads run main function:
if __name__ == '__main__':
data = load_data(INPUT_DIR)
t1_data = data[:250]
t2_data = data[250:]
try:
_thread.start_new_thread(main, (t1_data, LOG_FILE_T1))
_thread.start_new_thread(main, (t2_data, LOG_FILE_T2))
except:
print ("Error: unable to start thread")
while 1:
pass
This code I start the Webdriver:
def start_driver():
global driver
options = Options()
options.add_argument("--disable-notification")
options.add_argument("--disable-infobars")
options.add_argument("--mute-audio")
#options.add_argument("headless")
driver = webdriver.Chrome(options=options)
Two Webdrivers after started, I will fill in the username/password
information on facebook.com
def login(email, password):
""" Logging into our own profile """
try:
driver.get('https://mbasic.facebook.com')
time.sleep(DELAY_TIME)
driver.find_element_by_name('email').send_keys(email)
driver.find_element_by_name('pass').send_keys(password)
driver.find_element_by_name('login').click()
# deal with "Not Now" button if it show off at first time
not_now_button = driver.find_element_by_xpath("//a")
if not_now_button.size != 0:
not_now_button.click()
except Exception as e:
print('Error in Login')
print(e)
exit()
At send_keys step, both threads fill in the same text box in 1 Webdriver.
How can I change my code for 2 threads can see different Webdrive and fill the info in ?
Just found out the solution, I want to share here if someone needed.
Instead of global driver, I will change to a local driver and pass it to each function.
def start_driver():
options = Options()
options.add_argument("--disable-notification")
options.add_argument("--disable-infobars")
options.add_argument("--mute-audio")
#options.add_argument("headless")
driver = webdriver.Chrome(options=options)
return driver
def login(driver, email, password):
""" Logging into our own profile """
try:
driver.get('https://mbasic.facebook.com')
time.sleep(DELAY_TIME)
driver.find_element_by_name('email').send_keys(email)
driver.find_element_by_name('pass').send_keys(password)
driver.find_element_by_name('login').click()
# deal with "Not Now" button if it show off at first time
not_now_button = driver.find_element_by_xpath("//a")
if not_now_button.size != 0:
not_now_button.click()
except Exception as e:
print('Error in Login')
print(e)
exit()
By this, 2 threads can see different Webdriver and fill their own information.
Related
I launch multiple selenium sessions with multiprocessing and want to close them all or a few when user wants it
schema of my code:
from multiprocessing import Process
def drop():
...
driver = webdriver.Chrome(executable_path=os.path.join(path, '\\chromedriver.exe'),chrome_options=options)
...
for i in range(10):
s=Process(target = drop)
s.start()
input('press enter to stop all sessions')
###close selenium driver event
you can create an array process_list = [] and append the 's' variable in the list for every iteration process_list.append(s) and then to stop all processes
for i in process_list:
driver.close()
i.terminate()
Adding each driver instance to a list when creating them and then iterating through them one by one to close them afterwards might be able to solve your issue.
from multiprocessing import Process
list_drivers = []
def drop():
...
driver = webdriver.Chrome(executable_path=os.path.join(path, '\\chromedriver.exe'),chrome_options=options)
list_drivers.append(driver)
...
for i in range(10):
s=Process(target = drop)
s.start()
input('press enter to stop all sessions')
for driver in list_drivers:
driver.quit()
###close selenium driver event
I want to check if a browser opened with Selenium is still open. I would need this check for closing a program. If the browser is not open, then the GUI should be destroyed without a message (tk.messagebox) coming. But if the browser is open, then the message should come as soon as the function is activated. Here is the function:
def close():
msg_box = tk.messagebox.askquestion('Exit Application', 'Are you sure you want to exit the application?',
icon='warning')
if msg_box == 'yes':
root.destroy()
try:
web.close()
except NameError:
sys.exit(1)
except InvalidSessionIdException:
sys.exit(1)
except WebDriverException:
sys.exit(1)
else:
return
I don't think there is a direct api for checking browser status. But you can use the the work around
def isBrowserAlive(driver):
try:
driver.current_url
# or driver.title
return True
except:
return False
Unfortunately, there's not a standardized way to check a driver's status.
Here's the most reliable cross-browser workaround that I've come up with over the years:
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import WebDriverException
def is_webdriver_alive(driver):
print('Checking whether the driver is alive')
try:
assert(driver.service.process.poll() == None) #Returns an int if dead and None if alive
driver.service.assert_process_still_running() #Throws a WebDriverException if dead
driver.find_element_by_tag_name('html') #Throws a NoSuchElementException if dead
print('The driver appears to be alive')
return True
except (NoSuchElementException, WebDriverException, AssertionError):
print('The driver appears to be dead')
return False
except Exception as ex:
print('Encountered an unexpected exception type ({}) while checking the driver status'.format(type(ex)))
return False
Usage:
driver_is_alive = is_webdriver_alive(driver)
I want to run multiple chrome instances with selenium. I tried to loop the webdrivers but selenium keeps shutting the instances down.
Here is the code:
from selenium import webdriver
user = str(input("Do you want to run this program? "))
amount = 0
if user == "yes":
amount = int(input("How many instances do you want to run? "))
for w in range(1, amount+1):
webdriver.Chrome("path of my driver")
elif user == "no":
print("Program is closing...")
else:
print("Invalid input")
The weird thing is that the instances wont close if i write them without a loop:
from selenium import webdriver
user = str(input("Do you want to run this program? "))
if user == "yes":
driver1 = webdriver.Chrome("path of driver")
driver2 = webdriver.Chrome("path of driver")
driver3 = webdriver.Chrome("path of driver")
driver4 = webdriver.Chrome("path of driver")
driver5 = webdriver.Chrome("path of driver")
elif user == "no":
print("Program is closing...")
else:
print("Invalid input")
Is there any solution for my problem?
To close the instances when you write it without a loop then do the following.
driver.close() - It closes the the browser window on which the focus
is set.
driver.quit() – It basically calls driver.dispose method which in turn
closes all the browser windows and ends the WebDriver session
gracefully.
You should use driver.quit whenever you want to end the program. It will close all opened browser window and terminates the WebDriver session.
Consider writing your loop example in the following way, this will handle shutting the instances much better.
Using either multithreading or multiprocessing or subprocess module to trigger the task in parallel (near parallel).
Multithreading example
from selenium import webdriver
import threading
import time
def test_logic():
driver = webdriver.Firefox()
url = 'https://www.google.co.in'
driver.get(url)
# Implement your test logic
time.sleep(2)
driver.quit()
N = 5 # Number of browsers to spawn
thread_list = list()
# Start test
for i in range(N):
t = threading.Thread(name='Test {}'.format(i), target=test_logic)
t.start()
time.sleep(1)
print t.name + ' started!'
thread_list.append(t)
# Wait for all thre<ads to complete
for thread in thread_list:
thread.join()
print 'Test completed!'
Here, spawning 5 browsers to run test cases at one time. Instead of implementing the test logic I have put sleep time of 2 seconds for the purpose of demonstration. The code will fire up 5 firefox browsers (tested with python 2.7), open google and wait for 2 seconds before quitting.
Logs:
C:\Python27\python.exe C:/Users/swadchan/Documents/TestPyCharm/stackoverflow/so49617485.py
Test 0 started!
Test 1 started!
Test 2 started!
Test 3 started!
Test 4 started!
Test completed!
Process finished with exit code 0
I have a stdout error message that occurs after running my python selenium test using geckodriver (firefox) but doesn't occur when using chrome. I can't figure out where the error line is coming from. It happens after the call to sys.exit() so somewhere during the cleanup process.
[Errno 2] No such file or directory: '/var/folders/ml/tbnznl592397p4h1svx4t5xr0000gr/T/tmpr8ypd0y1'
MacOS 10.14.5 (Mojave)
Driver info: firefox=67.0
Session info: firefox=0.24.0
Selenium 3.141.0
Python 3.7.3
Here is the driver initialization for both firefox and chrome.
def createDriver(self, browser: str, headless: bool = False, performancelog=False,
consolelog=False, verbose: bool = False):
## DRIVERS
#chrome
# https://chromedriver.storage.googleapis.com/index.html
#firefox
# https://github.com/mozilla/geckodriver/releases
# 20.1 has a bug where headless doesn't work
# 19 has a bug where it closes a frame?
# safari
# edge
# https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/
logger.trace("createDriver()")
if browser.upper() == 'FIREFOX':
## setting log_path to /dev/null will prevent geckodriver from creating it's own log file.
## if we enable root logging, we can capture the logging from geckodriver, ourselves.
options = webdriver.FirefoxOptions()
profile = webdriver.FirefoxProfile()
profile.set_preference('app.update.auto', False)
profile.set_preference('app.update.enabled', False)
# options.binary_location = binary ## use this only if we want to move the executable outside the available path
if headless:
options.headless = True
caps = DesiredCapabilities.FIREFOX
caps['unhandledPromptBehavior'] = "ignore"
driver = webdriver.Firefox(profile, log_path='/dev/null', firefox_options=options,
desired_capabilities=caps
)
logger.debug(f"Driver info: firefox={driver.capabilities['browserVersion']}")
logger.debug(f"Session info: firefox={driver.capabilities['moz:geckodriverVersion']}")
if driver.capabilities['moz:geckodriverVersion'] == '0.20.1':
if headless:
raise Exception("Headless mode doesn't work in Gecko Driver 0.20.1")
elif browser.upper() == 'CHROME':
options = webdriver.ChromeOptions()
options.add_argument("--disable-extensions")
options.add_argument("--allow-running-insecure-content")
options.add_argument("--ignore-certificate-errors")
options.add_argument("--disable-single-click-autofill")
options.add_argument("--disable-autofill-keyboard-accessory-view[8]")
options.add_argument("--disable-full-form-autofill-ios")
options.add_argument("--dns-prefetch-disable") # https://bugs.chromium.org/p/chromedriver/issues/detail?id=402#c128
## ChromeDriver is just AWFUL because every version or two it breaks unless you pass cryptic arguments
options.add_argument("start-maximized") # https://stackoverflow.com/a/26283818/1689770
options.add_argument("enable-automation") # https://stackoverflow.com/a/43840128/1689770
options.add_argument("--no-sandbox") # https://stackoverflow.com/a/50725918/1689770
options.add_argument("--disable-infobars") # https://stackoverflow.com/a/43840128/1689770
options.add_argument("--disable-dev-shm-usage") # https://stackoverflow.com/a/50725918/1689770
options.add_argument("--disable-browser-side-navigation") # https://stackoverflow.com/a/49123152/1689770
options.add_argument("--disable-gpu") # https://stackoverflow.com/questions/51959986/how-to-solve-selenium-chromedriver-timed-out-receiving-message-from-renderer-exc
# options.add_argument("--window-size=1280,960")
options.add_argument("--enable-features=NetworkService,NetworkServiceInProcess") # https://groups.google.com/forum/m/#!topic/chromedriver-users/ktp-s_0M5NM[21-40]
if headless:
options.headless = True
# options.add_argument('--disable-gpu')
args = [
"--verbose"
]
caps = DesiredCapabilities.CHROME
caps['loggingPrefs'] = {
'browser': 'OFF',
'performance' : 'OFF',
'driver' : 'OFF'
}
# https://www.skptricks.com/2018/08/timed-out-receiving-message-from-renderer-selenium.html
# caps['pageLoadStrategy'] = 'none'
# caps['pageLoadStrategy'] = 'normal'
if consolelog:
caps['loggingPrefs']['browser'] = 'ALL'
## by default performance is disabled.
if performancelog:
caps['loggingPrefs']['performance'] = 'ALL'
caps['perfLoggingPrefs'] = {
'enableNetwork' : True,
'enablePage' : False,
'enableTimeline' : False
}
if verbose:
driver = webdriver.Chrome(options=options, service_log_path='chromedriver.log', service_args=args,
desired_capabilities=caps
)
else:
driver = webdriver.Chrome(options=options,
# service_log_path='chromedriver.log',
desired_capabilities=caps
)
logger.debug(f"Driver info: chrome={driver.capabilities['chrome']['chromedriverVersion']}")
logger.debug(f"Session info: chromedriver={driver.capabilities['version']}")
elif browser.upper() == 'EDGE':
# driver = webdriver.Edge()
raise NotImplemented("Edge not supported yet")
elif browser.upper() == 'SAFARI':
# driver = webdriver.Safari()
raise NotImplemented("Safari not supported yet")
# support is depricated for this -- should use chrome or firefox headless.
elif browser.upper() == 'PHANTOMJS':
# driver = webdriver.PhantomJS()
raise NotImplemented("PhantomJS not supported yet")
else:
raise ValueError(f"Unknown browser: {browser}")
# driver.set_page_load_timeout(self.timeout)
# driver.set_script_timeout(self.timeout)
return driver
In both cases I the driver sits in a wrapper that handles automatically quitting the driver.
############################################################################
def close(self):
if self.driver is not None:
self.driver.close()
############################################################################
def quit(self):
if self.driver is not None:
self.driver.quit()
############################################################################
def __del__(self):
self.quit()
Calling close or quit before exiting the script doesn't trigger the error message which has me confused. I had thought the the error message was something in the geckodriver upon garbage collection, but if that were the case I would think I could manually trigger the error by closing the driver prior to sys.exit().
Anyone know where this error message comes from after a firefox selenium run?
EDIT:
It would appear the folder in question is the temporary directory FirefoxProfile creates. I would appear that the firefox.WebDriver.quit() method is where the log line is coming from.
def quit(self):
"""Quits the driver and close every associated window."""
try:
RemoteWebDriver.quit(self)
except Exception:
# We don't care about the message because something probably has gone wrong
pass
if self.w3c:
self.service.stop()
else:
self.binary.kill()
if self.profile is not None:
try:
shutil.rmtree(self.profile.path)
if self.profile.tempfolder is not None:
shutil.rmtree(self.profile.tempfolder)
except Exception as e:
print(str(e)) #<-- this is the line where the error occurs.
Gah! I completely forgot I added a __del__ method to the Firefox webdriver. (this was before I had written the wrapper)
# hack to force firefox to quit when closing scope
def __del_replacement__(self: webdriver.Firefox):
print("__del__ called in Firefox")
self.quit()
webdriver.Firefox.__del__ = __del_replacement__
The problem is there is also a __del__ method in the wrapper around the webdriver which tries to call driver.quit() as well. The problem was caused by both calls to the quit() method trying to remove the temp directory. The 2nd was failing because it had already been removed.
Here is my code. I wrote script which can test multiple test cases. This scripts throws an exception when an error comes however i want that it keeps the count of errors and their descriptions and log after test execution whether it is success or failed. More important is should run complete test script in any how.
from selenium import webdriver
from generic_functions.FindElement import HandyWrappers
from generic_functions.takescreenshots import Screenshot
from generic_functions.error_handle import CatchExceptions
import os
import time
import unittest
class TestEnrollment(unittest.TestCase):
driverLocation =
"C:\\Users\\Sales\\Desktop\\Automation\\Trendz\\libs\\chromedriver.exe"
os.environ["webdriver.chrome.driver"] = driverLocation
driver = webdriver.Chrome(driverLocation)
driver.maximize_window()
driver.implicitly_wait(10)
def test_login(self):
try:
baseURL = "https://portal.ABCD.com"
driver = self.driver
driver.get(baseURL)
hw = HandyWrappers(driver)
username = hw.getElement(".//form[#id='login-form']/fieldset/section[1]/label[2]/input[#name='email']",
locatorType="xpath")
username.send_keys("ABCD#live.com")
time.sleep(2)
password = hw.getElement(".//form[#id='login-form']/fieldset/section[2]/label[2]/input[#name='password']",
locatorType="xpath")
password.send_keys("ABCD09")
signIn = hw.getElement(".//form[#id='login-form']/footer/button[contains(text(),'Sign in')]",
locatorType="xpath")
signIn.click()
self.assertTrue(driver.find_element_by_xpath(".//header[#id='header']/section/div/div[#id='logout']/span/a12"), "Sign Out")
except Exception as err:
raise err
def test_menu_enr(self):
driver = self.driver
hw = HandyWrappers(driver)
find_enrollment = hw.getElement(".//aside[#id='left-panel']/nav/ul[14]/li/a/span", locatorType="xpath")
find_enrollment.click()
if __name__ == '__main__':
unittest.main()
Don't raise it, Use pass or continue
def test_login():
try:
...
except Exception:
# log the exception here
pass # or you could use 'continue'
Note: Have to look into this also diff bw pass and continue
collect all errors in some list and then assert error list at end of test
from selenium import webdriver
from generic_functions.FindElement import HandyWrappers
from generic_functions.takescreenshots import Screenshot
from generic_functions.error_handle import CatchExceptions
import os
import time
import unittest
class TestEnrollment(unittest.TestCase):
driverLocation = "C:\\Users\\Sales\\Desktop\\Automation\\Trendz\\libs\\chromedriver.exe"
os.environ["webdriver.chrome.driver"] = driverLocation
driver = webdriver.Chrome(driverLocation)
driver.maximize_window()
driver.implicitly_wait(10)
def setUp(self):
self.errors = []
def tearDown(self):
self.assertEqual([], self.errors)
def test_login(self):
try:
baseURL = "https://portal.ABCD.com"
driver = self.driver
driver.get(baseURL)
hw = HandyWrappers(driver)
username = hw.getElement(".//form[#id='login-form']/fieldset/section[1]/label[2]/input[#name='email']",
locatorType="xpath")
username.send_keys("ABCD#live.com")
time.sleep(2)
password = hw.getElement(".//form[#id='login-form']/fieldset/section[2]/label[2]/input[#name='password']",
locatorType="xpath")
password.send_keys("ABCD09")
signIn = hw.getElement(".//form[#id='login-form']/footer/button[contains(text(),'Sign in')]",
locatorType="xpath")
signIn.click()
self.assertTrue(driver.find_element_by_xpath(
".//header[#id='header']/section/div/div[#id='logout']/span/a12"), "Sign Out")
except Exception as err:
self.errors.append(err)
def test_menu_enr(self):
driver = self.driver
hw = HandyWrappers(driver)
find_enrollment = hw.getElement(
".//aside[#id='left-panel']/nav/ul[14]/li/a/span", locatorType="xpath")
find_enrollment.click()
if __name__ == '__main__':
unittest.main()
tearDown method will run at end and shows all error messages if any present
As for i know , this is not ideal solution. its kind of hack fix
If you're using selenium webdriver with Python unittest (looks like you are), the SeleniumBase framework has the nifty delayed_assert_text and delayed_assert_element methods built-in for processing test failures at the end of the test. You can install seleniumbase with pip, or clone the repo from github.
The following code is taken from this example.
from seleniumbase import BaseCase
class MyTestClass(BaseCase):
def test_delayed_asserts(self):
self.open('http://xkcd.com/993/')
self.wait_for_element('#comic')
self.delayed_assert_element('img[alt="Brand Identity"]')
self.delayed_assert_element('img[alt="Rocket Ship"]') # Will Fail
self.delayed_assert_element('#comicmap')
self.delayed_assert_text('Fake Item', '#middleContainer') # Will Fail
self.delayed_assert_text('Random', '#middleContainer')
self.delayed_assert_element('a[name="Super Fake !!!"]') # Will Fail
self.process_delayed_asserts()
Inside the script, there are multiple asserts that will fail, but only after self.process_delayed_asserts() is called. Here's the output after running it:
pytest delayed_assert_test.py
=============================== test session starts ===============================
platform darwin -- Python 3.6.5, pytest-3.7.3, py-1.5.4, pluggy-0.7.1
delayed_assert_test.py F
==================================== FAILURES =====================================
________________________ MyTestClass.test_delayed_asserts _________________________
E Exception:
E *** DELAYED ASSERTION FAILURES FOR: examples.delayed_assert_test.MyTestClass.test_delayed_asserts
E CHECK #2: (https://xkcd.com/993/)
E Element {img[alt="Rocket Ship"]} was not visible after 1 second!
E CHECK #4: (https://xkcd.com/993/)
E Expected text {Fake Item} for {#middleContainer} was not visible after 1 second!
E CHECK #6: (https://xkcd.com/993/)
E Element {a[name="Super Fake !!!"]} was not visible after 1 second!
Exception
============================ 1 failed in 7.53 seconds =============================