I am trying to leverage testing Dash application as described here: https://dash.plotly.com/testing
However I found no way of specifying the Chromedriver path for the webdriver-manager under the hood of dash testing.
I tried this below which calls webdriver-manager before reaching the test code:
def test_bsly001_falsy_child(dash_duo):
app = import_app("my_app_path")
dash_duo.start_server(app)
webdriver-manager then would start downloading the latest Chrome version. But due to company policy we cannot just download things from the internet, it is blocked by firewall. We are supposed to use the Chromedriver which is already downloaded for us on the internal network.
I tried implementing a pytest fixture to set up the Chrome driver before the testing starts:
driver = webdriver.Chrome(executable_path="...")
But webdriver-manager does not accept this.
Do you know any ways of working around this? Any hints?
Any way of doing Dash testing without webdriver-manager?
Thanks.
I had a similar problem and ended up with another pytest fixture dash_duo_bis which uses another class DashComposite(Browser) inside the conftest.py where the _get_chrome method is overriden as follows:
class DashComposite(Browser):
def __init__(self, server, **kwargs):
super().__init__(**kwargs)
self.server = server
def get_webdriver(self):
return self._get_chrome()
def _get_chrome(self):
return webdriver.Chrome(executable_path = r"MY_CHROME_EXECUTABLE_PATH")
def start_server(self, app, **kwargs):
"""Start the local server with app."""
# start server with app and pass Dash arguments
self.server(app, **kwargs)
# set the default server_url, it implicitly call wait_for_page
self.server_url = self.server.url
Related
I need to scrape many urls with Selenium and Scrapy. To speed up whole process, I'm trying to create a bunch of shared Selenium instances. My idea is to have a set of parallel Selenium instances available to any Request if needed and released if done.
I tried to create a Middleware but the problem is that Middleware is sequential (I see all drivers (I call it browsers) loading urls and it seems to be sequential). I want all drivers work parallel.
class ScrapySpiderDownloaderMiddleware(object):
BROWSERS_COUNT = 10
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.free_browsers = set(
[webdriver.Chrome(executable_path=BASE_DIR + '/chromedriver') for x in range(self.BROWSERS_COUNT)])
def get_free_browser(self):
while True:
try:
return self.free_browsers.pop()
except KeyError:
time.sleep(0.1)
def release_browser(self, browser):
self.free_browsers.add(browser)
def process_request(self, request, spider):
browser = self.get_free_browser()
browser.get(request.url)
body = str.encode(browser.page_source)
self.release_browser(browser)
# Expose the driver via the "meta" attribute
request.meta.update({'browser': browser})
return HtmlResponse(
browser.current_url,
body=body,
encoding='utf-8',
request=request
)
I don't like solutions where you do:
driver.get(response.url)
in parse method because it causes redundant requests. Every url is being requested two times which I need to avoid.
For example this https://stackoverflow.com/a/17979285/2607447
Do you know what to do?
I suggest you look towards scrapy + docker. you can run many instances at once
As #Granitosaurus suggested, Splash is a good choice. I personally used Scrapy-splash - Scrapy takes care of parallel processing and Splash takes care of website rendering including JavaScript execution.
I have created a python code to initialise a chrome browser and I want to pass this driver instance to Robot Framework so that the keywords of RF will work on this instance. Please let me know how could i do the same.
The Py code for intializing a file is :
'class SeleniumKeywords:
ROBOT_LIBRARY_SCOPE = 'GLOBAL'
def __init__(self):
self.driver = None
def open_browser(self, browser="chrome"):
driver_class = drivers[browser]
self.driver = driver_class()
def go_to(self, url):
self.driver.get(url)'
Now when using it in Robot framework, the browser opens but the RF keywords doesnt work on it(selenium2library keywords). Basically I opening an browser instance using the custom keyword and maximizing using selenium2library keywords in RF. Unfortunately it doesnt work. Please let me know how to pass this browser instance to RF:
'*** Settings ***
Library ExtendedSelenium2Library
Library ../Resources/SeleniumKeywords.py
Resource ../Global/GlobalConfig.robot
*** Test Cases ***
Reuse Session ID
SeleniumKeywords.Open Browser chrome
maximize browser window
SeleniumKeywords.Go To ${URL} '
The maximize browser window is a RF keyword and I want it to work on my browserinstance
I have written my own library, but I extended the Selenium2Library and i can mix the Selenium2Library keywords with my own. The Selenium2Library or in your case, the ExtendedSelenium2Library will not recognize the session you just started in Python and will give the "No browser is open" error. The "Maximize Browser Window" keyword relies on a browser that was previoulsy opened with the "Open Browser" keyword.
If you really need your own Selenium library, you can do something like this:
class SeleniumKeywords(ExtendedSelenium2Library):
def go_to(self, url, browser="chrome"):
self.open_browser(url, browser)
Good day,
I am trying to implement the mitmproxy into a bigger application.
For that, I need to be able to load those so called inline scripts in my code and not via command line. I could not find any helpful information about that in the documentation.
I am using mitmproxy version 0.17 and Python 2.7.
I know there is a newer version available, but that one didnt worked using the code examples.
This is the base code I have:
from mitmproxy import controller, proxy
from mitmproxy.proxy.server import ProxyServer
class ProxyMaster(controller.Master):
def __init__(self, server):
controller.Master.__init__(self, server)
def run(self):
try:
return controller.Master.run(self)
except KeyboardInterrupt:
self.shutdown()
def handle_request(self, flow):
flow.reply()
def handle_response(self, flow):
flow.reply()
config = proxy.ProxyConfig(port=8080)
server = ProxyServer(config)
m = ProxyMaster(server)
m.run()
How could I run this proxy using inline scripts?
Thanks in advance
I figured myself out a really ugly workaround.
Instead of using controller.Master I had to use flow.FlowMaster as the controller.Master lib does not seem to be able to handle inline scripts.
For some reason just loading the files did not work, they get triggered immediately, but not by running their matching hooks.
Instead of using the hooks which are not working, I am loading the matching functions as you can see in handle_response (try/except is missing and threading could be useful)
from mitmproxy import flow, proxy
from mitmproxy.proxy.server import ProxyServer
import imp
class ProxyMaster(flow.FlowMaster):
def run(self):
try:
return flow.FlowMaster.run(self)
except KeyboardInterrupt:
self.shutdown()
def handle_request(self, flow):
flow.reply()
def handle_response(self, flow):
for inline_script in self.scripts:
script_file = imp.load_source("response", inline_script.filename)
script_file.response(self, flow)
flow.reply()
proxy_config = proxy.ProxyConfig(port=8080)
server = ProxyServer(proxy_config)
state = flow.State()
m = ProxyMaster(server, state)
m.load_script("upsidedowninternet.py")
m.load_script("add_header.py")
m.run()
Any ideas about doing it the right way are appreciated.
I am trying to write a automation test using selenium and python webdriver
class UserAccountsTest(unittest.TestCase):
def setup(self):
self.driver=webdriver.Firefox()
def testFirstUser(self):
driver = self.driver
driver.implicitly_wait(15)
driver.get('website.com')
driver.implicitly_wait(15)
My error:
object has no attribute 'driver'
Im not sure why I can't assign driver to self.driver. Am I passing in self correctly?
You need to call your setup() function setUp() as described in unittest wiki.
Luckily, we can factor out such set-up code by implementing a method
called setUp(), which the testing framework will automatically call
for us when we run the test
I'm trying to use IE8 through COM to access a secured site (namely, SourceForge), in Python. Here is the script:
from win32com.client import gencache
from win32com.client import Dispatch
import pythoncom
gencache.EnsureModule('{EAB22AC0-30C1-11CF-A7EB-0000C05BAE0B}', 0, 1, 1)
class SourceForge(object):
def __init__(self, baseURL='https://sourceforget.net/', *args, **kwargs):
super(SourceForge, self).__init__(*args, **kwargs)
self.__browser = Dispatch('InternetExplorer.Application')
self.__browser.Visible = True
self.__browser.Navigate(baseURL)
def run(self):
while True:
pythoncom.PumpMessages()
def main():
sf = SourceForge()
sf.run()
if __name__ == '__main__':
main()
If I launch IE by hand, fine. If I launch the script, I get a generic error page "Internet Explorer cannot display this page". If I change baseURL to use http instead of https, the script works. I guess this is some security "feature". I tried adding the site to the list of trusted sites. I tried to enable IE scripting in the options for the Internet zone. Doesn't work. Google was no help.
So, does anybody know something about this ? Is there a mysterious option to enable or am I doomed ?
I'm on Windows XP SP3 BTW, Python 2.5 and pywin32 build 213.
I can't open https://sourceforget.net/ -- not by hand, not by script.
Are you sure this link is right?