My code is running some instances of threading.Thread for some long asynchronous tasks.
This does not allow me running my django unittests using sqlite backend, because sqlite can not handle multiple connections in threads. Thus, I am successfully mocking Thread with a FakeThread class that i wrote (it simply runs the target synchronously).
However, the mock does not seem to work for selenium tests. I do:
from tests.stubs import FakeThread
# ...
class FunctionalTest(LiveServerTestCase):
#mock.patch('accounts.models.user_profile.Thread', new=FakeThread)
def test_register_agency(self):
self.browser.get("%s%s" % (self.live_server_url, "/register"))
# .. fill in form, submit, eventually calls something in user_profile
# using an instance of Thread. Thread seems to still be threading.Thread
Any idea how to mock Thread in the code that selenium runs when serving my browser calls? Thank you!
Related
I want to run a python script which basically monitors any changes happening to a particular directory (the directory to monitor is passed as part of the POST request). Every time the API is called (I'm using FastAPI), a new instance of the script has to be started to monitor that particular directory and send back a "success" message as response if the script was started successfully. Further I am planning to add another API endpoint that will stop the script that is running to watch a directory.
Can message queues like RQ or Celery be used to achieve this? Please note that I want new scripts to be started every time the API is called so multiple instances of the script should run at the same time. I am using watchdog module to monitor the file system.
I don't know how to do this in the correct way but this is what I have come up with so far where a new thread is created for each API call:
from fastapi import FastAPI
from schemas import Data # pydantic schema model for API
from threading import Thread
import filewatcher # the script that has to be run
#app.post('/register/event')
def register_watchdog(data: Data):
th = Thread(target=filewacther.create_watchdog, args=(data))
th.start()
return {"status": "success"}
What is the best way to achieve this? One further question is, can I implement my script as a Linux service that can run in the background?
In fact, it is rather trivial to call this function in the code if you need to return the result of directory browsing function.
...
def register_watchdog(data: Data):
return {"result": filewacther.create_watchdog(data)}
But if you want to run some time-consuming process in the background, you really should use AMQP with the workers. RabbitMQ with Celery is really the right choice for this, it will make it easy to scale your system.
What is the best way to achieve this? One further question is, can I implement my script as a Linux service that can run in the background?
Yes, you can indeed run RabbitMQ together with Celery as a linux service in the background e.g. using supervisor (example), but this is not the best practice. Look in the direction of containerizing elements of your system. You can wrap Celery in a Docker and make it easy enough to run that along with the AMQP service and your web application (example).
I have written a python test script to test out functionality of a website. Functionality such as Login into the web page, etc. In order to maximize testing, I have tried to implement multithreading to speed up the test process (so I could run two test cases concurrently). I found out that when I run the scripts, two browser would be open (which is correct), however, only one of the browser would be doing the actions I have scripted (such as clicking an element). I am able to browser.get(link) correctly but browser.find_element_by_xpath(xpath).click() didn't work.
thread1 = threading.Thread(target=runTC, args=(argument1,))
thread2 = threading.Thread(target=runTC, args=(argument2,))
# Will execute both in parallel
thread1.start()
thread2.start()
runTC() consists of the test functions I wrote.
To use benefit of multi-threading each your thread has to work with different WebDriver object since a WebDrider object is a sort of binding to a particular session/instance of your web browser.
A web driver is a service that exposes REST interface (W3C standard) to your PL bindings on one hand and translates the calls from your test script to what a particular browser understands on another hand (that is why you have different web drivers for different browsers).
When you create an instance of WebDriver in your test script it actually establishes a session that is associated with running browser. So if your multiple threads use the same object, they'll be acting within the same session, use the same cookies and hence impact each other.
If your threads use their own instances of WebDriver they'll be running in isolation in parallel each in its own browser.
Running selenium tests in python and chrome. Between each test, it logs that its creating a new http connection.
Is it possible to create a connection pool so that a new one isn't created for each test?
Based on the low details provided on your question, and that you're mentioning that this happens only between tests, leads me to think that you have something in your setUp or tearDown methods.
Check what gets executed there and see whether that is the source of your problem.
I wrote a Rhythmbox plugin and I'm trying to add some code to download some JSON asynchronously. Callbacks are registered in the do_activate function:
def do_activate(self):
shell = self.object
sp = shell.props.shell_player
self.db = shell.get_property('db')
self.qm = RB.RhythmDBQueryModel.new_empty(self.db)
self.pec_id = sp.connect('playing-song-changed', self.playing_entry_changed)
self.pc_id = sp.connect('playing-changed', self.playing_changed)
self.sc_id = sp.connect('playing-source-changed', self.source_changed)
self.current_entry = None
...
I'm trying to download some content when playing_changed is triggered. It currently uses urllib2 to download the content synchronously, but this has the potential to block the UI for a short while. I'd like to use Twisted to solve the problem, but all the examples I've seen use reactor.run(), which blocks indefinitely.
I'm pretty new to Twisted and I was wondering, is there some way to handle this case asynchronously without blocking the main thread?
The full code is here
There isn't any way in twisted to do asynchronous http requests without running IO-loop (reactor.run). Running reactor enables you to use async features not present in python by default. However if your only reason to use twisted is to make async http calls it might be an overkill. Use simple threading instead and make your thread wait for http response.
In the context of a Rhythmbox plugin, you probably need to deal with the fact that the GTK main loop is already running. This is a situation that Twisted supports in principle, but supported APIs to cooperatively initialize a reactor on a main loop that may or may not already have one are tricky.
You can work around it with a function like this:
def maybeInstallReactor():
import sys
if 'twisted.internet.reactor' not in sys:
from twisted.internet import gtk2reactor # s/2/3 if you're using gtk3
reactor = gtk2reactor.install()
reactor.startRunning()
reactor._simulate()
else:
from twisted.internet import reactor
return reactor
Make sure this function is called as early as possible in your program, before anything else gets imported (especially stuff from Twisted).
The startRunning call hooks the reactor up to the GLib main loop, and the _simulate call hooks up Twisted's timed events to a GLib timer.
Sadly, this does involve calling one private function, _simulate, so you'll have to be careful to make sure new versions of Twisted don't break it; but as a result of this question I opened a bug to make this use-case explicitly supported. Plus, beyond this one private method call, nothing else about your usage of Twisted needs to be weird.
I'm trying to debug an error where python import statements randomly fail, at other times they run cleanly.
This is an example of the exceptions I see. Sometimes I'll see this one, sometimes I'll see another one in a different module, though it seems to always hit in one of 4 modules.
ERROR:root:/home/user/projecteat/django/contrib/auth/management/__init__.py:25: RuntimeWarning: Parent module 'django.contrib.auth.management' not found while handling absolute import
from django.contrib.contenttypes.models import ContentType
Because of the random nature, I'm almost certain it's a threading issue, but I don't understand why I would get import errors, so I'm not sure what to look for in debugging. Can this be caused by filesystem contention if different threads are trying to load the same modules?
I'm trying to get Django 1.4's LiveServerTestCase working on Google App Engine's development server. The main thread runs django's test framework. When it loads up a LiveServerTestCase based test class, it spawns a child thread which launches the App Engine dev_appserver, which is a local webserver. The main thread continues to run the test, using the Selenium driver to make HTTP requests, which are handled by dev_appserver on the child thread.
The test framework may run a few tests in the LiveServerTestCase based class before tearing down the testcase class. At teardown, the child thread is ended.
It looks like the exceptions are happening in the child (HTTP server) thread, mostly between tests within a single testcase class.
The code for the App Engine LiveServerTestCase class is here: https://github.com/dragonx/djangoappengine/blob/django-1.4/test.py
It's pretty hard to provide all the debugging info required for this question. I'm mostly looking for suggestions as to why python import statements would give RuntimeWarning errors.
I have a partial answer to my own question. What's going on is that I have two threads running.
Thread 1 is running the main internal function inside dev_appserver (dev_appserver_main) which is handling HTTP requests.
Thread 2 is running the Selenium based testcases. This thread will send commands to the browser to do something (which then indirectly generates an HTTP request and re-enters in thread 1). It then either issues more requests to Selenium to check status, or makes a datastore query to check for a result.
I think the problem is that upon handling every HTTP request, Thread 1 (dev_appserver) changes the environment so that certain folders are not accessible (folder excluded in app.yaml, as well as the environment that is not part of appengine). If Thread 2 happens to run some code in this time, certain imports may fail to load if they are located in these folders.