I'm having some trouble trying to figure how to test my app architecture. I've already 60% of my website complete, with full unit testing coverage (over all utility/lib functions, celery tasks as simple functions, and so on).
The problem arises when I try to test django views (plain functions) that executes celery tasks (delay method).
Example:
def myview(request):
...
mytask.delay(myval)
...
What should be the right way to test that scenes without really generating a new task execution?
The obvious way is to set up a condition before every task delay call, executing it only if it's not in test environment, but it seems really dirty.
Any tip?
Use CELERY_ALWAYS_EAGER settings for test run.
It make the function to be called immediately instead of running it as a task.
Example django settings snippet:
if 'test' in sys.argv:
CELERY_ALWAYS_EAGER = True
Related
Without changing the code itself, Is there a way to ignore tasks in Celery?
For example, when using Django mails, there is a Dummy Backend setting. This is perfect since it allows me, from a .env file to deactivate mail sending in some environments (like testing, or staging). The code itself that handles mail sending is not changed with if statements or decorators.
For celery tasks, I know I could do it in code using mocks or decorators, but I'd like to do it in a clean way that is 12factors compliant, like with Django mails. Any idea?
EDIT to explain why I want to do this:
One of the main motivation behind this, is that it creates coupling between Django web server and Celery tasks.
For example, when running unit tests, if the broker server (Redis for me) is not running, then if delay() method is called, it freezes forever, because there is no timeout when Celery tries to send a task to Redis.
From an architecture view, this is very bad. I'd like my unit tests can run properly without the requirement to run a Celery broker!
Thanks!
As far as the coupling is concerned, your Django application would still be tied to celery if you use a dummy backend. Just your tasks won't execute. Maybe this is acceptable in your case but in my opinion, it can cause some problems. For example, if the piece of code you are trying to test, submits a task to celery, and in a later part, tries to retrieve the result for that task, it will fail. Because the dummy backend will never execute the task.
For unit testing, as you mentioned in your question, you can use the task_always_eager setting. If you turn it on, your Django app will no longer depend upon a running worker. It will execute tasks in the same thread in a synchronous fashion and return the result.
I have set up a testing environment where I have Celery workers actually running in other processes, so that the full functionality of my system with Celery can be tested. This way, tasks actually run in a worker process and communicate back to the test runner, and so I don't need CELERY_ALWAYS_EAGER to test this functionality.
That being said, in some situations I have tasks which trigger off other tasks without caring when they finish, and I'd like to create tests which do - that is, to wait for those subtasks to finish. In these cases, the simplest approach seems to be to run just these tests eagerly (i.e. with CELERY_ALWAYS_EAGER set to true).
However, I don't see a straightforward way to change the config after Celery is initialized... and indeed, from a glance at the source code, it seems that it assumes the config won't change once the app starts.
This makes sense for a lot of options, since the worker would have to actually see the change, and changing it from the main program wouldn't do anything. But in the case of CELERY_ALWAYS_EAGER, this makes sense for the main program to be able to change it.
Is there any straightforward/well-supported way to do this? If not, what's a preferably not-too-hacky way to do this?
Another option is to make the task in question return the task ids it started off, so that the test can then wait on them... but I don't like the idea of changing my API for the sole purpose of making it runnable in a unit test.
Simply changing variables on Celery's .conf object (an instance of Settings) works:
app.conf.CELERY_ALWAYS_EAGER = True
Although conf is indeed a #cached_property of Celery (in version 3.1.22 anyway), this caches the instance returned, not all the values - so the configuration is indeed dynamically updatable.
BACKGROUND: I'm working on a project that uses Celery to schedule tasks that will run at a certain time in the future. These tasks push the state of the Final State Machine forward. Here's an example:
A future reminder is scheduled to be sent to the user in 2 days.
When that scheduled task runs, an email is sent, and the FSM is advanced to the next state
The next state is to schedule a reminder to run in another two days
When this task runs, it will send another email, advance state
Etc...
I'm currently using CELERY_ALWAYS_EAGER as suggested by this SO answer
The problem with using that technique in tests, is that the task code, which is meant to run in a separate thread is running in the same one as the one that schedules it. This causes the FSM state to not be saved properly, and making it hard to test. I haven't been able to determine what exactly causes it, but it seems like at the bottom of the call stack you are saving to the current state, but as you return up the call stack, a previous state is being saved. I could possibly spend more time determining what is going wrong when the code is not running how it should, but it seems more logical to try to get the code running how it should and make sure it's doing what it should.
QUESTION: I would therefore like to know if there is a way to run a full on celery setup that django can use during a test run. If it could be run automagically, that would be ideal, but even some manual intervention would be better than having to test behavior by hand. I'm thinking something could be possible if I set a break in the tests, run the celery worker to connect to the test DB, continue the django tests. Has anyone tried something like this before?
What you are trying to do is not unit testing but rather functional / integration testing.
I would recommend to use some BDD framework (Behave, Lettuce) and run BDD tests from a CI server (TravisCI or Jenkins) against external server (staging environment for example).
So, the process could be:
Push changes to GitHub
GitHub launches build on CI server
CI server runs unit tests
CI server deploys to integration environment (or staging, if you don't have integration)
CI server runs integration end to end tests against the new deployed code
If all succeeds, this build will be promoted to "can be deploy to production" or something like that
I have a Flask application that uses Nose to discover and run a series of tests in a particular directory. The tests take a long time to run, so I want to report the progress to the user as things are happening.
I use Celery to create a task that runs the test so I can return immediately and start displaying a results page. Now I need to start reporting results. I'm thinking in the test that I can just put a message on the queue that says 'I've completed step N'.
I know that Celery has task context I could use to determine which queue to write to, but the test isn't part of the task, it's a function that's called from the task. I also can't use a flask session, because that context is gone when the test run is moved to a task.
I have seen several ways to do data driven nose tests, such as test generators or nose-testconfig, but that doesn't meet the requirement that the message queue name will be dynamic and there may be several threads running the same test.
So, my question is: How do I tell the test that it corresponds to a particular celery task, ie: the one that started the test, so I can report it's status on the correct message queue?
I am working on a Django application. We're using celery to queue writes to our Mongo database. I'm trying to write a functional test (using Selenium) for a function that queues something in celery.
The problem is that celery writes to the main Mongo database instead of the test database. How can I set up my functional tests to work with an instance of celery that writes to the test database?
We're using 'django_nose.NoseTestSuiteRunner' as our TEST_RUNNER.
UPDATE:
I haven't been able to figure out how to use another instance of celery for the tests, but I have found a way to bypass celery for the functional tests.
In my settings.py:
FUNC_TEST_COMMAND=['functional']
func_test_command = filter(lambda element: element in FUNC_TEST_COMMAND, sys.argv)
if len(func_test_command) > 0:
CELERY_ALWAYS_EAGER = True
This mimics the behaviour of an AsyncResult without sending anything through a message queue when running the functional test suite. (See http://celery.readthedocs.org/en/2.4/configuration.html#celery-always-eager for more info.)
This solution is probably not ideal for functional tests, because it cuts out one of the application layers.
Using CELERY_ALWAYS_EAGER = True does indeed bypass Celery's asynchornous processing. In order to write to the test database, you'll need to start your celeryd worker using the connection settings to the test database.
I'd suggest that you take a look at LiveServerTestCase if your using an automated test client for running functional tests.
Then make sure you have a separate settings module your running your tests with that properly configures Celery to use your project's database for transport.