I have a Celery shared_task in a module tasks that looks like this:
#shared_task
def task():
from core.something import send_it
send_it()
and I am writing a test attempting to patch the send_it method. So far I have:
from ..tasks import task
class TestSend(TestCase):
#override_settings(CELERY_TASK_ALWAYS_EAGER=True)
#patch("core.tasks.send_it")
def test_task(self, send_it_mock):
task()
send_it_mock.assert_called_once()
When I run this, I get the error: AttributeError: <module 'core.tasks' from 'app/core/tasks.py'> does not have the attribute 'send_it'
Out of desperation I've used #patch("tasks.task.send_it") instead, as the import happens inside the shared_task, but I get a similar result. Does anyone know how I can effectively patch the send_it call? Thanks!
I am migrating a project to Django and like to use the django-rq module.
However, I am stuck at what to put here:
import django_rq
queue = django_rq.get_queue('high')
queue.enqueue(func, foo, bar=baz)
How to call func ? Can this be a string like path.file.function ?
Does the function need to reside in the same file?
Create tasks.py file to include
from django_rq import job
#job("high", timeout=600) # timeout is optional
def your_func():
pass # do some logic
and then in your code
import django_rq
from tasks import your_func
queue = django_rq.get_queue('high')
queue.enqueue(your_func, foo, bar=baz)
I have a couple of celery tasks that are included in my Django tests. Unfortunately exceptions are not thrown when tasks are invoked via .delay(). I am setting CELERY_ALWAYS_EAGER to True.
tasks.py
import celeryapp as app
#app.task()
def exception_task():
print 'CELERY_ALWAYS_EAGER:', app.conf['CELERY_ALWAYS_EAGER']
raise Exception('foo')
tests.py
def test_exception_in_task(self):
from tasks import exception_task
exception_task.delay()
Output
CELERY_ALWAYS_EAGER: True
.
----------------------------------------------------------------------
Ran 1 test in 0.686s
When removing the .delay the test exits with an error as excpected:
ERROR: test_exception_in_task
Exception: foo
Versions
celery==3.1.4
Django==1.6.4
Seems I additionally had to set CELERY_EAGER_PROPAGATES_EXCEPTIONS to True.
Under celery 4.0, I had to use CELERY_TASK_EAGER_PROPAGATES
#override_settings(task_always_eager=True, task_eager_propagates=True)
def test_my_task(self):
pass
is what worked for me using Celery 5.
Documentation on config name changes: https://docs.celeryq.dev/en/stable/userguide/configuration.html
I have implemented a Celery task which makes use of some external libraries.
Let's say the task code is something like this:
# mypackage.tasks
import logging
from extpackage1 import module1
from extpackage2 import module2
from celery import Celery
from celery.utils.log import get_task_logger
celery = Celery('tasks', broker='amqp://user:pass#host/vhname')
logger = get_task_logger(__name__)
#celery.task
def my_task(my_job_id):
logger.info('My task executed with my_job_id=%s' % my_job_id)
module1.func1()
module2.func2()
while the 2 modules I import are basically:
# extpackage1.module1
import logging
logger = logging.getLogger(__name__)
def func1():
# let's do something here ...
logger.info('Logging inside module1')
and:
# extpackage2.module2
import logging
logger = logging.getLogger(__name__)
def func2():
# let's do something here ...
logger.info('Logging inside module2')
I would like to obtain a log file which include the correct value of my_job_id in each line, depending on the task which is executed.
It would be something such as:
[1][...other info..] My task executed with my_job_id=1
[1][...other info..] Logging inside module1
[1][...other info..] Logging inside module2
[2][...other info..] My task executed with my_job_id=2
[2][...other info..] Logging inside module1
[2][...other info..] Logging inside module2
I could easily include the value of my_job_id into the task logger, correctly obtaining the lines I log from inside the task function (like [1][...other info..] My task executed with my_job_id=1 and [2][...other info..] My task executed with my_job_id=2).
But what about the external libraries? Is there an elegant way to 'wrap' the log produced by the external script and format it according to my requirements?
The Celery documentation mentions testing Celery within Django but doesn't explain how to test a Celery task if you are not using Django. How do you do this?
It is possible to test tasks synchronously using any unittest lib out there. I normaly do 2 different test sessions when working with celery tasks. The first one (as I'm suggesting bellow) is completely synchronous and should be the one that makes sure the algorithm does what it should do. The second session uses the whole system (including the broker) and makes sure I'm not having serialization issues or any other distribution, comunication problem.
So:
from celery import Celery
celery = Celery()
#celery.task
def add(x, y):
return x + y
And your test:
from nose.tools import eq_
def test_add_task():
rst = add.apply(args=(4, 4)).get()
eq_(rst, 8)
Here is an update to my seven years old answer:
You can run a worker in a separate thread via a pytest fixture:
https://docs.celeryq.dev/en/v5.2.6/userguide/testing.html#celery-worker-embed-live-worker
According to the docs, you should not use "always_eager" (see the top of the page of the above link).
Old answer:
I use this:
with mock.patch('celeryconfig.CELERY_ALWAYS_EAGER', True, create=True):
...
Docs: https://docs.celeryq.dev/en/3.1/configuration.html#celery-always-eager
CELERY_ALWAYS_EAGER lets you run your task synchronously, and you don't need a celery server.
Depends on what exactly you want to be testing.
Test the task code directly. Don't call "task.delay(...)" just call "task(...)" from your unit tests.
Use CELERY_ALWAYS_EAGER. This will cause your tasks to be called immediately at the point you say "task.delay(...)", so you can test the whole path (but not any asynchronous behavior).
For those on Celery 4 it's:
#override_settings(CELERY_TASK_ALWAYS_EAGER=True)
Because the settings names have been changed and need updating if you choose to upgrade, see
https://docs.celeryproject.org/en/latest/history/whatsnew-4.0.html?highlight=what%20is%20new#lowercase-setting-names
unittest
import unittest
from myproject.myapp import celeryapp
class TestMyCeleryWorker(unittest.TestCase):
def setUp(self):
celeryapp.conf.update(CELERY_ALWAYS_EAGER=True)
py.test fixtures
# conftest.py
from myproject.myapp import celeryapp
#pytest.fixture(scope='module')
def celery_app(request):
celeryapp.conf.update(CELERY_ALWAYS_EAGER=True)
return celeryapp
# test_tasks.py
def test_some_task(celery_app):
...
Addendum: make send_task respect eager
from celery import current_app
def send_task(name, args=(), kwargs={}, **opts):
# https://github.com/celery/celery/issues/581
task = current_app.tasks[name]
return task.apply(args, kwargs, **opts)
current_app.send_task = send_task
As of Celery 3.0, one way to set CELERY_ALWAYS_EAGER in Django is:
from django.test import TestCase, override_settings
from .foo import foo_celery_task
class MyTest(TestCase):
#override_settings(CELERY_ALWAYS_EAGER=True)
def test_foo(self):
self.assertTrue(foo_celery_task.delay())
Since Celery v4.0, py.test fixtures are provided to start a celery worker just for the test and are shut down when done:
def test_myfunc_is_executed(celery_session_worker):
# celery_session_worker: <Worker: gen93553#mymachine.local (running)>
assert myfunc.delay().wait(3)
Among other fixtures described on http://docs.celeryproject.org/en/latest/userguide/testing.html#py-test, you can change the celery default options by redefining the celery_config fixture this way:
#pytest.fixture(scope='session')
def celery_config():
return {
'accept_content': ['json', 'pickle'],
'result_serializer': 'pickle',
}
By default, the test worker uses an in-memory broker and result backend. No need to use a local Redis or RabbitMQ if not testing specific features.
reference
using pytest.
def test_add(celery_worker):
mytask.delay()
if you use flask, set the app config
CELERY_BROKER_URL = 'memory://'
CELERY_RESULT_BACKEND = 'cache+memory://'
and in conftest.py
#pytest.fixture
def app():
yield app # Your actual Flask application
#pytest.fixture
def celery_app(app):
from celery.contrib.testing import tasks # need it
yield celery_app # Your actual Flask-Celery application
In my case (and I assume many others), all I wanted was to test the inner logic of a task using pytest.
TL;DR; ended up mocking everything away (OPTION 2)
Example Use Case:
proj/tasks.py
#shared_task(bind=True)
def add_task(self, a, b):
return a+b;
tests/test_tasks.py
from proj import add_task
def test_add():
assert add_task(1, 2) == 3, '1 + 2 should equal 3'
but, since shared_task decorator does a lot of celery internal logic, it isn't really a unit tests.
So, for me, there were 2 options:
OPTION 1: Separate internal logic
proj/tasks_logic.py
def internal_add(a, b):
return a + b;
proj/tasks.py
from .tasks_logic import internal_add
#shared_task(bind=True)
def add_task(self, a, b):
return internal_add(a, b);
This looks very odd, and other than making it less readable, it requires to manually extract and pass attributes that are part of the request, for instance the task_id in case you need it, which make the logic less pure.
OPTION 2: mocks
mocking away celery internals
tests/__init__.py
# noinspection PyUnresolvedReferences
from celery import shared_task
from mock import patch
def mock_signature(**kwargs):
return {}
def mocked_shared_task(*decorator_args, **decorator_kwargs):
def mocked_shared_decorator(func):
func.signature = func.si = func.s = mock_signature
return func
return mocked_shared_decorator
patch('celery.shared_task', mocked_shared_task).start()
which then allows me to mock the request object (again, in case you need things from the request, like the id, or the retries counter.
tests/test_tasks.py
from proj import add_task
class MockedRequest:
def __init__(self, id=None):
self.id = id or 1
class MockedTask:
def __init__(self, id=None):
self.request = MockedRequest(id=id)
def test_add():
mocked_task = MockedTask(id=3)
assert add_task(mocked_task, 1, 2) == 3, '1 + 2 should equal 3'
This solution is much more manual, but, it gives me the control I need to actually unit test, without repeating myself, and without losing the celery scope.
I see a lot of CELERY_ALWAYS_EAGER = true in unit tests methods as a solution for unit tests, but since the version 5.0.5 is available there are a lot of changes which makes most of the old answers deprecated and for me a time consuming nonsense, so for everyone here searching a Solution, go to the Doc and read the well documented unit test examples for the new Version:
https://docs.celeryproject.org/en/stable/userguide/testing.html
And to the Eager Mode with Unit Tests, here a quote from the actual docs:
Eager mode
The eager mode enabled by the task_always_eager setting is by
definition not suitable for unit tests.
When testing with eager mode you are only testing an emulation of what
happens in a worker, and there are many discrepancies between the
emulation and what happens in reality.
Another option is to mock the task if you do not need the side effects of running it.
from unittest import mock
#mock.patch('module.module.task')
def test_name(self, mock_task): ...