We Keep Coding

Mocking code run inside an rq SimpleWorker

I have code which uses Python requests to kick off a task which runs in a worker that is started with rq. (Actually, the GET request results in one task which itself starts a second task. But this complexity shouldn't affect things, so I've left that out of the code below.) I already have a test which uses rq's SimpleWorker class to cause the code to run synchronously. This works fine. But now I'm adding requests_ratelimiter to the second task, and I want to be sure it's behaving correctly. I think I need to somehow mock the time.sleep() function used by the rate limiter, and I can't figure out how to patch it.
routes.py
#app.route("/do_work/", methods=["POST"])
def do_work():
rq_job = my_queue.enqueue(f"my_app.worker.do_work", job_timeout=3600, *args, **kwargs)
worker.py
from requests_ratelimiter import LimiterSession
#job('my_queue', connection=redis_conn, timeout=3600, result_ttl=24 * 60 * 60)
def do_work():
session = LimiterSession(per_second=1)
r = session.get(WORK_URL)
test.py
import requests_mock
def test_get(client):
# call the Flask function to kick off the task
client.get("/do_work/")
with requests_mock.Mocker() as m:
# mock the return value of the requests.get() call in the worker
response_success = {"result": "All good"}
m.get(WORK_URL, json=response_success)
worker = SimpleWorker([my_queue], connection=redis_conn)
worker.work(burst=True) # Work until the queue is empty
A test in requests_ratelimiter patches the sleep function using a target path of 'pyrate_limiter.limit_context_decorator.sleep', but that doesn't work for me because I'm not declaring pyrate_limiter at all. I've tried mocking the time function and then passing that into the LimiterSession, and that sort of works:
worker.py
from requests_ratelimiter import LimiterSession
from time import time
#job('my_queue', connection=redis_conn, timeout=3600, result_ttl=24 * 60 * 60)
def do_work():
session = LimiterSession(per_second=1, time_function=time)
r = session.get(WORK_URL)
test.py
import requests_mock
def test_get(client):
# call the Flask function to kick off the task
client.get("/do_work/")
with patch("my_app.worker.time", return_value=None) as mock_time:
with requests_mock.Mocker() as m:
response_success = {"result": "All good"}
m.get(URL, json=response_success)
worker = SimpleWorker([my_queue], connection=redis_conn)
worker.work(burst=True) # Work until the queue is empty
assert mock_time.call_count == 1
However, then I see time called many more times than sleep would be, so I don't get the info I need from it. And patching my_app.worker.time.sleep results in the error:
AttributeError: does not have the attribute 'sleep'
I have also tried patching the pyrate_limiter as the requests_ratelimiter testing code does:
with patch(
"my_app.worker.requests_ratelimiter.pyrate_limiter.limit_context_decorator.sleep", return_value=None
) as mock_sleep:
But this fails with:
ModuleNotFoundError: No module named 'my_app.worker.requests_ratelimiter'; 'my_app.worker' is not a package
How can I test and make sure the rate limiter is engaging properly?

The solution was indeed to use 'pyrate_limiter.limit_context_decorator.sleep', despite the fact that I wasn't importing it.
When I did that and made the mock return None, I discovered that sleep() was being called tens of thousands of times because it's in a while loop.
So in the end, I also needed to use freezegun and a side effect on my mock_sleep to get the behavior I wanted. Now time is frozen, but sleep() jumps the test clock forward synchronously and instantly by the amount of seconds passed as an argument.
from datetime import timedelta
from unittest.mock import patch
import requests_mock
from freezegun import freeze_time
from rq import SimpleWorker
def test_get(client):
with patch("pyrate_limiter.limit_context_decorator.sleep") as mock_sleep:
with freeze_time() as frozen_time:
# Make sleep operate on the frozen time
# See: https://github.com/spulec/freezegun/issues/47#issuecomment-324442679
mock_sleep.side_effect = lambda seconds: frozen_time.tick(timedelta(seconds=seconds))
with requests_mock.Mocker() as m:
m.get(URL, json=response_success)
worker = SimpleWorker([my_queue], connection=redis_conn)
worker.work(burst=True) # Work until the queue is empty
# The worker will do enough to get rate limited once
assert mock_sleep.call_count == 1

Unable to patch logging.Logger.info called inside celery task

I want to test that a specific celery task call logger.info exactly once when the task is invoked with the delay() API.
And I want to do the test by patching logger.info .
I want to test as described here for the Product.order case https://docs.celeryproject.org/en/latest/userguide/testing.html.
The setup is : python 2.7 on ubuntu 16.04 . celery 4.3.0 , pytest 4.0.0, mock 3.0.3.
I have the following file system structure:
poc/
prj/
-celery_app.py
-tests.py
celery_app.py
from __future__ import absolute_import
from celery import Celery
from celery.utils.log import get_task_logger
app = Celery('celery_app')
logger = get_task_logger(__name__)
#app.task(bind=True)
def debug_task(self):
logger.info('Request: {0!r}'.format(self.request))
tests.py
from __future__ import absolute_import
from mock import patch
from prj.celery_app import debug_task
#patch('logging.Logger.info')
def test_log_info_is_called_only_once_when_called_sync(log_info):
debug_task()
log_info.assert_called_once()
#patch('logging.Logger.info')
def test_log_info_is_called_only_once_when_called_async(log_info):
debug_task.delay()
log_info.assert_called_once()
I expect both tests to have success.
Instead the first has success, while the second fails with AssertionError: Expected 'info' to have been called once. Called 0 times.
I expect that the evaluation of the expression logger.info inside debug_task() context evaluates to <MagicMock name='info' id='someinteger'> in both cases, instead it evaluates to <bound method Logger.info of <logging.Logger object at 0x7f894eeb1690>> in the second case, showing no patching.
I know that in the second case the celery worker executes the task inside a thread.
I ask for a way to patch the logger.info call when debug_task.delay() is executed.
Thanks in advance for any answer.

Module 'a' has no attribute 'b' while importing module from same directory

I have following directory structure in my Python project:
- dump_specs.py
/impa
- __init__.py
- server.py
- tasks.py
I had a problem with circular references. dump_specs.py needs a reference to app from server.py. server.py is a Flask app which needs a references to celery tasks from tasks.py. So dump_specs.py looks like:
#!/usr/bin/env python3
import impa.server
def dump_to_dir(dir_path):
# Do something
client = impa.server.app.test_client()
# Do the rest of things
impa/server.py looks like:
#!/usr/bin/env python3
import impa.tasks
app = Flask(__name__)
# Definitions of endpoints, some of them use celery tasks -
# that's why I need impa.tasks reference
And impa/tasks.py:
#!/usr/bin/env python3
from celery import Celery
import impa.server
def make_celery(app):
celery = Celery(app.import_name,
broker=app.config['CELERY_BROKER_URL'],
backend=app.config['CELERY_RESULT_BACKEND'])
TaskBase = celery.Task
class ContextTask(TaskBase):
abstract = True
def __call__(self, *args, **kwargs):
with app.app_context():
return TaskBase.__call__(self, *args, **kwargs)
celery.Task = ContextTask
return celery
celery = make_celery(impa.server.app)
When I'm trying to dump specs with ./dump_specs.py I've got an error:
./dump_specs.py specs
Traceback (most recent call last):
File "./dump_specs.py", line 9, in <module>
import impa.server
File "/build/impa/server.py", line 23, in <module>
import impa.tasks
File "/build/impa/tasks.py", line 81, in <module>
celery = make_celery(impa.server.app)
AttributeError: module 'impa' has no attribute 'server'
And I can't understand what's wrong. Could someone explain what's happening and how to get rid of this error?

If I have managed to reproduce your problem correctly on my host, it should help youto insert import impa.tasks into dump_specs.py above import impa.server.
The way your modules depend on each other, the loading order is important. IIRC (the loading machinery is described in greater details in the docs), when you first try to import impa.server, it will on line 23 try to import impa.tasks, but import of impa.server is not complete at this point. There is import impa.server in impa.tasks, but we do not got back and import it at this time (we'd otherwise end up in a full circle) and continue importing impa.tasts until we want to access impa.server.app, but we haven't gotten to the point we could do that yet, impa.server has not been imported yet.
When possible, it would also help if the code accessing another module in your package wasn't executed on import (directly called as part of the modules instead of being in a function or a class which would be called/used after the imports have completed).

Django-RQ: How to call function?

I am migrating a project to Django and like to use the django-rq module.
However, I am stuck at what to put here:
import django_rq
queue = django_rq.get_queue('high')
queue.enqueue(func, foo, bar=baz)
How to call func ? Can this be a string like path.file.function ?
Does the function need to reside in the same file?

Create tasks.py file to include
from django_rq import job
#job("high", timeout=600) # timeout is optional
def your_func():
pass # do some logic
and then in your code
import django_rq
from tasks import your_func
queue = django_rq.get_queue('high')
queue.enqueue(your_func, foo, bar=baz)

AttributeError - module is not available in threading.Thread

In Django I wrote a custom command, called update.
This command is continously called by a shell script and updates some values in the database. The updates are done in threads. I put the class doing all-threading related things in the same module as the custom command.
On my production server, when I try to run that shell script, I get an error when I try to access my models:
antennas = Antenna.objects.all()
The error is:
AttributeError: 'NoneType' object has no attribute 'objects'
As you can see, however, I did import app.models.Antenna in that file.
So to me, it seems as if the reference to the whole site is somehow "lost" inside the threading class.
site/app/management/commands/update.py
(I tried to remove all non-essential code here, as it would clutter everything, but I left the imports intact)
from django.core.management.base import NoArgsCommand, CommandError
from django.utils import timezone
from datetime import datetime, timedelta
from decimal import *
from django.conf import settings
from _geo import *
import random, time, sys, traceback, threading, Queue
from django.db import IntegrityError, connection
from app.models import Bike, Antenna
class Command(NoArgsCommand):
def handle_noargs(self, **options):
queue = Queue.Queue()
threadnum = 2
for bike in Bike.objects.filter(state__idle = False):
queue.put(bike)
for i in range(threadnum):
u = UpdateThread(queue)
u.setDaemon(True)
u.start()
queue.join()
return
class UpdateThread(threading.Thread):
def init(self, queue):
threading.Thread.init(self)
self.queue = queue
def run(self):
antennas = Antenna.objects.all()
while not self.queue.empty():
try:
[...]
except Exception:
traceback.print_exc(file=sys.stdout)
finally:
self.queue.task_done()
When I try to from app.models import Antenna inside the UpdateThread-class I get an error:
ImportError: No module named app.models
The site runs great on a webserver. No model-related problems there, only when I do anything inside a thread - all app.models-imports are failing.
Additionally, it seems to me as exactly the same configuration (I'm using git) does run on another machine, which runs the same operating system (debian wheezy, python 2.7.3rc2 & django 1.4.2).
Very probably I'm missing something obvious about threading here but I'm stuck for far too long now. Your help would be very much appreciated!
PS: I did check for circular imports - and those shouldn't happen anyway, as I can use my models in the other (management-)class, right?