We Keep Coding

dask worker ModuleNotFoundError when import is not in current directory

I have setup the file system as such:
\project
\something
__init__.py
some.py (with a function test() defined)
run.py
And my run.py looks like this:
import os
import sys
import dask
from dask.distributed import Client
from dask_jobqueue import SLURMCluster
import time
def run_task1():
sys.path.append('/project/something')
from some import test
return test()
def run_task2():
from something.some import test # because something dir is in the current working dir
return test()
def run_controller():
cluster = SlurmCluster(...)
cluster.scale_up(2)
client = Client(cluster)
sys.path.append('/project/something')
os.environ['PATH'].append('/project/something')
os.environ['PYTHONPATH'].append('/project/something')
from some import test
v1 = [
#dask.delayed(run_task1)() for _ in range(2) #<--- this works
#dask.delayed(run_task2)() for _ in range(2) #<--- this works too
dask.delayed(test)() for _ in range(2) #<--- fails, but I need to do this
]
values = dask.compute(*v1)
return values
values = run_controller()
And the error is that worker fails immediately as it could not run test() as it could not import it from some.py. I verified that dask worker's os.environ['PATH'], os.environ['PYTHONPATH'] and sys.path all have the added path to some.py, but the dask worker still could not run it. Below is the error logged in the slurm log.
'''
ModuleNotFoundError: No module named 'some'
distributed.worker - ERROR - Could not deserialize task
'''
I need to run the function directly, aka I cannot have a wrapper that executes an explicit import in the dask worker, but that is the method that does not work.
I have a hacky solution to get a solution like run_task2(), by creating a symlink to some.py in the current working dir. But I am wondering if there is a proper way to setup the environment of the dask worker so that a direct dask.delayed call on test() works.

Mocking code run inside an rq SimpleWorker

I have code which uses Python requests to kick off a task which runs in a worker that is started with rq. (Actually, the GET request results in one task which itself starts a second task. But this complexity shouldn't affect things, so I've left that out of the code below.) I already have a test which uses rq's SimpleWorker class to cause the code to run synchronously. This works fine. But now I'm adding requests_ratelimiter to the second task, and I want to be sure it's behaving correctly. I think I need to somehow mock the time.sleep() function used by the rate limiter, and I can't figure out how to patch it.
routes.py
#app.route("/do_work/", methods=["POST"])
def do_work():
rq_job = my_queue.enqueue(f"my_app.worker.do_work", job_timeout=3600, *args, **kwargs)
worker.py
from requests_ratelimiter import LimiterSession
#job('my_queue', connection=redis_conn, timeout=3600, result_ttl=24 * 60 * 60)
def do_work():
session = LimiterSession(per_second=1)
r = session.get(WORK_URL)
test.py
import requests_mock
def test_get(client):
# call the Flask function to kick off the task
client.get("/do_work/")
with requests_mock.Mocker() as m:
# mock the return value of the requests.get() call in the worker
response_success = {"result": "All good"}
m.get(WORK_URL, json=response_success)
worker = SimpleWorker([my_queue], connection=redis_conn)
worker.work(burst=True) # Work until the queue is empty
A test in requests_ratelimiter patches the sleep function using a target path of 'pyrate_limiter.limit_context_decorator.sleep', but that doesn't work for me because I'm not declaring pyrate_limiter at all. I've tried mocking the time function and then passing that into the LimiterSession, and that sort of works:
worker.py
from requests_ratelimiter import LimiterSession
from time import time
#job('my_queue', connection=redis_conn, timeout=3600, result_ttl=24 * 60 * 60)
def do_work():
session = LimiterSession(per_second=1, time_function=time)
r = session.get(WORK_URL)
test.py
import requests_mock
def test_get(client):
# call the Flask function to kick off the task
client.get("/do_work/")
with patch("my_app.worker.time", return_value=None) as mock_time:
with requests_mock.Mocker() as m:
response_success = {"result": "All good"}
m.get(URL, json=response_success)
worker = SimpleWorker([my_queue], connection=redis_conn)
worker.work(burst=True) # Work until the queue is empty
assert mock_time.call_count == 1
However, then I see time called many more times than sleep would be, so I don't get the info I need from it. And patching my_app.worker.time.sleep results in the error:
AttributeError: does not have the attribute 'sleep'
I have also tried patching the pyrate_limiter as the requests_ratelimiter testing code does:
with patch(
"my_app.worker.requests_ratelimiter.pyrate_limiter.limit_context_decorator.sleep", return_value=None
) as mock_sleep:
But this fails with:
ModuleNotFoundError: No module named 'my_app.worker.requests_ratelimiter'; 'my_app.worker' is not a package
How can I test and make sure the rate limiter is engaging properly?

The solution was indeed to use 'pyrate_limiter.limit_context_decorator.sleep', despite the fact that I wasn't importing it.
When I did that and made the mock return None, I discovered that sleep() was being called tens of thousands of times because it's in a while loop.
So in the end, I also needed to use freezegun and a side effect on my mock_sleep to get the behavior I wanted. Now time is frozen, but sleep() jumps the test clock forward synchronously and instantly by the amount of seconds passed as an argument.
from datetime import timedelta
from unittest.mock import patch
import requests_mock
from freezegun import freeze_time
from rq import SimpleWorker
def test_get(client):
with patch("pyrate_limiter.limit_context_decorator.sleep") as mock_sleep:
with freeze_time() as frozen_time:
# Make sleep operate on the frozen time
# See: https://github.com/spulec/freezegun/issues/47#issuecomment-324442679
mock_sleep.side_effect = lambda seconds: frozen_time.tick(timedelta(seconds=seconds))
with requests_mock.Mocker() as m:
m.get(URL, json=response_success)
worker = SimpleWorker([my_queue], connection=redis_conn)
worker.work(burst=True) # Work until the queue is empty
# The worker will do enough to get rate limited once
assert mock_sleep.call_count == 1

When using functions framework, and google.cloud.logging in Python, why do you have to do setup_logging inside the function?

Here's a tiny example. When running under functions framework on a GCE VM with an appropriate service account,
import logging
import google.cloud.logging
logging_client = google.cloud.logging.Client()
logging_client.setup_logging()
def test(request):
logging.warning("testing warning")
return "done"
does not log a warning message.
Changing it to:
import logging
import google.cloud.logging
logging_client = google.cloud.logging.Client()
def test(request):
logging_client.setup_logging()
logging.warning("testing warning")
return "done"
does produce the correct message.
And if I take the version that doesn't work, add a call to the function and run it normally without functions framework, it produces a warning as expected.
import logging
import google.cloud.logging
logging_client = google.cloud.logging.Client()
logging_client.setup_logging()
def test(request):
logging.warning("testing warning")
return "done"
test("foo")
Can someone explain what's going on, either with how the functions frameworks works, or how the logging handler works?
Thanks!

I had a similar issue using gcsfs with functions framework that provided the answer. It looks like when you use functions framework the function call runs in a different process from the initialization.
I mimicked this with:
import logging
import google.cloud.logging
import os
import time
logging_client = google.cloud.logging.Client()
logging_client.setup_logging()
def test(request):
logging.warning("testing warning")
return "done"
pid = os.fork()
if pid > 0:
time.sleep(10)
else:
test("foo")
and it does not work. Move the setup_logging to inside of the function call, and it does. So it seems that you have to do the setup_logging() in the subprocess for it to work, and that's why it has to be done in the function call rather than as part of initialization in function framework.

Mocking Method Call in Celery shared_task

I have a Celery shared_task in a module tasks that looks like this:
#shared_task
def task():
from core.something import send_it
send_it()
and I am writing a test attempting to patch the send_it method. So far I have:
from ..tasks import task
class TestSend(TestCase):
#override_settings(CELERY_TASK_ALWAYS_EAGER=True)
#patch("core.tasks.send_it")
def test_task(self, send_it_mock):
task()
send_it_mock.assert_called_once()
When I run this, I get the error: AttributeError: <module 'core.tasks' from 'app/core/tasks.py'> does not have the attribute 'send_it'
Out of desperation I've used #patch("tasks.task.send_it") instead, as the import happens inside the shared_task, but I get a similar result. Does anyone know how I can effectively patch the send_it call? Thanks!

How to test celery periodic_task in Django?

I have a simple periodic task:
from celery.decorators import periodic_task
from celery.task.schedules import crontab
from .models import Subscription
#periodic_task(run_every=crontab(minute=0, hour=0))
def deactivate_subscriptions():
for subscription in Subscription.objects.filter(is_expired=True):
print(subscription)
subscription.is_active = False
subscription.can_activate = False
subscription.save()
And I want to cover it with tests.
I found information about how to test simple tasks, like #shared_task, but nowhere can I find an example of testing #periodic_task

When having a periodic task defined with a decorator you can access the crontab configuration the following way:
tasks.py
#periodic_task(
run_every=(crontab(minute="*/1")),
)
def my_task():
pass
some file where you need to access it
from .tasks import my_task
crontab = my_task.run_every
hours_when_it_will_run = crontab.hour
minutes_when_it_will_run = crontab.minute
day_of_the_week_when_it_will_run = crontab.day_of_week
day_of_the_month_when_it_will_run = crontab.day_of_month
month_of_year_when_it_will_run = crontab.month_of_year
This way you can access when task will be executed and you check in your test if the expected time is there.

Use the function apply(). Documentation for apply().