We Keep Coding

How to catch task arguments with default values in Celery signal?

I use Celery task_postrun signal to execute some logic after a task my_task has been executed. The Celery task receives arguments from a dictionary with the use of ** to unpack them.
The problem is that args inside Celery signal only contain argument without a default value, and I guess the use dictionary with ** to unpack key:value is like arguments with default values.
Task
#app.task(name='Task_example')
def my_task(arg1, arg2):
...
Signal
#task_postrun.connect
def task_postrun_handler(task_id=None, task=None, args=None, state=None, retval=None, **kwargs):
if task.name == 'Task_example':
log.info(len(args))
Now when I execute the task I'm unable to catch arguments passed with the dic. How can I do that without unpacking everything ?
kwargs = dict(arg1=1, arg2=2)
my_task.delay(**kwargs)
It says there is 0 argument in args, and when I execute the task with:
kwargs = dict(arg2=2)
my_task.delay(arg1, **kwargs)
Then it finds arg1 but not arg2.

Run a function only once during runtime in python

There is a requirement to load some data into memory. To do this, I need to ensure that my function that does this is run only once at runtime no matter how many times it is called.
I'm using a decorator to do this in a thread-safe manner.
Here's the code I'm using:
import threading
# Instantiating a lock object
# This will be used to ensure that multiple parallel threads will not be able to run the same function at the same time
# in the #run_once decorator written below
__lock = threading.Lock()
def run_once(f):
"""
Decorator to run a function only once.
:param f: function to be run only once during execution time despite the number of calls
:return: The original function with the params passed to it if it hasn't already been run before
"""
def wrapper(*args, **kwargs):
"""
The actual wrapper where the business logic to call the original function resides
:param args:
:param kwargs:
:return: The original function unless the wrapper has been run already
"""
if not wrapper.has_run:
with __lock:
if not wrapper.has_run:
wrapper.has_run = True
return f(*args, **kwargs)
wrapper.has_run = False
return wrapper
Do I need to do a double check on the has_run flag once outside and once inside the lock so that a read is not being done on a stale object ?

In Celery+Python, how do I access to task parameters from the task_revoked function?

I'm trying to get to clean up some stuff after i kill a running task within celery. I'm currently hitting 2 problems:
1) Inside the task revoked function body, how can i get access to the parameters that the task function was called: so for example if the task is defined as:
#app.task()
def foo(bar, baz):
pass
How will i get access to bar and baz inside the task_revoked.connect code?
2) I want to kill a task only when it's state is anything but X. That means inspecting the task on one hand, and setting the state on the other. Inspecting the state could be done I guess, but I'm having difficulty getting my head around the context inside the task function body.
If I define foo like this:
#app.task(bound=True)
def foo(self, bar, baz):
pass
and call it from say.... Flask like foo(bar, baz), then I'll get an error that the third parameter is expected, which means the decorator does not add any context automatically through the self parameter.
the app is simply defined as celery.Celery()
Thanks in advance

You can get tasks args from request object.
from celery.signals import task_revoked
#task_revoked.connect
def my_task_revoked_handler(sender=None, body=None, *args, **kwargs):
print(kwargs['request'].args)
This prints arguments given to the task.
Update:
You have to use bind not bound.
#app.task(bind=True)
def foo(self, bar, baz):

How to toggle the action of a decorator in Python3

I have a decorator #newthread which wraps functions to run in a separate thread (using wraps from functools and Thread from threading). However, there are some functions for which I only want this to happen some of the time.
At the moment, I have #newthread check the keyword arguments of the function to be wrapped and if it finds a bool new_thread equal to True then it runs the function in a separate thread, otherwise it runs the function normally. For example,
#newthread
def foo(new_thread=False)
# Do stuff...
foo() # Runs normally
foo(new_thread=True) # Runs in new thread
Is this the canonical way of doing this, or am I missing something?

Don't use newthread as a decorator, then. A decorator is just a function that takes a function and returns a function.
If you want it to run in the current thread, call
foo(some, params)
If you want to run foo in a new thread, call
newthread(foo)(some, params)

#newthread
def foo(new_thread=False)
# Do stuff...
foo() # Runs normally
foo(new_thread=True) # Runs in new thread
That is good - but, I for one, would prefer to have the decorator do consume the "new_thread" argument, instead of having it showing on the parameter list of the decorated functions.
Also, you could use a "default" value so that you'd pick the actual need to use a different thread from somewhere else (like an enviroment variable):
MARKER = object()
def newthread(func):
def wrapper(*args, newthread=MARKER, **kwargs):
if newthread is MARKER:
newthread = os.environ.get("force_threads", True)
if newthread:
...
# cretae new thread and return future-like object
else:
return func(*args, **kwargs)
return wrapper

Parallel processing loop using multiprocessing Pool

I want to process a large for loop in parallel, and from what I have read the best way to do this is to use the multiprocessing library that comes standard with Python.
I have a list of around 40,000 objects, and I want to process them in parallel in a separate class. The reason for doing this in a separate class is mainly because of what I read here.
In one class I have all the objects in a list and via the multiprocessing.Pool and Pool.map functions I want to carry out parallel computations for each object by making it go through another class and return a value.
# ... some class that generates the list_objects
pool = multiprocessing.Pool(4)
results = pool.map(Parallel, self.list_objects)
And then I have a class which I want to process each object passed by the pool.map function:
class Parallel(object):
def __init__(self, args):
self.some_variable = args[0]
self.some_other_variable = args[1]
self.yet_another_variable = args[2]
self.result = None
def __call__(self):
self.result = self.calculate(self.some_variable)
The reason I have a call method is due to the post I linked before, yet I'm not sure I'm using it correctly as it seems to have no effect. I'm not getting the self.result value to be generated.
Any suggestions?
Thanks!

Use a plain function, not a class, when possible. Use a class only when there is a clear advantage to doing so.
If you really need to use a class, then given your setup, pass an instance of Parallel:
results = pool.map(Parallel(args), self.list_objects)
Since the instance has a __call__ method, the instance itself is callable, like a function.
By the way, the __call__ needs to accept an additional argument:
def __call__(self, val):
since pool.map is essentially going to call in parallel
p = Parallel(args)
result = []
for val in self.list_objects:
result.append(p(val))

Pool.map simply applies a function (actually, a callable) in parallel. It has no notion of objects or classes. Since you pass it a class, it simply calls __init__ - __call__ is never executed. You need to either call it explicitly from __init__ or use pool.map(Parallel.__call__, preinitialized_objects)