We Keep Coding

How do I decorate an __init__ method with Click?

I frequently use the following pattern, where I create an instance of a Main object, typically for a batch, then call process() on it. All the necessary options are passed into the __init__. This means that, provided I create a Main with the appropriate parameters, I can run it from anywhere (celery being an example).
(I have learned to minimize in-celery debugging to the greatest extent to keep my sanity - the scripts are debugged fully from the command line first, and only then are Mains created and launched from celery)
This is my working Click approach:
Pretty simple. the decorated run function defines its arguments and passes them to Main.__init__.
Conceptually however, run and its docstring are very separated from that __init__. So on the __init__ I have to look at run to understand what to pass and when I --help the command line, I get the docstring for run.
import click
#click.command()
#click.option('--anoption')
def run(**kwargs):
mgr = Main(**kwargs)
mgr.process()
class Main:
def __init__(self, **kwargs):
self.__dict__.update(**kwargs)
def process(self):
print(f"{self}.process({vars(self)})")
if __name__ == "__main__":
run()
output:
<__main__.Main object at 0x10d5e42b0>.process({'anoption': None})
This is what I would like instead:
import click
class Main:
#click.command()
#click.option('--anoption')
def __init__(self, **kwargs):
self.__dict__.update(**kwargs)
def process(self):
print(f"{self}.process({vars(self)})")
if __name__ == "__main__":
main = Main(standalone_mode=False)
main.process()
But that gets me TypeError: __init__() missing 1 required positional argument: 'self'
Using a factory method
the closest I've come is with a factory staticmethod, but that's not great. At least the factory is sitting on the class, but there's still not much of a link to the init.
import click
class Main:
#staticmethod
#click.command()
#click.option('--anoption')
def factory(**kwargs):
return Main(**kwargs)
def __init__(self, **kwargs):
self.__dict__.update(**kwargs)
def process(self):
print(f"{self}.process({vars(self)})")
if __name__ == "__main__":
main = Main.factory(standalone_mode=False)
main.process()
output:
<__main__.Main object at 0x10ef59128>.process({'anoption': None})
What's a simple Pythonic way to bring click and that Main.__init__ closer together?

How to apply the multiprocessing module on my PyQt-GUI

I know this is an old question and have already found answers in other questions like this thread here. However, I have some problems applying it in my case.
The way I have things right now are the following: I have my MainWindow class where I can input some data. Then I have a Worker class which is a PySide2.QtCore.QThread object. To this class I pass some input data from the MainWindow. Inside this Worker class I have a method which sets up some ODEs, which in another method of the Worker class are being solved by scipy.integrate.solve_ivp. When the integration is done, I send the results via a signal back to the MainWindow. So the code roughly looks like this:
import PySide2
from scipy.integrate import solve_ivp
class Worker(QtCore.QThread):
def __init__(self,*args,**kwargs):
super(Worker,self).__init__()
"Here I collect input parameters"
def run(self):
"Here I call solve_ivp for the integration and send a signal with the
solution when it is done"
def ode_fun(self,t,c):
"Function where the ode equations are set up"
class Ui_MainWindow(QtWidgets.QMainWindow):
def __init__(self):
"set up the GUI"
self.btnStartSimulation.clicked.connect(self.start_simulation) #button to start the integration
def start_simulation(self):
self.watchthread(Worker)
self.thread.start()
def watchthread(self,worker):
self.thread = worker("input values")
"connect to signals from the thread"
Now I understand, that using the multiprocessing module I should be able to run the thread with the integration on another processor core to make it faster and make the GUI less laggy. However, from the link above I am not sure how I should apply this module or even how to restructure my code. Do I have to put the code that I now have in my Worker class into another class or am I somehow able to apply the multiprocessing module on my existing thread?
Any help is greatly appreciated!
Edit:
The new code looks like this:
class Worker(QtCore.QThread):
def __init__(self,*args,**kwargs):
super(Worker,self).__init__()
self.operation_parameters = args[0]
self.growth_parameters = args[1]
self.osmolality_parameters = args[2]
self.controller_parameters = args[3]
self.c_zero = args[4]
def run(self):
data = multiprocessing.Queue()
input_dict = {"function": self.ode_fun_vrabel_rushton_scaba_cont_co2_oxygen_biomass_metabol,
"time": [0, self.t_final],
"initial values": self.c_zero}
data.put(input_dict)
self.ode_process = multiprocessing.Process(target=self.multi_process_function, args=(data,))
self.ode_process.start()
self.solution = data.get()
def multi_process_function(self,data):
self.message_signal = True
input_dict = data.get()
solution = solve_ivp(input_dict["function"], input_dict["time"],
input_dict["initial values"], method="BDF")
data.put(solution)
def ode_fun(self,t,c):
"Function where the ode equations are set up"
(...) = self.operation_parameters
(...) = self.growth_parameters
(...) = self.osmolality_parameters
(...) = self.controller_parameters
Is it okay if I access the parameters in the ode_fun function via self."parameter_name"? Or do I also have to pass them with the data-parameter?
With the current code I receive the following error: TypeError: can't pickle Worker objects

You could call it from your worker like this:
import PySide2
from scipy.integrate import solve_ivp
import multiprocessing
class Worker(QtCore.QThread):
def __init__(self,*args,**kwargs):
super(Worker, self).__init__()
self.ode_process = None
"Here I collect input parameters"
def run(self):
"Here I call solve_ivp for the integration and send a signal with the solution when it is done"
data = multiprocessing.Queue()
data.put("all objects needed in the process, i would suggest a single dict from which you extract all data")
self.ode_process = multiprocessing.Process(target="your heavy duty function", args=(data,))
self.ode_process.start() # this is non blocking
# if you want it to block:
self.ode_process.join()
# make sure you remove all input data from the queue and fill it with the result, then to get it back:
results = data.get()
print(results) # or do with it what you want to do...
def ode_fun(self, t, c):
"Function where the ode equations are set up"
class Ui_MainWindow(QtWidgets.QMainWindow):
def __init__(self):
"set up the GUI"
self.btnStartSimulation.clicked.connect(self.start_simulation) #button to start the integration
def start_simulation(self):
self.watchthread(Worker)
self.thread.start()
def watchthread(self,worker):
self.thread = worker("input values")
"connect to signals from the thread"
Also beware that you would overwrite the running process now every time you press to start the simulation. You may want to use some sort of lock for that.

luigi - how to create a dependency not between files, but between tasks? (or how to not involve the output method)

Given two luigi tasks, how can I add one as a requirement for the other, in a way that if the required is done, the second task could start, with no output involved?
Currently I get RuntimeError: Unfulfilled dependency at run time: MyTask___home_... even though the task completed ok, because my requires / output methods are not configured right...
class ShellTask(ExternalProgramTask):
"""
ExternalProgramTask's subclass dedicated for one task with the capture output ability.
Args:
shell_cmd (str): The shell command to be run in a subprocess.
capture_output (bool, optional): If True the output is not displayed to console,
and printed after the task is done via
logger.info (both stdout + stderr).
Defaults to True.
"""
shell_cmd = luigi.Parameter()
requirement = luigi.Parameter(default='')
succeeded = False
def on_success(self):
self.succeeded = True
def requires(self):
return eval(self.requirement) if self.requirement else None
def program_args(self):
"""
Must be implemented in an ExternalProgramTask subclass.
Returns:
A script that would be run in a subprocess.Popen.
Args:
shell_cmd (luigi.Parameter (str)): the shell command to be passed as args
to the run method (run should not be overridden!).
"""
return self.shell_cmd.split()
class MyTask(ShellTask):
"""
Args: if __name__ == '__main__':
clean_output_files(['_.txt'])
task = MyTask(
shell_cmd='...',
requirement="MyTask(shell_cmd='...', output_file='_.txt')",
)
"""
pass
if __name__ == '__main__':
task_0 = MyTask(
shell_cmd='...',
requirement="MyTask(shell_cmd='...')",
)
luigi.build([task_0], workers=2, local_scheduler=False)
I hoped using the on_success could prompt something to the caller task, but I didn't figure out how to.
I'm currently overcoming this in the following way:
0) implement the output method based on the input of the task (much like the eval(requirement) I did
2) implement the run method (calling the super run and then writing "ok" to output
3) deleting the output files from main.
4) calling it somehitng like this:
if __name__ == '__main__':
clean_output_files(['_.txt'])
task = MyTask(
shell_cmd='...',
requirement="MyTask(shell_cmd='...', output_file='_.txt')",
)

So within your first luigi task, you could call your second Task within by making it a requirement.
For example:
class TaskB(luigi.Task):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.complete_flag = False
def run(self):
self.complete_flag = True
print('do something')
def complete(self):
return self.is_complete
class TaskA(luigi.Task):
def requires(self):
return TaskB()
def run(self):
print('Carry on with other logic')

how to decorate a function with decorators when a argument is passed or say a condition is matched

This is the simple decorators with arguments:
app.py
from __future__ import print_function
import time
def newdecorator(arg1):
def benchmarking(funct):
#The argument is accessible here
print("this is the value of argument",arg1)
def timercheck(*args, **kwarg):
starttime=time.time()
print("starting time",time.time())
funct(*args, **kwarg)
print("TOTAL TIME TAKEN ::",time.time()-starttime)
return timercheck
return benchmarking
#passing value to the decorators with arguments
#newdecorator('arg value')
def tara():
print("hellow testing")
if __name__=="__main__":
tara()
The decorators are working well. It decorates the function with a functionality to show the starting time an the time taken by it.
What I want to achieve:
I want the decorators to be conditionally implemented I mean.
when a user runs app.py let the decorators not be implemented
user runs app.py -t than want the decorators to be implemented
I will be using the argparse module for the argument parsing that is not a big problem what I am wondering is how to make the decorators work conditionally.

You can also use a global variable in the module or some other namespace, but that might only be useful if you want to use this state for something else as well. It's not uncommon to have application settings that configure decorator behaviour.
import time
TIMING_ENABLED = True
def newdecorator():
def benchmarking(funct):
def timercheck(*args, **kwarg):
if TIMING_ENABLED:
starttime = time.time()
print("starting time", time.time())
funct(*args, **kwarg)
if TIMING_ENABLED:
print("TOTAL TIME TAKEN ::", time.time() - starttime)
return timercheck
return benchmarking
# passing value to the decorators with arguments
#newdecorator()
def tara():
print("hellow testing")
if __name__ == "__main__":
TIMING_ENABLED = False
tara()
It's important to put the conditional checks inside the timercheck function because the lower scopes are executed during the module initialisation (before main() is executed) and we will not get the chance to set the TIMING_ENABLED variable.
If you only want this to be an on/off thing, then #Rawing's answer is the way to go.
Since you don't need any parameters to this specific decorator you can also simplify it. We apply the decorator without () and can drop one level of nesting. In addition we add the functools.wraps decorator to timecheck so this function will look like the tara function to python. (See additional prints in main) since what the decorator does is actually replacing the tara function with timecheck.
from __future__ import print_function
import time
from functools import wraps
TIMING_ENABLED = True
def newdecorator(funct):
#wraps(funct)
def timercheck(*args, **kwarg):
if TIMING_ENABLED:
starttime=time.time()
print("starting time",time.time())
funct(*args, **kwarg)
if TIMING_ENABLED:
print("TOTAL TIME TAKEN ::",time.time()-starttime)
return timercheck
#newdecorator
def tara():
"""Docs for tara function"""
print("hellow testing")
if __name__=="__main__":
TIMING_ENABLED = True
print(tara.__name__, tara.__doc__)
tara()
You can always remove the #wraps(funct) line and see what main will print out differently about the tara function.

The #decorator syntax is just syntactic sugar. Under the hood, all that happens is that the decorator is called with the decorated function as its argument.
So you can just define your function without a decorator and then apply the decorator if a condition is met:
def tara():
print("hellow testing")
if script_called_with_t_flag:
tara = newdecorator('arg value')(tara)

you can use sys module to run Python script using arguments in windows command line
Run this code in terminal with python app.py True to enable decorator and for disable decorator python app.py or python app.py False
import time
import sys
TIMING_ENABLED = True
def newdecorator():
def benchmarking(funct):
def timercheck(*args, **kwarg):
if TIMING_ENABLED:
starttime = time.time()
print("starting time", time.time())
funct(*args, **kwarg)
if TIMING_ENABLED:
print("TOTAL TIME TAKEN ::", time.time()-starttime)
return timercheck
return benchmarking
# passing value to the decorators with arguments
#newdecorator()
def tara():
print("hellow testing")
if __name__ == "__main__":
TIMING_ENABLED = sys.argv[1]
tara()

One way to do this would be to conditionally apply the decorator only if the script was called with -t and otherwise do nothing:
import time
script_called_with_t = True
def newdecorator(arg1):
if script_called_with_t:
def benchmarking(funct):
#The argument is accessible here
print("this is the value of argument",arg1)
def timercheck(*args, **kwarg):
starttime=time.time()
print("starting time",time.time())
funct(*args, **kwarg)
print("TOTAL TIME TAKEN ::",time.time()-starttime)
return timercheck
return benchmarking
else:
return lambda funct: funct
#passing value to the decorators with arguments
#newdecorator('arg value')
def tara():
print("hellow testing")
if __name__ == "__main__":
TIMING_ENABLED = False
tara()

Sending arguments to Python threads blocks other threads

Consider this test application for passing arguments to Python threads:
#!/usr/bin/python3
from pprint import pprint
import signal
import sys
import threading
class CallThreads(threading.Thread):
def __init__(self, target, *args):
self._target = target
threading.Thread.__init__(self)
target(*args)
def main(argv):
phrases = ['hello', 'goodbye']
num = 0
for phrase in phrases:
num += 1
thread_handler = CallThreads(someFunction, phrase, num)
thread_handler.daemon = True
thread_handler.start()
return True
def someFunction(a, b):
print("Hi: "+str(a)+" and "+str(b))
return True
def signal_handler(signal, frame):
print(["Got SIGINT!"])
sys.exit(0)
if __name__ == '__main__':
signal.signal(signal.SIGINT, signal_handler)
main(sys.argv)
In it's current state, it seems as though the for phrase in phrases loop is waiting for the thread to finish before starting another thread. That is, if someFunction() takes a long time to complete, then the then next thread will not start until the previous thread returns. Why is this, and how can I work around it while still sending arguments to the threads?
Edit:
I've tried saving the args array in self._args in the constructor, and then calling self._target(*self._args) or self._target(self._args) in def run (self):. This actually works in Python 2, but not in Python 3. What should I do?
Edit:
It seems that the problem is that in Python 3, the run method cannot access the private variables. That is, for the following improved code:
def __init__(self, target, *args):
self._args = args
threading.Thread.__init__(self)
def run(self):
someFunction(*self._args)
Note the following output:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.3/threading.py", line 639, in _bootstrap_inner
self.run()
File "./test.py", line 19, in run
someFunction(*self._args)
TypeError: someFunction() missing 2 required positional arguments: 'a' and 'b'
And adding a pprint(self._args) to the run() method indeed shows that the tuple returned is empty. However, changing the variables to non-private works! The following code runs fine:
def __init__(self, target, *args):
self.target = target
self.args = args
threading.Thread.__init__(self)
def run(self):
self.target(*self.args)
Therefore, I can use the application with public variables in Python 3. However, is there any way to use private variables in the CallThreads class, as in Python 2?
Thanks!

The problem is that you call target(args) on the constructor of CallThreads.
Therefore the following call blocks until CallThreads.__init__() is finished:
thread_handler = CallThreads(someFunction, phrase, num)
Update:
One possibility could be the following:
class CallThreads(threading.Thread):
def __init__(self, *args):
self._args = args
threading.Thread.__init__(self)
def run(self):
someFunction(*self._args)