I'm importing multiple python threads from different directories and then want to run them simultaneously.
Here's my parent:
import sys
import thread
sys.path.append('/python/loanrates/test')
import test2
thread.start_new_thread(test2.main())
and here's one of my child's:
import json
def main():
data = 'ello world'
print data
with open( 'D:/python/loanrates/test/it_worked.json', 'w') as f:
json.dump(data, f)
if __name__ == '__main__':
main()
but I am getting this error:
TypeError: start_new_thread expected at least 2 arguments, got 1
What is a simple way I can get this thread started (and then sequentially run multiple threads using the same method)
You also need to provide a tuple with the argument to run the function with. If you have none, pass an empty tuple.
thread.start_new_thread(test2.main, ())
From the docs of thread.start_new_thread(function, args[, kwargs]) (boldface mine):
Start a new thread and return its identifier. The thread executes the function function with the argument list args (which must be a tuple). The optional kwargs argument specifies a dictionary of keyword arguments. When the function returns, the thread silently exits. When the function terminates with an unhandled exception, a stack trace is printed and then the thread exits (but other threads continue to run).
You can also:
thread = Thread(target = test2.main, args, kwargs)
thread.start() // starts the thread
thread.join() // wait
Read more on this approach to creating and working with threads here.
Related
Hi I have a problem with multiprocessing in python 3.7
I've made a listener, that should be waiting for a response from a server without blocking the rest of the program (asynchronous communication):
self = cl.appendSend('bar', base_list)
print("client erstellt neuen nebenläufigen listener, für die Antwort des Servers")
multiprocessing.set_start_method("spawn")
queue = multiprocessing.Queue()
process = multiprocessing.Process(target = cl.appendResponse(), args=(self))
process.start()
print("listener aktiv")
thread = threading.Thread(target= waitingPrinter(), args=(process, queue))
print(thread)
is where everything is started
but the line process = multiprocessing.Process(target = cl.appendResponse(), args=(self)) is started once, runs through and then after being done, it just runs again. The debugger never leaves this line.
The method run in the process is:
def appendResponse(self):
print("nebenläufiger listener aktiv")
msgrcv = self.chan.receive_from(self.server)
print("nebenläufiger listener hat Antwort erhalten")
return msgrcv # pass it to caller
Sadly becaus of copyright I can't really post more, but the method runs through fine the first time and fails the second with the message :
Traceback (most recent call last):
> File "D:/Verteile Systeme 2/neues Lab/git/vs2lab/lab2/rpc/runcl.py",
> line 27, in <module>
> process = multiprocessing.Process(target = cl.appendResponse(), args=(self)) File "C:\Program Files
> (x86)\Python37-32\lib\multiprocessing\process.py", line 82, in
> __init__
> self._args = tuple(args) TypeError: 'Client' object is not iterable
So I am wondering, why is the process with cl.appendResponse() even started upon binding to the process and doesn't wait for process.start() and if not already in the answer to that, why does it then run directly a second time. And of course how can I fix that.
Also is there a way to replaces processing with thread and still get a return value?
I am having a lot of trouble with processing and return values.
target = cl.appendResponse() will run the function and return the result to target.
The correct syntax would be target=cl.appendResponse which will tell Process to run cl.appendResponse on start().
The cause of the the apparent immediate execution of the process has been correctly stated by philipp in their answer.
The target argument to Process takes a callable object, that is to be invoked by the run() method. Your code passes whatever is returned by self.chan.receive_from(self.server).
There is no subprocess running in or from the line process = multiprocessing.Process(target = cl.appendResponse(), args=(self)). Your method runs in the main process and blocks it.
On a side note: you will have the exact same issue with your thread, for the same reason: thread = threading.Thread(target= waitingPrinter(), args=(process, queue))
After your method has finished executing in the main process, the initialization of your process object raises the TypeError inside the __init__ method of the BaseProcess class.
You pass an argument, self, to your process, but you do it incorrectly. The args argument requires a tuple of arguments. The creation of a tuple through a literal needs a trailing comma if only a single value is specified: args=(self,). Your code effectively passes self, i.e. a Client object directly, which is not iterable and thus causes the error.In your case, appendResponse appears to be a bound method of the Client object. It will receive the self argument through the inner workings of Python's class system. Passing it explicitly through the process will raise another TypeError for passing two positional arguments to a method that only takes one. Unless appendSend returns something else than the Client instance cl, that you call it on, drop the args parameter in the process instantiation.
On another side note: the start method spawn is the only one available on Windows and thus the default. Unless your code needs to run under Unix using that start method, this line is redundant: multiprocessing.set_start_method("spawn")
So I'm having a though time wrapping my head around multiprocessing library and all the functionality. Basicly what I'm trying to accomplish is to start a separate process from a background thread that receives function object and it's positional and keyword arguments.
I have a thread that is started at the beginning and it's job is to execute functions that are passed to it via dependency injection. Once the thread detects that new job is scheduled it takes the job and executes it. The problem is that I have no idea how long that job will take and I would like to terminate it if let's say 10 minutes have passed. Since this can't be accomplished via threading module I decided to take a look at multiprocessing since it's processes can be terminated.
Dependency injection is solved via decorator that encapsulates each function (that is intended to be executed by the thread) that passes function object and it's positional and keyword arguments to the thread that's gonna execute it via * and **. The thread at the end gets all arguments and the function object (this works).
The problem begins when i try to create a Pool and assing work to a single worker. Since I have no idea of function input arguments, how am I able to use apply_async functin with * and **?
def intercept(callback):
def wrapper(*args, **kwargs):
# pass callback, args and kwargs to the thread
pass
return wrapper
#intercept
def do_some_work(first, second, third=None):
time.sleep(10)
def bg_thread():
while True:
# acquire callback, args and kwargs from intercept decorator
# if new job is scheduled create a process and execute it
# if process did not finish in timeout, terminate it
p = multiprocessing.Pool()
ret = p.apply_async(callback, args, kwargs)
p.close()
try:
ret.get(5)
except:
p.terminate()
t = threading.Thread(target=bg_thread)
t.start()
do_some_work()
I have some code that does the same thing to several files in a python 3 application and so seems like a great candidate for multiprocessing. I'm trying to use Pool to assign work to some number of processes. I'd like the code to continue do other things (mainly displaying things for the user) while these calculations are going on, so i'd like to use the map_async function of the multiprocessing.Pool class for this. I would expect that after calling this, the code will continue and the result will be handled by the callback I've specified, but this doesn't seem to be happening. The following code shows three ways I've tried calling map_async and the results I've seen:
import multiprocessing
NUM_PROCS = 4
def func(arg_list):
arg1 = arg_list[0]
arg2 = arg_list[1]
print('start func')
print ('arg1 = {0}'.format(arg1))
print ('arg2 = {0}'.format(arg2))
time.sleep(1)
result1 = arg1 * arg2
print('end func')
return result1
def callback(result):
print('result is {0}'.format(result))
def error_handler(error1):
print('error in call\n {0}'.format(error1))
def async1(arg_list1):
# This is how my understanding of map_async suggests i should
# call it. When I execute this, the target function func() is not called
with multiprocessing.Pool(NUM_PROCS) as p1:
r1 = p1.map_async(func,
arg_list1,
callback=callback,
error_callback=error_handler)
def async2(arg_list1):
with multiprocessing.Pool(NUM_PROCS) as p1:
# If I call the wait function on the result for a small
# amount of time, then the target function func() is called
# and executes sucessfully in 2 processes, but the callback
# function is never called so the results are not processed
r1 = p1.map_async(func,
arg_list1,
callback=callback,
error_callback=error_handler)
r1.wait(0.1)
def async3(arg_list1):
# if I explicitly call join on the pool, then the target function func()
# successfully executes in 2 processes and the callback function is also
# called, but by calling join the processing is not asynchronous any more
# as join blocks the main process until the other processes are finished.
with multiprocessing.Pool(NUM_PROCS) as p1:
r1 = p1.map_async(func,
arg_list1,
callback=callback,
error_callback=error_handler)
p1.close()
p1.join()
def main():
arg_list1 = [(5, 3), (7, 4), (-8, 10), (4, 12)]
async3(arg_list1)
print('pool executed successfully')
if __name__ == '__main__':
main()
When async1, async2 or async3 is called in main, the results are described in the comments for each function. Could any one explain why the different calls are behaving the way they are? Ultimately I'd like to call map_async as done in async1, so i can do something in else the main process while the worker processes are busy. I have tested this code with python 2.7 and 3.6, on an older RH6 linux box and a newer ubuntu VM, with the same results.
This is happening because when you use the multiprocessing.Pool as a context manager, pool.terminate() is called when you leave the with block, which immediately exits all workers, without waiting for in-progress tasks to finish.
New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().
IMO using terminate() as the __exit__ method of the context manager wasn't a great design choice, since it seems most people intuitively expect close() will be called, which will wait for in-progress tasks to complete before exiting. Unfortunately all you can do is refactor your code away from using a context manager, or refactor your code so that you guarantee you don't leave the with block until the Pool is done doing its work.
I want to run a function independently. From the function I call, I want return without waiting for the other function ending.
I tried with threadind, but this will wait, the end.
thread = threading.Thread(target=myFunc)
thread.daemon = True
thread.start()
return 'something'
Is it possible to return immediately and the other process still run?
Thanks for the Answers.
EDITED
The working code looks like:
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(2)
executor.submit(myFunc, arg1, arg2)
You are more or less asking the following question:
Is it possible to run function in a subprocess without threading or writing a separate file/script
You have to change the example code from the link like this:
from multiprocessing import Process
def myFunc():
pass # whatever function you like
p = Process(target=myFunc)
p.start() # start execution of myFunc() asychronously
print)'something')
p.start() is executed asychronously, i.e. 'something' is printed out immediately, no matter how time consuming the execution of myFunc() is. The script executes myFunc() and does not wait for it to finish.
if I understood your request correctly, you might want to take a look on worker queues
https://www.djangopackages.com/grids/g/workers-queues-tasks/
Basically it's not a good idea to offload the work to thread created in view, this is usually handled by having a pool of background workers (processes, threads) and the queue for incoming requests.
I think the syntax you are using is correct and I don't see why your request shouldn't return immediately. Did you verify the request actually hang till the thread is over?
I would suggest to set myFunc to write to a file for you to track this
def myFunc():
f = open('file.txt', 'w')
while True:
f.write('hello world')
Im trying to understand multiprocessing.Process class. I want to collect data asynchronously storing it somewhere. After having stored the data, it somehow gets lost. Here is my MWE:
from __future__ import print_function
import multiprocessing as mp
def append_test(tgt):
tgt.append(42)
print('Appended:', tgt)
l = []
p = mp.Process(target=lambda: append_test(l))
p.run()
print('l is', l)
p.start()
p.join()
print('l is', l)
If I'm running that snippet, I get
Appended: [42]
l is [42]
Appended: [42, 42]
l is [42]
As you can see, there is a difference between calling run and using start/join. It has nothing to do with the order (using run afterwards) - I've tried that. Can someone elaborate how the second 42 gets lost? It seems to be stored at some time? But at some other time its definetly not.
Just in case that could make any difference: I've tried python2.7 and python3.4, both with the exact same result described above.
Update: Apparently only start spawns a new process where run will be invoked afterwards. Then my actual problem translates to the following question: How do I pass l to the spawned process s.t. I can see the actual result?
Solution: The following example shows how to pass shared data safely to a Process:
from __future__ import print_function
import multiprocessing as mp
def append_test(tgt):
tgt.append(42)
print('Appended:', tgt)
m = mp.Manager()
l = m.list()
p = mp.Process(target=lambda: append_test(l))
p.start()
p.join()
print('l is', l)
Further reading: Multiprocessing Managers Documentation
From Python: Essential Reference by Beazley:
p.run(): The method that runs when the process starts. By default, this invokes target that was passed to the Process constructor. ...
p.start(): Starts the process. This launches the subprocess that represents the process and invokes p.run() in that subprocess.
So, they are not meant to be doing the same thing. It looks to me like in this case, p.run() is being invoked for the ongoing process and p.start() calls p.run() in a new process with the original target that was passed to the constructor (in which l is [ ] still).
Run executes the callable object that you target in multiprocessing. Start will call the run() method for the object.
From multiprocessing's documentation
run() Method representing the process’s activity.
You may override this method in a subclass. The standard run() method
invokes the callable object passed to the object’s constructor as the
target argument, if any, with sequential and keyword arguments taken
from the args and kwargs arguments, respectively.
start() Start the process’s activity.
This must be called at most once per process object. It arranges for
the object’s run() method to be invoked in a separate process.