My module, which is also a script, calls some internally defined functions that use multiprocessing.
Running the module as a script works just fine on Windows and Linux. Calling its main function from another python script works fine on Linux but not on Windows.
The core, multi-processed function (the function passed to the multiprocessing.Process constructor as the target) never gets executed when my module calls the Process's start() function.
The module must be doing something too demanding for this usage (multiprocessing on Windows when called from a script), but how can I get to the source of this problem?
Here's some example code to demonstrate the behavior. First the module:
# -*- coding: utf-8 -*-
'my_mp_module.py'
import argparse
import itertools
import Queue
import multiprocessing
def meaty_function(**kwargs):
'Do a meaty calculation using multiprocessing'
task_values = kwargs['task_values']
# Set up a queue of tasks to perform, one for each element in the task_values array
in_queue = multiprocessing.Queue()
out_queue = multiprocessing.Queue()
reduce(lambda a, b: a or b,
itertools.imap(in_queue.put, enumerate(task_values)))
core_procargs=(
in_queue ,
out_queue,
)
core_processes = [multiprocessing.Process(target=_core_function,
args=core_procargs) for ii in xrange(len(task_values))]
for p in core_processes:
p.daemon = True # I've tried both ways, setting this to True and False
p.start()
sum_of_results = 0
for result_count in xrange(len(task_values)):
a_result = out_queue.get(block=True)
sum_of_results += a_result
for p in core_processes:
p.join()
return sum_of_results
def _core_function(inp_queue, out_queue):
'Perform the core calculation for each task in the input queue, placing the results in the output queue'
while 1:
try:
task_idx, task_value = inp_queue.get(block=False)
# Perform a calculation with this task value.
task_result = task_idx + task_value # The real calculation is more complicated than this
out_queue.put(task_result)
except Queue.Empty:
break
def get_command_line_arguments(command_line=None):
'parse the given command_line (list of strings) or from sys.argv, return the corresponding argparse.Namespace object'
aparse = argparse.ArgumentParser(description=__doc__)
aparse.add_argument('--task_values', '-t',
action='append',
type=int,
help='''The value for each task to perform.''')
return aparse.parse_args(args=command_line)
def main(command_line=None):
'perform a meaty calculation with the input from the command line, and print the results'
# collect input from the command line
args=get_command_line_arguments(command_line)
keywords = vars(args)
# perform a meaty calculation with the input
meaty_results = meaty_function(**keywords)
# display the results
print(meaty_results)
if __name__ == '__main__':
multiprocessing.freeze_support()
main(command_line=None)
Now the script that calls the module:
# -*- coding: utf-8 -*-
'my_mp_script.py:'
import my_mp_module
import multiprocessing
multiprocessing.freeze_support()
my_mp_module.main(command_line=None)
Running the module as a script gives the expected results:
C:\Users\greg>python -m my_mp_module -t 0 -t 1 -t 2
6
But running another script that simply calls the module's main() function gives an error message under Windows (here I stripped out the error message duplicated from each of the multiple processes):
C:\Users\greg>python my_mp_script.py -t 0 -t 1 -t 2
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\greg\AppData\Local\Continuum\anaconda2-64\lib\multiprocessing\forking.py", line 380, in main
prepare(preparation_data)
File "C:\Users\greg\AppData\Local\Continuum\anaconda2-64\lib\multiprocessing\forking.py", line 510, in prepare
'__parents_main__', file, path_name, etc
File "C:\Users\greg\Documents\PythonCode\Scripts\my_mp_script.py", line 7, in <module>
my_mp_module.main(command_line=None)
File "C:\Users\greg\Documents\PythonCode\Lib\my_mp_module.py", line 72, in main
meaty_results = meaty_function(**keywords)
File "C:\Users\greg\Documents\PythonCode\Lib\my_mp_module.py", line 28, in meaty_function
p.start()
File "C:\Users\greg\AppData\Local\Continuum\anaconda2-64\lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "C:\Users\greg\AppData\Local\Continuum\anaconda2-64\lib\multiprocessing\forking.py", line 258, in __init__
cmd = get_command_line() + [rhandle]
File "C:\Users\greg\AppData\Local\Continuum\anaconda2-64\lib\multiprocessing\forking.py", line 358, in get_command_line
is not going to be frozen to produce a Windows executable.''')
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.
This probably means that you are on Windows and you have
forgotten to use the proper idiom in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce a Windows executable.
Linux and Windows work a little differently in the way they create additional processes. Linux forks the code but Windows creates a new Python interpreter to run the spawned process. The effect here is that all your code gets re-loaded just as if it were the first time. There is a similar question that might be informative to look at see... How to stop multiprocessing in python running for the full script.
The solution here is to modify the my_mp_script.py script so the call to my_mp_module.main() is guarded like so..
import my_mp_module
import multiprocessing
if __name__ == '__main__':
my_mp_module.main(command_line=None)
Note that I've also removed the freeze_support() functions for now, however those may be acceptable to put back in if needed.
Related
I have a program that uses many asynchronous functions that are important for its general operation, but I need one of them to execute 2 processes at the same time.
For this I decided to use the multiprocess, creating and starting the process p1 and the process p2.
p1 will be a kind of timer that will count 8 seconds and that will be the time limit that the process p2 will have to finish your tasks, but if p2 does not complete its tasks before 8 seconds (that is, p1 finishes before p2) then both processes are closed, and the program continues.
In this case p2 is a process that does operations with strings, and encodes the content of the variable called text, returning it to the main program after finishing. On the other hand, p1 is a process whose goal is to limit the time that process p2 has to perform string operations and load the result into the text variable.
The objective of this algorithm would be to avoid the possible case that if a user sent super long operations (or in a possible case that the system was saturated to process that information in a reasonable time) that took more than 8 seconds, then p2 would stop doing them and the content of the variable text would not change.
This is my simplified code, where I try to make 2 threads (called p1 and p2) each one executing a different function, but both functions are within the same class, and I am calling this class from an asynchronous function:
import asyncio, multiprocessing, time
#class BotBrain(BotModule):
class BotBrain:
# real __init__()
'''
def __init__(self, chatbot: object) -> None:
self.chatbot = chatbot
self.max_learn_range = 4
corpus_name = self.chatbot.config["CorpusName"]
self.train(corpus_name)
'''
# test __init__()
def __init__(self):
self.finish_state = multiprocessing.Event()
def operation_process(self, input_text, text):
#text = name_identificator(input_text, text)
text = "aaa" #for this example I change the info directly without detailing the nlp process
return input_text, text
def finish_process(self, finish_state):
time.sleep(8)
print("the process has been interrupted!")
self.finish_state.set()
def process_control(self, finish_state, input_text, text):
p1 = multiprocessing.Process(target=self.finish_process, args=(finish_state,))
p1.start()
p2 = multiprocessing.Process(target=self.operation_process, args=(input_text, text,))
p2.start()
#close all unfinished processes, p1 or p2
finish_state.wait()
for process in [p1, p2]:
process.terminate()
def process(self, **kwargs) -> str:
input_text, text = "Hello, how are you?", ""
text = self.process_control(self.finish_state, input_text, text) #for multiprocessing code
async def main_call():
testInstance = BotBrain() #here execute the instructions of the __init__() function
testInstance.process()
asyncio.run(main_call())
But it is giving me these errors, that I do not know how to solve them:
File "J:\async_and_multiprocess.py", line 46, in process_control
p1.start()
File "C:\Users\MIPC\anaconda3\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\MIPC\anaconda3\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\MIPC\anaconda3\lib\multiprocessing\context.py", line 327, in _Popen
File "C:\Users\MIPC\anaconda3\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Users\MIPC\anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
return Popen(process_obj)
File "C:\Users\MIPC\anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\MIPC\anaconda3\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
prep_data = spawn.get_preparation_data(process_obj._name)
File "C:\Users\MIPC\anaconda3\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "C:\Users\MIPC\anaconda3\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
_check_not_importing_main()
File "C:\Users\MIPC\anaconda3\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
What does freeze_support() mean? And where in the class should I put that?
This is the flowchart of how my program should work, though it doesn't consider freeze_support() because I don't know what you mean by that.
Multiprocessing can be tricky, in part because of Python's import system. From the programming guidelines of the multiprocessing module:
Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).
This is what is happening here. Importing your main module executes asyncio.run(main_call()) which in turn would spawn another subprocess and so on.
You need to guard your asyncio.run from being exectuted on import with:
if __name__ == "__main__":
asyncio.run(main_call())
If you don't want to produce a frozen binary (e.g. with pyinstaller) you can ignore the part with the freeze_support().
This is the exact code from Python.org. If you comment out the time.sleep(), it crashes with a long exception traceback. I would like to know why.
And, I do understand why Python.org included it in their example code. But artificially creating "working time" via time.sleep() shouldn't break the code when it's removed. It seems to me that the time.sleep() is affording some sort of spin up time. But as I said, I'd like to know from people who might actually know the answer.
A user comment asked me to fill in more details on the environment this was happening in. It was on OSX Big Sur 11.4. Using a clean install of Python 3.95 from Python.org (no Homebrew, etc). Run from within Pycharm inside a venv. I hope that helps add to understanding the situation.
import time
import random
from multiprocessing import Process, Queue, current_process, freeze_support
#
# Function run by worker processes
#
def worker(input, output):
for func, args in iter(input.get, 'STOP'):
result = calculate(func, args)
output.put(result)
#
# Function used to calculate result
#
def calculate(func, args):
result = func(*args)
return '%s says that %s%s = %s' % \
(current_process().name, func.__name__, args, result)
#
# Functions referenced by tasks
#
def mul(a, b):
#time.sleep(0.5*random.random()) # <--- time.sleep() commented out
return a * b
def plus(a, b):
#time.sleep(0.5*random.random()). # <--- time.sleep() commented out
return a + b
#
#
#
def test():
NUMBER_OF_PROCESSES = 4
TASKS1 = [(mul, (i, 7)) for i in range(20)]
TASKS2 = [(plus, (i, 8)) for i in range(10)]
# Create queues
task_queue = Queue()
done_queue = Queue()
# Submit tasks
for task in TASKS1:
task_queue.put(task)
# Start worker processes
for i in range(NUMBER_OF_PROCESSES):
Process(target=worker, args=(task_queue, done_queue)).start()
# Get and print results
print('Unordered results:')
for i in range(len(TASKS1)):
print('\t', done_queue.get())
# Add more tasks using `put()`
for task in TASKS2:
task_queue.put(task)
# Get and print some more results
for i in range(len(TASKS2)):
print('\t', done_queue.get())
# Tell child processes to stop
for i in range(NUMBER_OF_PROCESSES):
task_queue.put('STOP')
if __name__ == '__main__':
freeze_support()
test()
This is the traceback if it helps anyone:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
Traceback (most recent call last):
File "<string>", line 1, in <module>
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Here's a technical breakdown.
This is a race condition where the main process finishes, and exits before some of the children have a chance to fully start up. As long as a child fully starts, there are mechanisms in-place to ensure they shut down smoothly, but there's an unsafe in-between time. Race conditions can be very system dependent, as it is up to the OS and the hardware to schedule the different threads, as well as how fast they chew through their work.
Here's what's going on when a process is started... Early on in the creation of a child process, it registers itself in the main process so that it will be either joined or terminated when the main process exits depending on if it's daemonic (multiprocessing.util._exit_function). This exit function was registered with the atexit module on import of multiprocessing.
Also during creation of the child process, a pair of Pipes are opened which will be used to pass the Process object to the child interpreter (which includes what function you want to execute and its arguments). This requires 2 file handles to be shared with the child, and these file handles are also registered to be closed using atexit.
The problem arises when the main process exits before the child has a chance to read all the necessary data from the pipe (un-pickling the Process object) during the startup phase. If the main process first closes the pipe, then waits for the child to join, then we have a problem. The child will continue spinning up the new python instance until it gets to the point when it needs to read in the Process object containing your function and arguments it should run. It will try to read from a pipe which has already been closed, which is an error.
If all the children get a chance to fully start-up you won't see this ever, because that pipe is only used for startup. Putting in a delay which will in some way guarantee that all the children have some time to fully start up is what solves this problem. Manually calling join will provide this delay by waiting for the children before any of the atexit handlers are called. Additionally, any amount of processing delay means that q.get in the main thread will have to wait a while which also gives the children time to start up before closing. I was never able to reproduce the problem you encountered, but presumably you saw the output from all the TASKS (" Process-1 says that mul(19, 7) = 133 "). Only one or two of the child processes ended up doing all the work, allowing the main process to get all the results, and finish up before the other children finished startup.
EDIT:
The error is unambiguous as to what's happening, but I still can't figure how it happens... As far as I can tell, the file handles should be closed when calling _run_finalizers() in _exit_function after joining or terminating all active_children rather than before via _run_finalizers(0)
EDIT2:
_run_finalizers will seemingly actually never call Popen.finalizer to close the pipes, because exitpriority is None. I'm very confused as to what's going on here, and I think I need to sleep on it...
Apparently #user2357112supportsMonica was on the right track. It totally solves the problem if you join the processes before exiting the program. Also #Aaron's answer has the deep knowledge as to why this fixes the issue!
I added the following bits of code as was suggested and it totally fixed the need to have time.sleep() in there.
First I gathered all the processes when they were started:
processes: list[Process] = []
# Start worker processes
for i in range(NUMBER_OF_PROCESSES):
p = Process(target=worker, args=(task_queue, done_queue))
p.start()
processes.append(p)
Then at the end of the program I joined them as follows:
# Join the processes
for p in processes:
p.join()
Totally solved the issues. Thanks for the advice.
I want to execute a Python file from another Python file and show all print() outputs and error outputs without waiting (realtime).
The simplified version of my code is as follows and I would like to show "start" and an error message without waiting for "end" (the end of the script).
def main():
# Function that takes a long time (in my actual code)
x += 1 # this raises an error
if __name__ == "main":
print("start")
main()
print("end")
I also have run.py:
import subprocess
def run():
subprocess.run(["python", "main.py"])
if __name__ == '__main__':
run()
I tried this blog post and several other similar answers on stackoverflow, but none of them worked, so I decided to put my original code here, which is above.
The following seems to work for me (on Windows). It uses subprocess.Popen() to execute the other script because this gives more control over what goes on. It turns buffering off to eliminate any delays that might cause, plus it redirects stderr to stdout to so all output can be retrieved from a single source.. Also note it also includes the correction #Ketan Mukadam mentions is his answer tregarding the value of __name__ in your first script.
main_script.py:
def main():
# Function that takes a long time (in my actual code)
x += 1 # this raises an error
if __name__ == '__main__':
print("start")
main()
print("end")
run.py:
import subprocess
import sys
def run():
kwargs = dict(bufsize=0, # No buffering.
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, # Redirect stderr to stdout.
universal_newlines=True)
args = [sys.executable, 'main_script.py']
with subprocess.Popen(args, **kwargs).stdout as output:
for line in output:
print(line, end='') # Process the output...
if __name__ == '__main__':
run()
Output from executing run.py:
start
Traceback (most recent call last):
File "main_script.py", line 10, in <module>
main()
File "main_script.py", line 6, in main
x += 1 # this raises an error
UnboundLocalError: local variable 'x' referenced before assignment
Is this line a mistake?
if __name__ == "main":
The symbol is __main__ set by interpreter and not main. It is possible that because of this typo error no code is running from main script. Try first executing the main script directly on command shell.
I wish to have a process continually monitoring RPi input, and set a variable (I have chosen a queue) to True or False to reflect the debounced value. Another process will then capture an image (from a stream). I have written some code just to check I can get multiprocessing and signalling (the queue) working ok (I'm an amature coder...).
It all works fine with threading, but multiprocessing is giving an odd error. Specifically 'multiprocessing, EOFError: EOF when reading a line'. Code outputs:-
this computer has the following number of CPU's 6
OK, started thread on separate processor, now we monitor variable
enter something, True is the key word:
Process Process-1:
Traceback (most recent call last):
File "c:\Python34\lib\multiprocessing\process.py", line 254, in _bootstrap
self.run()
File "c:\Python34\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Peter\Documents\NetBeansProjects\test_area\src\test4.py", line 16, in Wait4InputIsTrue
ValueIs = input("enter something, True is the key word: ")
EOFError: EOF when reading a line
This module monitors the 'port' (I am using the keyboard as an input):
#test4.py
from time import sleep
from multiprocessing import Lock
def Wait4InputIsTrue(TheVar, TheLock):
while True:
sleep(0.2)
TheLock.acquire()
#try:
ValueIs = input("enter something, True is the key word: ")
#except:
# ValueIs = False
if ValueIs == "True":
TheVar.put(True)
print("changed TheVar to True")
TheLock.release()
This module monitors the status, and acts on it:
#test5.py
if __name__ == "__main__":
from multiprocessing import Process, Queue, Lock, cpu_count
from time import sleep
from test4 import Wait4InputIsTrue
print("this computer has the following number of CPU's", cpu_count())
LockIt = Lock()
IsItTrue = Queue(maxsize = 3)
Wait4 = Process(target = Wait4InputIsTrue, args = (IsItTrue, LockIt))
Wait4.start()
print("OK, started thread on separate processor, now we monitor variable")
while True:
if IsItTrue.qsize():
sleep(0.1)
print("received input from separate thread:", IsItTrue.get())
Note that I have tried adding a try: to the input statement in test4.py, in which case it keeps printing "enter something, True is the key word: " indefinitely, without a cr.
I added Lock in wild attempt to fix it, makes no difference
Anyone any idea why this is happening?
Your problem can be boiled down to a simpler script:
import multiprocessing as mp
import sys
def worker():
print("Got", repr(sys.stdin.read(1)))
if __name__ == "__main__":
process = mp.Process(target=worker)
process.start()
process.join()
When run, it produces
$ python3 i.py
Got ''
Reading zero bytes means the pipe is closed and input(..) turns that into an EOFError exception.
The multiprocessing module doesn't let you read stdin. That makes sense generally because mixing stdin readers from multiple children is a risky business. In fact, digging into the implementation, multiprocessing/process.py explicitly sets stdin to devnull:
sys.stdin.close()
sys.stdin = open(os.devnull)
If you are just using stdin for test, then the solution is simple: Don't do that! If you really need user input, life is quite a bit more difficult. You can use additional queues plus code in the parent to prompt users and get input.
I'm having a problem with python 2.7 multiprocessing (64 bit windows). Suppose I have a file pathfinder.py with the code:
import multiprocessing as mp
class MWE(mp.Process):
def __init__(self, n):
mp.Process.__init__(self)
self.daemon = True
self.list = []
for i in range(n):
self.list.append(i)
def run(self):
print "I'm running!"
if __name__=='__main__':
n = 10000000
mwe = MWE(n)
mwe.start()
This code executes fine for arbitrarily large values of n. However if I then import and run a class instance in another file
from pathfinder import MWE
mwe = MWE(10000)
mwe.start()
I get the following traceback if n >= ~ 10000:
Traceback (most recent call last):
File <filepath>, in <module>
mwe.start()
File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "C:\Python27\lib\multiprocessing\forking.py", line 280, in __init__
to_child.close()
IOError: [Errno 22] Invalid argument
I thought this might be some sort of race condition bug, but using time.sleep to delay mwe.start() doesn't appear to affect this behavior. Does anyone know why this is happening, or how to get around it?
The problem is with how you use multiprocessing in Windows. When importing a module that defines a Process class, e.g.:
from pathfinder import MWE
you must encapsulate the running code inside if __name__ == '__main__': block. Therefore, change your client code to read:
from pathfinder import MWE
if __name__ == '__main__':
mwe = MWE(10000)
mwe.start()
mwe.join()
(Also, note that you want to join() your process at some point.)
Check out the Windows-specific Python restrictions doc https://docs.python.org/2/library/multiprocessing.html#windows.
See https://stackoverflow.com/a/16642099/1510289 and https://stackoverflow.com/a/20222706/1510289 for similar questions.