We Keep Coding

Python Multiprocessing within Jupyter Notebook

I am new to the multiprocessing module in Python and work with Jupyter notebooks. I have tried the following code snippet from PMOTW:
import multiprocessing
def worker():
"""worker function"""
print('Worker')
return
if __name__ == '__main__':
jobs = []
for i in range(5):
p = multiprocessing.Process(target=worker)
jobs.append(p)
p.start()
When I run this as is, there is no output.
I have also tried creating a module called worker.py and then importing that to run the code:
import multiprocessing
from worker import worker
if __name__ == '__main__':
jobs = []
for i in range(5):
p = multiprocessing.Process(target=worker)
jobs.append(p)
p.start()
There is still no output in that case. In the console, I see the following error (repeated multiple times):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 116, in _main
self = pickle.load(from_parent)
AttributeError: Can't get attribute 'worker' on <module '__main__' (built-in)>
However, I get the expected output when the code is saved as a Python script and exectued.
What can I do to run this code directly from the notebook without creating a separate script?

I'm relatively new to parallel computing so I may be wrong with some technicalities. My understanding is this:
Jupyter notebooks don't work with multiprocessing because the module pickles (serialises) data to send to processes.
multiprocess is a fork of multiprocessing that uses dill instead of pickle to serialise data which allows it to work from within Jupyter notebooks. The API is identical so the only thing you need to do is to change
import multiprocessing
to...
import multiprocess
You can install multiprocess very easily with a simple
pip install multiprocess
You will however find that your processes will still not print to the output, (although in Jupyter labs they will print out to the terminal the server out is running in). I stumbled upon this post trying to work around this and will edit this post when I find out how to.

I'm not an export either in multiprocessing or in ipykernel(which is used by jupyter notebook) but because there seems nobody gives an answer, I will tell you what I guessed. I hope somebody complements this later on.
I guess your jupyter notebook server is running on Windows host. In multiprocessing there are three different start methods. Let's focus on spawn, which is the default on windows, and fork, the default on Unix.
Here is a quick overview.
spawn
(cpython) interactive shell - always raise an error
run as a script - okay only if you nested multiprocessing code in if __name__ == '__main'__
Fork
always okay
For example,
import multiprocessing
def worker():
"""worker function"""
print('Worker')
return
if __name__ == '__main__':
multiprocessing.set_start_method('spawn')
jobs = []
for i in range(5):
p = multiprocessing.Process(target=worker)
jobs.append(p)
p.start()
This code works when it's saved and run as a script, but raises an error when entered in an python interactive shell. Here is the implementation of ipython kernel, and my guess is that that it uses some kind of interactive shell and so doesn't go well with spawn(but please don't trust me).
For a side note, I will give you an general idea of how spawn and fork are different. Each subprocess is running a different python interpreter in multiprocessing. Particularly, with spawn, a child process starts a new interpreter and imports necessary module from scratch. It's hard to import code in interactive shell, so it may raise an error.
fork is different. With fork, a child process copies the main process including most of the running states of the python interpreter and then continues execution. This code will help you understand the concept.
import os
main_pid = os.getpid()
os.fork()
print("Hello world(%d)" % os.getpid()) # print twice. Hello world(id1) Hello world(id2)
if os.getpid() == main_pid:
print("Hello world(main process)") # print once. Hello world(main process)

Much like you I encountered the attribute error. The problem seems to be related how jupyter handles multithreading. The fastest result I got was to follow the Multi-processing example.
So the ThreadPool took care of my issue.
from multiprocessing.pool import ThreadPool as Pool
def worker():
"""worker function"""
print('Worker\n')
return
pool = Pool(4)
for result in pool.map(worker, range(5)):
pass # or print diagnostics

This works for me on MAC (cannot make it work on windows):
import multiprocessing as mp
mp_start_count = 0
if __name__ == '__main__':
if mp_start_count == 0:
mp.set_start_method('fork')
mp_start_count += 1

Save the function to a separate Python file then import the function back in. It should work fine that way.

Empty python process hangs on join [sys.stderr.flush()]

Python guru I need your help. I faced quite strange behavior:
empty python Process hangs on joining. Looks like it forks some locked resource.
Env:
Python version: 3.5.3
OS: Ubuntu 16.04.2 LTS
Kernel: 4.4.0-75-generic
Problem description:
1) I have a logger with thread to handle messages in background and queue for this thread. Logger source code (a little bit simplified).
2) And I have a simple script which uses my logger (just code to display my problem):
import os
from multiprocessing import Process
from my_logging import get_logger
def func():
pass
if __name__ == '__main__':
logger = get_logger(__name__)
logger.start()
for _ in range(2):
logger.info('message')
proc = Process(target=func)
proc.start()
proc.join(timeout=3)
print('TEST PROCESS JOINED: is_alive={0}'.format(proc.is_alive()))
logger.stop()
print('EXIT')
Sometimes this test script hangs. Script hangs on joining process "proc" (when script completes execution). Test process "proc" stay alive.
To reproduce this problem you can run the script in loop:
$ for i in {1..100} ; do /opt/python3.5.3/bin/python3.5 test.py ; done
Investigation:
Strace shows following:
strace: Process 25273 attached
futex(0x2275550, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff
And I figured out the place where process hangs. It hangs in multiprocessing module, file process.py, line 269 (python3.5.3), on flushing STDERR:
...
267 util.info('process exiting with exitcode %d' % exitcode)
268 sys.stdout.flush()
269 sys.stderr.flush()
...
If line 269 commented the script completes successfully always.
My thoughts:
By default logging.StreamHandler uses sys.stderr as stream.
If process has been forked when logger flushing data to STDERR, process context gets some locked resource and further hangs on flushing STDERR.
Some workarounds which solves problem:
Use python2.7. I can't reproduce it with python2.7. Maybe timings prevent me to reproduce the problem.
Use process to handle messages in logger instead of thread.
Do you have any ideas on this behavior? Where is the problem? Am I doing something wrong?

It looks like this behaviour is related to this issue: http://bugs.python.org/issue6721

Question: Sometimes ... Test process "proc" stay alive.
I could only reproduce your
TEST PROCESS:0 JOINED: is_alive=True
by adding a time.sleep(5) to def func():.
You use proc.join(timeout=3), that's the expected behavior.
Conclusion:
Overloading your System, starts in my Environment with 30 Processes running, triggers your proc.join(timeout=3).
You may rethink your Testcase to reproduce your problem.
One Approach I think, is fine-tuning your Process/Thread with some time.sleep(0.05) to give off a timeslice.
Your are using from multiprocessing import Queue
use from queue import Queue instead.
From the Documentation
Class multiprocessing.Queue
A queue class for use in a multi-processing (rather than multi-threading) context.
In class QueueHandler(logging.Handler):, prevent to do
self.queue.put_nowait(record)
after
class QueueListener(object):
...
def stop(self):
...
implement, for instance
class QueueHandler(logging.Handler):
def __init__(self):
self.stop = Event()
...
In def _monitor(self): use only ONE while ... loop.
Wait until the self._thread stoped
class QueueListener(object):
...
def stop(self):
self.handler.stop.set()
while not self.queue.empty():
time.sleep(0.5)
# Don't use double flags
#self._stop.set()
self.queue.put_nowait(self._sentinel)
self._thread.join()

Pathos multiprocessing pipe and queue on windows

I'm trying to use the pathos library to replace the builtin multiprocessing library, but am having difficulty using either pipes or queues on windows. Here's a representative example:
from pathos.helpers import mp
#import multiprocessing as mp
def f(pipe, queue):
if sys.gettrace():
print 'Debug mode. About to crash'
else:
print 'Execute mode'
pipe.send('pipe')
queue.put('queue')
if __name__ == '__main__':
mp.freeze_support()
to_child, to_self = mp.Pipe()
queue = mp.Queue()
p = mp.Process(target=f, args=(to_child, queue))
p.start()
p.join()
pipe.send('pipe') raises IOError: (6, 'The handle is invalid') and queue.put('queue') raises WindowsError: [Error 5] Access is denied. Both work correctly using the vanilla multiprocessing module.
Am I doing something wrong?
Edit:
This crash only occurs when I'm trying to debug child processes (I use WingIDE). I can accurately predict the crash by checking sys.gettrace(), as above.

It turns out the issue arises when Wing IDE is set to debug child processes. As I understand it, wing enables child process debugging by inserting itself between parent and child processes (which means that 'child' processes are now actually the grandchildren of the parent process). On windows, this is made possible by monkeypatching multiprocessing to cause Popen to set the 'inheritable' flag to True when calling duplicate() on handles.
Without the inheritable setting, grandchild processes cannot access the handles, and hence the exceptions above are raised.
It seems likely that a similar monkey patch could be applied to pathos's helpers.mp module to allow wing to debug child processes with pathos.

Difference in behavior between os.fork and multiprocessing.Process

I have this code :
import os
pid = os.fork()
if pid == 0:
os.environ['HOME'] = "rep1"
external_function()
else:
os.environ['HOME'] = "rep2"
external_function()
and this code :
from multiprocessing import Process, Pipe
def f(conn):
os.environ['HOME'] = "rep1"
external_function()
conn.send(some_data)
conn.close()
if __name__ == '__main__':
os.environ['HOME'] = "rep2"
external_function()
parent_conn, child_conn = Pipe()
p = Process(target=f, args=(child_conn,))
p.start()
print parent_conn.recv()
p.join()
The external_function initializes an external programs by creating the necessary sub-directories in the directory found in the environment variable HOME. This function does this work only once in each process.
With the first example, which uses os.fork(), the directories are created as expected. But with second example, which uses multiprocessing, only the directories in rep2 get created.
Why isn't the second example creating directories in both rep1 and rep2?

The answer you are looking for is in detail addressed here. There is also an explanation of differences between different OS.
One big issue is that the fork system call does not exist on Windows. Therefore, when running a Windows OS you cannot use this method. multiprocessing is a higher-level interface to execute a part of the currently running program. Therefore, it - as forking does - creates a copy of your process current state. That is to say, it takes care of the forking of your program for you.
Therefore, if available you could consider fork() a lower-level interface to forking a program, and the multiprocessing library to be a higher-level interface to forking.

To answer your question directly, there must be some side effect of external_process that makes it so that when the code is run in series, you get different results than if you run them at the same time. This is due to how you set up your code, and the lack of differences between os.fork and multiprocessing.Process in systems that os.fork is supported.
The only real difference between the os.fork and multiprocessing.Process is portability and library overhead, since os.fork is not supported in windows, and the multiprocessing framework is included to make multiprocessing.Process work. This is because os.fork is called by multiprocessing.Process, as this answer backs up.
The important distinction, then, is os.fork copies everything in the current process using Unix's forking, which means at the time of forking both processes are the same with PID differences. In Window's, this is emulated by rerunning all the setup code before the if __name__ == '__main__':, which is roughly the same as creating a subprocess using the subprocess library.
For you, the code snippets you provide are doing fairly different things above, because you call external_function in main before you open the new process in the second code clip, making the two processes run in series but in different processes. Also the pipe is unnecessary, as it emulates no functionality from the first code.
In Unix, the code snippets:
import os
pid = os.fork()
if pid == 0:
os.environ['HOME'] = "rep1"
external_function()
else:
os.environ['HOME'] = "rep2"
external_function()
and:
import os
from multiprocessing import Process
def f():
os.environ['HOME'] = "rep1"
external_function()
if __name__ == '__main__':
p = Process(target=f)
p.start()
os.environ['HOME'] = "rep2"
external_function()
p.join()
should do exactly the same thing, but with a little extra overhead from the included multiprocessing library.
Without further information, we can't figure out what the issue is. If you can provide code that demonstrates the issue, that would help us help you.

Python losing control of subprocess?

I'm using a commercial application that uses Python as part of its scripting API. One of the functions provided is something called App.run(). When this function is called, it starts a new Java process that does the rest of the execution. (Unfortunately, I don't really know what it's doing under the hood as the supplied Python modules are .pyc files, and many of the Python functions are SWIG generated).
The trouble I'm having is that I'm building the App.run() call into a larger Python application that needs to do some guaranteed cleanup code (closing a database, etc.). Unfortunately, if the subprocess is interrupted with Ctrl+C, it aborts and returns to the command line without returning control to the main Python program. Thus, my cleanup code never executes.
So far I've tried:
Registering a function with atexit... doesn't work
Putting cleanup in a class __del__ destructor... doesn't work. (App.run() is inside the class)
Creating a signal handler for Ctrl+C in the main Python app... doesn't work
Putting App.run() in a Thread... results in a Memory Fault after the Ctrl+C
Putting App.run() in a Process (from multiprocessing)... doesn't work
Any ideas what could be happening?

This is just an outline- but something like this?
import os
cpid = os.fork()
if not cpid:
# change stdio handles etc
os.setsid() # Probably not needed
App.run()
os._exit(0)
os.waitpid(cpid)
# clean up here
(os.fork is *nix only)
The same idea could be implemented with subprocess in an OS agnostic way. The idea is running App.run() in a child process and then waiting for the child process to exit; regardless of how the child process died. On posix, you could also trap for SIGCHLD (Child process death). I'm not a windows guru, so if applicable and subprocess doesn't work, someone else will have to chime in here.
After App.run() is called, I'd be curious what the process tree looks like. It's possible its running an exec and taking over the python process space. If thats happening, creating a child process is the only way I can think of trapping it.

If try: App.run() finally: cleanup() doesn't work; you could try to run it in a subprocess:
import sys
from subprocess import call
rc = call([sys.executable, 'path/to/run_app.py'])
cleanup()
Or if you have the code in a string you could use -c option e.g.:
rc = call([sys.executable, '-c', '''import sys
print(sys.argv)
'''])
You could implement #tMC's suggestion using subprocess by adding
preexec_fn=os.setsid argument (note: no ()) though I don't see how creating a process group might help here. Or you could try shell=True argument to run it in a separate shell.
You might give another try to multiprocessing:
import multiprocessing as mp
if __name__=="__main__":
p = mp.Process(target=App.run)
p.start()
p.join()
cleanup()

Are you able to wrap the App.Run() in a Try/Catch?
Something like:
try:
App.Run()
except (KeyboardInterrupt, SystemExit):
print "User requested an exit..."
cleanup()