I want to use multiprocessing to spread work across a system's multiple cores. As part of the work, they will run subprocess.call(..., shell=True). What happens when they do that? Does the subprocess fork stay on that core?
If the main work is done in the child processes created using subprocess module then you don't need multiprocessing to spread the work across multiple CPU cores. See Python threading multiple bash subprocesses?
What happens when they do that?
subprocess.call() runs an external command and waits for it to complete. It doesn't matter whether it is started inside a worker process created by multiprocessing module or not.
Does the subprocess fork stay on that core?
If you need it; you should set CPU affinity explicitly. psutil provides a portable way to set/get CPU affinity for a process.
If you use numpy then it may affect the cpu affinity. See Why does multiprocessing use only a single core after I import numpy?
Related
I run code on ARM Raspbian linux, Python 2.7.13 and amd64 Gentoo linux, Python 2.7.14
I have a function
import threading
def r() :
s = 1
while True:
s = s + 1
Then I create thread for this function
t = threading.Thread(target=r)
t.start()
And after that in htop I can see another process spawned (with its own PID)! And processing.Thread documentation says:
CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation).If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing.
Why is this behavior differs from documentation?
That's actually an OS thing: processes and threads do not differ that much on Linux. Htop will list all the separate threads as if they were processes. See this Unix Stack Exchange question for more info.
I am using Python3 to execute PYQT code; and at the same time, I need to call Python2.7 code, for operations that I cannot perform via Python3.
I did implement the 2.7 code execution via Popen; although it takes a considerable amount of time to run the 2.7 code, when called from Popen. The same operation is performed much faster if I run it directly from Python2.7.
Would be possible to use multiprocessing instead of subprocess.Popen for the same purpose, to speed up the execution of the 2.7 code?
And if that is appropriate; what would be the correct way to call Python2.7 code from a multiprocessing.Process? Or is it a waste to use multiprocess, since I am executing only one operation?
multiprocessing is similar to subprocess only on non-POSIX systems that cannot fork processes so you could, theoretically, hack away multiprocessing to use a different interpreter. It would be more trouble than its worth, tho, because at that point you wouldn't get any performance boost between spawning a subprocess and using a multiprocessing.Process (in fact, it would probably end slower due to the communication overhead added to multiprocessing.Process).
So, if we're talking only about a single task that has to execute in a different interpreter, this is as fast as you're gonna get. If there are multiple tasks to be executed in a different interpreter you may still benefit from multiprocessing.Process by spawning a single subprocess to run the different interpreter and then using multiprocessing within it to distribute multiple tasks over your cores.
I write deep learning software using Python and the Tensorflow library under Windows. Sometimes by mistake I load too much into memory and the computer stops responding; i cannot even kill the process.
Is it possible to limit the memory and CPU usage for Python scripts under Windows? I use PyCharm as an editor. Under UNIX Systems there seems to be the possibility to use resource.RLIMIT_VMEM, but under Windows I get the notification no module named resource.
This is a common problem when running resource-intensive processes, where the total amount of memory required might be hard to predict.
If the main issue is the whole system halting, you can create a watchdog process preventing that from happening and killing the process. It is a bit hacky, not as clean as the UNIX solution, and it will cost you a bit of overhead, but at least it can save you a restart!
This can easily be done in python, using the psutil package. This short piece of code runs whenever over 90% of virtual memory has been used and kills the python.exe process which is using the most memory:
import time
import psutil
while True:
if psutil.virtual_memory().percent > 90:
processes = []
for proc in psutil.process_iter():
if proc.name() == 'python.exe':
processes.append((proc, proc.memory_percent()))
sorted(processes, key=lambda x: x[1])[-1][0].kill()
time.sleep(10)
This can also be adapted for CPU, using psutil.cpu_percent().
You can, of course, use the Win32 Jobs API (CreateJobObject & AssignProcessToJobObject) to spawn your program as a sub-process and manage its resources.
But I guess a simpler solution, without going through all the hassle of coding, is to use Docker to create a managed environment.
When you create a Pool of processes using Python's multiprocessing, those processes will fork and globals in the parent process will show up in the child processes, noted in this question below:
How can I restrict the scope of a multiprocessing process?
This appears to include imported modules. That's a problem for modules that have side effects in __init__. Tensorflow is one such module, as soon as it's imported it tries to allocate memory to the GPU. This causes the sub process to crash because the parent already took that action.
Is there a good way to avoid the tensorflow module loading in the forked process?
Python 2.7 on Ubuntu (posix)
After much debugging I realize my problem statement wasn't really sufficient. The problem was that I did load tensorflow in one of the sub processes (I forgot!) and the sub processes needed to use the CPU only, not the GPU. I was forced to change environment variables to disable CUDA on the subprocesses with:
os.environ['CUDA_VISIBLE_DEVICES'] = ''
p = multiprocessing.Pool(processes=4)
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
In addition to David Parks's self answer, it is safer to create Pool and worker instances before importing TensorFlow or modules that can be ruined by os.fork().
I am trying to restrict the number of CPUs used by Python (for benchmarking & to see if it speeds up my program).
I have found a few Python modules for achieving this ('os', 'affinity', 'psutil') except that their methods for changing affinity only works with Linux (and sometimes Windows). There is also a suggestion to use the 'taskset' command (Why does multiprocessing use only a single core after I import numpy?) but this command not available on macOS as far as I know.
Is there a (preferable clean & easy) way to change affinity while running Python / iPython on macOS? It seems like changing processor affinity in Mac is not as easy as in other platforms (http://softwareramblings.com/2008/04/thread-affinity-on-os-x.html).
Not possible. See Thread Affinity API Release Notes:
OS X does not export interfaces that identify processors or control thread placement—explicit thread to processor binding is not supported. Instead, the kernel manages all thread placement. Applications expect that the scheduler will, under most circumstances, run its threads using a good processor placement with respect to cache affinity.
Note that thread affinity is something you'd consider fairly late when optimizing a program, there are a million things to do which have a larger impact on your program.
Also note that Python is particularly bad at multithreading to begin with.