I run code on ARM Raspbian linux, Python 2.7.13 and amd64 Gentoo linux, Python 2.7.14
I have a function
import threading
def r() :
s = 1
while True:
s = s + 1
Then I create thread for this function
t = threading.Thread(target=r)
t.start()
And after that in htop I can see another process spawned (with its own PID)! And processing.Thread documentation says:
CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation).If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing.
Why is this behavior differs from documentation?
That's actually an OS thing: processes and threads do not differ that much on Linux. Htop will list all the separate threads as if they were processes. See this Unix Stack Exchange question for more info.
Related
I am using Python3 to execute PYQT code; and at the same time, I need to call Python2.7 code, for operations that I cannot perform via Python3.
I did implement the 2.7 code execution via Popen; although it takes a considerable amount of time to run the 2.7 code, when called from Popen. The same operation is performed much faster if I run it directly from Python2.7.
Would be possible to use multiprocessing instead of subprocess.Popen for the same purpose, to speed up the execution of the 2.7 code?
And if that is appropriate; what would be the correct way to call Python2.7 code from a multiprocessing.Process? Or is it a waste to use multiprocess, since I am executing only one operation?
multiprocessing is similar to subprocess only on non-POSIX systems that cannot fork processes so you could, theoretically, hack away multiprocessing to use a different interpreter. It would be more trouble than its worth, tho, because at that point you wouldn't get any performance boost between spawning a subprocess and using a multiprocessing.Process (in fact, it would probably end slower due to the communication overhead added to multiprocessing.Process).
So, if we're talking only about a single task that has to execute in a different interpreter, this is as fast as you're gonna get. If there are multiple tasks to be executed in a different interpreter you may still benefit from multiprocessing.Process by spawning a single subprocess to run the different interpreter and then using multiprocessing within it to distribute multiple tasks over your cores.
CPython is a multi-threaded application, and as such on Unix it uses (p)threads. Python extensions (written in C, say) often need to hold GIL to make sure Python objects don't get corrupted in critical sections of the code. How about other types of data? Specifically, does holding GIL in a Python extension guarantee that all other threads of CPython stop?
The reason for asking is that I am trying to port to FreeBSD a Python extension (which works on Linux and OSX) that embeds a Lisp compiler/system ECL using Boehm GC, and which crashes during the initialisation of the embedded Boehm GC. Backtraces suggest that another thread kicks in and causes havoc (pthread implementation on Linux are sufficiently different from FreeBSD's to expect trouble along these lines, too). Is there another mutex in CPython that may be used to achieve the locking?
Specifically, does holding GIL in a Python extension guarantee that all other threads of CPython stop?
The short answer is no - if other threads are executing code without holding the GIL (e.g. if they're running a C extension that releases the GIL), then they will keep running until try try to re-acquire the GIL (usually when they try to return (data) to the world of python).
It's also possible that core parts of the CPython (the core interpreter and/or built in fucntions/packages could release the GIL in similar circumstances/reasons that you would do in an extension. I have no idea if they actually do though.
I am trying to restrict the number of CPUs used by Python (for benchmarking & to see if it speeds up my program).
I have found a few Python modules for achieving this ('os', 'affinity', 'psutil') except that their methods for changing affinity only works with Linux (and sometimes Windows). There is also a suggestion to use the 'taskset' command (Why does multiprocessing use only a single core after I import numpy?) but this command not available on macOS as far as I know.
Is there a (preferable clean & easy) way to change affinity while running Python / iPython on macOS? It seems like changing processor affinity in Mac is not as easy as in other platforms (http://softwareramblings.com/2008/04/thread-affinity-on-os-x.html).
Not possible. See Thread Affinity API Release Notes:
OS X does not export interfaces that identify processors or control thread placement—explicit thread to processor binding is not supported. Instead, the kernel manages all thread placement. Applications expect that the scheduler will, under most circumstances, run its threads using a good processor placement with respect to cache affinity.
Note that thread affinity is something you'd consider fairly late when optimizing a program, there are a million things to do which have a larger impact on your program.
Also note that Python is particularly bad at multithreading to begin with.
AFAIK, Python, using import thread, (or C#) doesn't do "real" multithreading, meaning all threads run on 1 CPU core.
But in C, using pthreads in linux, You get real multithreading.
Is this true ?
Assuming it is true, is there any difference between them when you have only 1 CPU core (I have it in a VM)?
Python uses something called a Global Interpreter Lock which means multiple python threads can only run within one native Thread.
There is more documentation in the official Docs here: https://wiki.python.org/moin/GlobalInterpreterLock
There shouldn't be a real performance difference on single core systems. On multicore systems the difference will varie based on what you do. (I/O is for the most part not affected by the GIL).
I'm not aware of how it works C# internally, but for CPython (the "official" python interpreter) it is true: threads are not really parallel due to GIL.
Other implementation of the Python interpreter do not suffer of this problem (like C's pthreads library).
Howevere if you only have 1 CPU you won't notice any difference.
As a side note: if you need real parallelism in CPython you could you multiprocessing module, which uses processes instead of threads.
EDIT:
Also thread module is a bit deprecated, you should consider using threading.
In order to do quality assurance in a critical multicore (8) workstation, I want to run the same code in different processors but not in parallel or concurrently.
I need to run it 8 times, one run for each processor.
What I don't know is how to select the processor I want.
How can this be accomplished in Python?
In Linux with schedutils, I believe you'd use taskset -c X python foo.py to run that specific Python process on CPU X (exactly how you identify your CPUs may vary, but I believe numbers such as 1, 2, 3, ... should work anywhere). I'm sure Windows, BSD versions, etc, have similar commands to support direct processor assignment, but I don't know them.
Which process goes on which core is normaly decided by your OS. On linux there is taskset from the schedutils package to explicitly run a program on a processor.
Python 2.6 has a multiprocessing module that takes python functions and runs them in seperate processes, probably moving each new process to a different core.