We Keep Coding

Make C-Extension calls "green" in python

I have some python code which is heavily dependent on greenlets. I can use either gevent or eventlet.
I have packaged some sections of the code in a C-extension but these calls do not yield to other greenlets. Is it possible to write my extension such that it will yield control to other python threads while it does not require the GIL?

You can use normal PyObject_CallFunction(eventlet/greenlet.sleep) to yield control to other green threads. It must be run with GIL locked, like any other Python code.
You can not run Python code without GIL. (you can but it will quickly go sideways and corrupt memory).

Speed comparison using multiprocessing.Process versus subprocess.Popen

I am using Python3 to execute PYQT code; and at the same time, I need to call Python2.7 code, for operations that I cannot perform via Python3.
I did implement the 2.7 code execution via Popen; although it takes a considerable amount of time to run the 2.7 code, when called from Popen. The same operation is performed much faster if I run it directly from Python2.7.
Would be possible to use multiprocessing instead of subprocess.Popen for the same purpose, to speed up the execution of the 2.7 code?
And if that is appropriate; what would be the correct way to call Python2.7 code from a multiprocessing.Process? Or is it a waste to use multiprocess, since I am executing only one operation?

multiprocessing is similar to subprocess only on non-POSIX systems that cannot fork processes so you could, theoretically, hack away multiprocessing to use a different interpreter. It would be more trouble than its worth, tho, because at that point you wouldn't get any performance boost between spawning a subprocess and using a multiprocessing.Process (in fact, it would probably end slower due to the communication overhead added to multiprocessing.Process).
So, if we're talking only about a single task that has to execute in a different interpreter, this is as fast as you're gonna get. If there are multiple tasks to be executed in a different interpreter you may still benefit from multiprocessing.Process by spawning a single subprocess to run the different interpreter and then using multiprocessing within it to distribute multiple tasks over your cores.

Does holding CPython's GIL guarantee that all cpython's threads stop?

CPython is a multi-threaded application, and as such on Unix it uses (p)threads. Python extensions (written in C, say) often need to hold GIL to make sure Python objects don't get corrupted in critical sections of the code. How about other types of data? Specifically, does holding GIL in a Python extension guarantee that all other threads of CPython stop?
The reason for asking is that I am trying to port to FreeBSD a Python extension (which works on Linux and OSX) that embeds a Lisp compiler/system ECL using Boehm GC, and which crashes during the initialisation of the embedded Boehm GC. Backtraces suggest that another thread kicks in and causes havoc (pthread implementation on Linux are sufficiently different from FreeBSD's to expect trouble along these lines, too). Is there another mutex in CPython that may be used to achieve the locking?

Specifically, does holding GIL in a Python extension guarantee that all other threads of CPython stop?
The short answer is no - if other threads are executing code without holding the GIL (e.g. if they're running a C extension that releases the GIL), then they will keep running until try try to re-acquire the GIL (usually when they try to return (data) to the world of python).
It's also possible that core parts of the CPython (the core interpreter and/or built in fucntions/packages could release the GIL in similar circumstances/reasons that you would do in an extension. I have no idea if they actually do though.

When is PyEval_InitThreads meant to be called? [duplicate]

This question already has answers here:
PyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseam)
(7 answers)
Closed 9 years ago.
I'm a bit confused about when I'm supposed to call PyEval_InitThreads. In general, I understand that PyEval_InitThreads must be called whenever a non-Python thread (i.e. a thread that is spawned within an extension module) is used.
However, I'm confused if PyEval_InitThreads is for C programs which embed the Python interpreter, or Python programs which import C-extension modules, or both.
So, if I write a C extension module that will internally launch a thread, do I need to call PyEval_InitThreads when initializing the module?
Also, PyEval_InitThreads implicitly acquires the Global Interpreter Lock. So after calling PyEval_InitThreads, presumably the GIL must be released or deadlock will ensue. So how do you release the lock? After reading the documentation, PyEval_ReleaseLock() appears to be the way to release the GIL. However, in practice, if I use the following code in a C extension module:
PyEval_InitThreads();
PyEval_ReleaseLock();
...then at runtime Python aborts with:
Fatal Python error: drop_gil: GIL is not locked
So how do you release the GIL after acquiring it with PyEval_InitThreads?

Most applications never need to know about PyEval_InitThreads() at all.
The only time you should use it is if your embedding application or extension module will be making Python C API calls from more than one thread that it spawned itself outside of Python.
Don't call PyEval_ReleaseLock() in any thread which will later be making Python C API calls (unless you re-acquire it before those). In that case you should really use the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros instead.

Interrupt Python program deadlocked in a DLL

How can I ensure a python program can be interrupted via Ctrl-C, or a similar mechanism, when it is deadlocked in code within a DLL?

Not sure if this is exactly what you are asking, but there are issues when trying to interrupt (via Ctrl-C) a multi-threaded python process. Here is a video of a talk about the python Global Interpreter Lock that also discusses that issue:
Mindblowing Python GIL

You might want to take a look at this mailing list for a couple other suggestions, but there aren't any conclusive answers.
I've encountered the issue several times, and I can at least confirm that this happens when using FFI in Haskell. I could have sworn that I once saw something in Haskell's FFI documentation mentioning that DLLs would not return from a ctrl-c signal, but I'm not having any luck finding that document.
You can try using ctrl-break, but that's not working to break out of a DLL in Haskell and I'm doubting it will work in Python either.
Update: ctrl-break does work for me in Python when ctrl-c does not, during a call to a DLL function in an infinite loop.