What is the best way to store thread local data in a PyQt application?
My application uses both QThreads and, through some dependencies, also native Python threads (from the threading module). My specific use case for thread local storage is mostly related to the former.
I can see some options.
Use threading.local. When called from a thread foreign to threading, threading.current_thread() returns a _DummyThread object. AFAICT, threading.local does support dummy threads, but it feels fragile. In particular, since a DummyThread is never deleted, the store will not be cleared when the thread exits. Obviously I want no memory leaks.
When I know that the code in question will always run in a QThread, I could just store the data directly in the thread object obtained via QThread::getCurrentThread(). No idea what would happen if called from a non-Qt thread.
Is there a QThreadStorage equivalent in PyQt? In Qt, it is a template, not a class, so I don't think it is available in PyQt.
Related
I have been trying to find the most elegant way to decouple my programs from the GUI, such that I can change my front-end without needing to re-write a whole lot of code.
I work with threads a lot, so I often have the need to notify the main GUI thread of asynchronous happenings either through events (for wxPython) or signals (for PyQt). I have experimented a bit with PyPubSub, which may be what I am looking for, but while there are tons of wxPython examples (since it was originally included with it in early development).
I am not aware if there is a 'proper' way to use it with PyQt without running into race conditions. If anyone has some insight on this, I would really appreciate it!
PyPubSub's sendMessage() will call listeners in the same thread as the sender (default Python behavior). In a multithreaded GUI app, you must ensure that listeners that interact with GUI are called in the main thread. Also threads execute independently, so you need each thread to call its own listeners, based on a timed or idle callback mechanism.
The way to call listeners in the correct thread in PyQt is via signals. PyPubSub can still be used in a multithreaded PyQt GUI, but the mechanism used to transfer the "message" from sender to listener would have to be via a signal. There isn't one best way to do it I don't think, depends on details of your app design. You could for example have a QtPubsubHandler that derives from QObject and gets created in main thread, and a QtPubsubSender class that also derives from QObject and gets created in each worker thread. The QtPubSubSender defines a custom signal, say pubsub, which QtPubsubHandler connects to. Then to send a message, the sender does qtPubsubHandler.sendMessage(topic, **data), which causes a pubsub signal to get emitted, which Qt properly queues and eventually signals the QtPubsubHandler, which actually calls pub.sendMessage().
There are many other ways, but you get the general idea: two classes that derive from QObject, and one of them does the actual sending in the same thread as the intended listener, the other uses a signal so everything is thread safe. Actually you don't have to use PyQt signals: but then you would have to have a queue in main thread and have an idle callback that allows it to process any items on the queue.
My understanding is that the typical GIL manipulations involve, e.g., blocking I/O operations. Hence one would want to release the lock before the I/O operation and reacquire it once it has completed.
I'm currently facing a different scenario with a C extension: I am creating X windows that are exposed to Python via the Canvas class. When the method show() is called on an instance, a new UI thread is started using PyThreads (with a call to PyThread_start_new_thread). This new thread is responsible for drawing on the X window, using the Python code specified in the on_draw method of a subclass of Canvas. A pure C event loop is started in the main thread that simply checks for events on the X window and, for the time being, only captures the WM_DELETE_EVENT.
So I have potentially many threads (one for each X window) that want to execute Python code and the main thread that does not execute any Python code at all.
How do I release/acquire the GIL in order to allow the UI threads to get into the interpreter orderly?
The rule is easy: you need to hold the GIL to access Python machinery (any API starting with Py<...> and any PyObject).
So, you can release it whenever you don't need any of that.
Anything further than this is the fundamental problem of locking granularity: potential benefits vs locking overhead. There was an experiment for Py 1.4 to replace the GIL with more granular locks that failed exactly because the overhead proved prohibitive.
That's why it's typically released for code chunks involving call(s) to extental facilities that can take arbitrary time (especially if they involve waiting for external events) -- if you don't release the lock, Python will be just idling during this time.
Heeding this rule, you will get to your goal automatically: whenever a thread can't proceed further (whether it's I/O, signal from another thread, or even so much as a time.sleep() to avoid a busy loop), it will release the lock and allow other threads to proceed in its stead. The GIL assigning mechanism strives to be fair (see issue8299 for exploration on how fair it is), releasing the programmer from bothering about any bias stemming solely from the engine.
I think the problem stems from the fact that, in my opinion, the official documentation is a bit ambiguous on the meaning of Non-Python created threads. Quoting from it:
When threads are created using the dedicated Python APIs (such as the threading module), a thread state is automatically associated to them and the code showed above is therefore correct. However, when threads are created from C (for example by a third-party library with its own thread management), they don’t hold the GIL, nor is there a thread state structure for them.
I have highlighted in bold the parts that I find off-putting. As I have stated in the OP, I am calling PyThread_start_new_thread. Whilst this creates a new thread from C, this function is not part of a third-party library, but of the dedicated Python (C) APIs. Based on this assumption, I ruled out that I actually needed to use the PyGILState_Ensure/PyGILState_Release paradigm.
As far as I can tell from what I've seen with my experiments, a thread created from C with (just) PyThread_start_new_thread should be considered as a non-Python created thread.
Specifically I'm talking about Python. I'm trying to hack something (just a little) by seeing an object's value without ever passing it in, and I'm wondering if it is thread safe to use thread local to do that. Also, how do you even go about doing such a thing?
No -- thread local means that each thread gets its own copy of that variable. Using it is (at least normally) thread-safe, simply because each thread uses its own variable, separate from variables by the same name that's accessible to other threads. OTOH, they're not (normally) useful for communication between threads.
I noticed that when the function setModel is executed in parallel thread (I tried threading.Timer or threading.thread), I get this:
QObject: Cannot create children for a parent that is in a different thread.
(Parent is QHeaderView(0x1c93ed0), parent's thread is QThread(0xb179c0), current thread is QThread(0x23dce38)
QObject::startTimer: timers cannot be started from another thread
QObject: Cannot create children for a parent that is in a different thread.
(Parent is QTreeView(0xc65060), parent's thread is QThread(0xb179c0), current thread is QThread(0x23dce38)
QObject::startTimer: timers cannot be started from another thread
Is there any way to solve this?
It is indeed a fact of life that multithreaded use of Qt (and other rich frameworks) is a delicate and difficult job, requiring explicit attention and care -- see Qt's docs for an excellent coverage of the subject (for readers experienced in threading in general, with suggested readings for those who yet aren't).
If you possibly can, I would suggest what I always suggest as the soundest architecture for threading in Python: let each subsystem be owned and used by a single dedicated thread; communicate among threads via instances of Queue.Queue, i.e., by message passing. This approach can be a bit restrictive, but it provides a good foundation on which specifically identified and carefully architected exceptions (based on thread pools, occasional new threads being spawned, locks, condition variables, and other such finicky things;-). In the latter category I would also classify Qt-specific things such as cross-thread signal/slot communication via queued connections.
Looks like you've stumped on a Qt limitation there. Try using signals or events if you need objects to communicate across threads.
Or ask the Qt folk about this. It doesn't seem specific to PyQt.
I'm trying to wrap my head around what is happening in this recipe, because I'm planning on implementing a wx/twisted app similar to this (ie. wx and twisted running in separate threads). I understand that both twisted and wx event-loops need to be accessed in a thread-safe manner (ie. reactor.callFromThread, wx.PostEvent, etc). What I am questioning is the thread-safety of passing in instance methods of objects instantiated in one thread (in the case of this recipe, the GUI thread) as deferred callBack and errBack methods for a reactor running in a separate thread. Is that a good idea?
There is a wxreactor available in twisted, but googling reveals that there have been numerous problems with it since it was introduced to the library. Even the person who initially came up with the wxreactor technique, advocates running wx and twisted in separate threads.
I haven't been able to find any other examples of this technique, but I'd love to see some.
I wouldn't say that it's a "good idea". You should just run the reactor and the GUI in the same thread with wxreactor.
The timer-driven event-loop starving approach described by Mr. Schroeder is the worst possible fail-safe way to implement event-loop integration. If you use wxreactor (not wxsupport) Twisted now uses an approach where multiplexing is shunted off to a thread internally so that nothing needs to use a timer. Better yet would be for wxpython to expose wxSocket and have someone base a reactor on it.
However, if you're set on using a separate thread to communicate with Twisted, the one thing to keep in mind is that while you can use objects that originate from any thread you like as the value to pass to Deferred.callback, you must call Deferred.callback only in the reactor thread itself. Deferreds are not threadsafe; thanks to some debugging utilities, not even the Deferred class is threadsafe, so you need to be very careful when you are using them to never leave the Twisted main thread. i.e. when you have a result in the UI thread, use reactor.callFromThread(myDeferred.callback, myresult).
The sole act of passing instance methods between threads is safe as long as you properly synchronize eventual destruction of those instances (threads share memory so it really doesn't matter which one did the allocation/initialization of a bit of it).
The overall thread safety depends on what those methods actually do, so just make them "play nice" and you should be ok.