I saw already discussions about the synchronised question: How to prevent two threads entering the same section of code? They introduce some kind of synchronised decorator like in Java.
But that is not my problem. I would like to run a few lines of code at once. The Python interpreter shall not switch to another thread in between those lines of code. Maybe "without context switching" is the right description. Other threads shall wait until the python interpreter has completed the specified lines. And afterwards it can decide again to stop the execution of the current thread and excecute another thread for a while.
Is there a way to do that?
Something like this:
with atomic_lock:
do_stuff()
# no context switch here - executed directly after another
do_other_stuff()
What I would like to do:
I would like to loop through the keys in a dictionary. The for loop fails if the dictionary is changed while the loop is executing. For example if a key is added. That happens on many other places in my code. So my idea was to prevent any other code from executing while the for loop is executed.
But maybe there is a different solution? Can I get a lock on that dictionary?
The simple answer is: That is not possibele yet.
The only possible way is to work with locks, because that is the only suitable thing that is implemented in the python interpreter.
Related
I'm building a Python IDE, which needs to highlight all occurrences of the name under cursor (using Jedi library). The process of finding the occurrences can be quite slow.
In order to avoid freezing the GUI, I could run the search in another thread, but when the user moves quickly over several words, the background threads could pile up while working on now obsolete tasks. I would like to cancel the search for previous occurrences when user moves to new name.
Looks like killing a thread is complicated in Python. What are the other options for creating an easily cancellable background tasks in Python 3.4+?
I think concurrent.futures is the answer.
You can create a Thread / Process pool, submit any callable, receive a Future, which you can cancel if needed.
Reference: https://docs.python.org/3/library/concurrent.futures.html
A thread cannot be stopped by another one. This is a OS limitation rather than a Python one. Only thing you can do is periodically inspect a variable and, if set, stop the thread itself (just return).
Moreover, threads in Python suffer from the GIL. This means that CPU intensive operations, when carried out in a separate thread, will still affect your main loop as only one thread per process can run at a time.
I'd recommend you to run the search in a separate process which you can easily cancel whenever you want.
What the guys of YouCompleteMe are doing for example is wrapping Jedi in a HTTP server which they can query in the background. If the user moves the cursor before the completion comes back, the IDE can simply drop the request.
Well, my personal favorites are work queues. If it's a one-time application you should take a look at python rq. Extremely easy and fun to use. If you want to build something more "professional-grade" take a look at something like celery.
You might also want to look at multiprocessing
I am implementing a Python plugin that is part of a larger C++ program. The goal of this program is to allow the user to input a command's actions in Python. It currently receives a string from the C++ function and runs it via the exec() function. The user can then use an API to affect changes on the larger C++ program.
The current feature I am working on is a pause execution feature. It needs to remember where it is in the code execution as well as the state of any local variables, and resume execution once a condition has been met. I am not very familiar with Python, and I would like some advice how to implement this feature. My first design ideas:
1) Using the yield command.
This seemed to be a good idea at the start since when you use the next command it remembers everything I needed it to, but the problem is that yield only returns to the previous level in the call stack as far as I can tell. So if the user calls a function that yields it will simply return to the user's code, and not the larger C++ program. As far as I can tell there isn't a way to propagate the yield command up the stack???
2) Threading
Create a main python thread that creates a thread for each command. This main thread would spawn a thread for each command executed and kill it when it is done. If it needs to be suspended and restarted it could do so through a queue of locks.
Those were the only two options I came up with. I am not sure the yield function would work or is what it was designed to do. I think the Threading approach would work but might be overkill, and take a long time to develop. I also was looking for some sort of Task Module in Python, but couldn't find exactly what I was looking for. I was wondering if anyone has any other suggestions as I am not very familiar with Python.
EDIT: As mentioned in the comments I did not explain what needs to happen when the script "Pauses". The python plugin needs to allow the C++ program to continue execution. In my mind this means A) returning if we are talking about a single threaded approach, or B) Sending a message(Function call?) to C++
EDIT EDIT: As stated I didn't fully explain the problem description. I will make another post that has a better statement of what currently exists, and what needs to happen as well as providing some sudo code. I am new to Stack Overflow, so if this is not the appropriate response please let me know.
Whenever a signal is sent in Python, execution is immediately paused until whatever signal handler function is being used is finished executing; at that point, the execution continues right where it left off. My suggestion would be to use one of the user-defined signals (signal.SIGUSR1 and signal.SIGUSR2). Take a look at the signal documentation here:
https://docs.python.org/2/library/signal.html
At the beginning of the program, you'd define a signal handler function like so:
def signal_pause(signum, frame):
if signum == signal.SIGUSR1:
# Do your pause here - function processing, etc
else:
pass
Then in the main program somewhere, you'll switch out the default signal handler for the one you just created:
signal.signal(signal.SIGUSR1, signal_pause)
And finally, whenever you want to pause, you'll send the SIGUSR1 signal like so:
os.kill(os.getpid(),signal.SIGUSR1)
Your code will immediately pause, saving its state, and head to the signal_pause function to do whatever you need to do. Once that function exits, normal program execution will resume.
EDIT: this assumes you want to do something sophisticated while you're pausing the program. If all you want to do is wait a few seconds or ask for some user input, there are some much easier ways (time.sleep or input respectively).
EDIT EDIT: this assumes you're on a Unix system.
If you need to communicate with a C program, then sockets are probably the way to go.
https://docs.python.org/2/library/socket.html
One of your two programs acts as the socket server, and the other connects to it as the socket client. When you want the C++ program to continue, you use socket.send() to transmit a continue message. Then your Python program would use socket.recv(), which will cause it to wait around until it receives a message back from the C++ program.
If you need two programs to send signals to each other, this is probably the safest way to go about it.
I have a lot of long running tasks that run in the background of my Python app. I put them all in the global QThreadPool. When the user quits, all of those background tasks need to stop.
Right now, I have the following code:
app.aboutToQuit.connect(killAllThreads)
def killAllThreads():
QtCore.QThreadPool.globalInstance().waitForDone()
I've seen suggestions to add a global variable that says whether the application should be quitting, and to have threads terminate themselves, but this sounds like a terribly messy and inelegant solution. Would you really propose adding a check before every line of code in a background task to make sure that the application shouldn't be quitting yet? That's hundreds of checks that I would have to add.
The suggestion seems to make the assumption that my tasks are simple and/or have complex clean ups involved, but actually, I have just the opposite: the tasks involve hundreds of lines of code, each of which can take several seconds, but no clean up needs to be done at all.
I've heard simply killing the threads would be a bad idea, as then they wouldn't be guaranteed to clean up properly, but as no clean up is necessary, that's exactly what I want to do. Additionally, race conditions could occur, but again, the tasks need to stop right now, so I really don't care if they end up in an invalid state.
So I need to know the following:
How do I get a list of all the running threads in a QThreadPool?
How do I have them abort what they're doing?
The simple answer to this question is that you cannot abort any of the threads in QThreadPool, because they are wrapped in instances of QRunnable. There is no external way to terminate a QRunnable; it has to terminate itself from inside its reimplemented run() method.
However, it sounds like the tasks running inside your run() method don't lend themselves to periodically checking a flag to see if they should terminate.
If that is the case, you only have two options:
Re-write the tasks in such a way that they can periodically check a flag.
Don't use QThreadPool/QRunnable.
Obviously, choosing (2) implies switching to a more low-level solution, like QThread, and managing the pool of threads yourself.
use daemons, they are automatically terminated when the main thread is ended
from threading import Thread
t = Thread(target=self.ReadThread)
t.setDaemon(True)
I have a code-base that I'm looking to split up and add to by using threading, however I'm relatively new on how to handle it. Please before reading further respect my wish of NOT just re-writing this code and tossing it back at me with the problem solved. I would much rather work the problem out by someone pointing me in the right direction, than someone solving it FOR me; I don't learn well that way.
The fully functioning code-base is here -- It requires the mechanize and beautifulsoup libraries which can be installed via easy_install.
I've separated out all of my functions, and tried to keep the code as clean as possible (I'm sure there are some optimizations in there that I'll get reamed for, but the main problem is how to thread this.
My ultimate goal is to pack this into a thread, and then share cookies between other initialized browser objects in order to do other things while my original code is running 'backgrounded'.
I've tried thus:
class Recon(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
#Packed the stuff above my original while loop in here, minus functions.
def run(self):
#Packed my code past the while loop in here.
somevar = Recon()
somevar.start()
Problem I'm having is that, once I run the program it will run the things in init, but afterwards it just sits there and freezes on me. No traceback, no errors, just doesn't do anything, doesn't even return my command prompt back to my control.
Could I just get some tips, or a general flow of how to convert this? I got overwhelmed and deleted the code I was trying with so I don't have that example, but do I need to be prepending 'self.' to all of my variables? Do I need to just define my vars as global?
Here is a reproduction of what I'm having trouble with after having tried to convert the script to use threading.
As long as you have a single thread (as in the above snippet, where you instantiate Recon just once), it shouldn't matter much what you do where; but of course I imagine the reason you're introducing threading is to eventually move to having multiple threads active.
If that's the case, then the first key issue is to ensure that you never have two or more threads simultaneously trying to use the same shared system/resource -- for example, multiple threads writing at the same time to ReconFile, in the case of the code at the pastebin URL you mention.
The classic way to avoid such issues is to use locking, but my favorite way is quite different: make sure any such resource is accessed by only one dedicated thread, and use a Queue.Queue instance (intrinsically threadsafe) to have other threads post work-request to the dedicated thread (so instead of writing to ReconFile directly each other thread would make a list of lines to be written contiguously, then .put the list on the queue where the "recon file writing" worker thread is waiting via .get).
When you need to get results back from such actions (not the case here), the requesting thread would place its own personal "queue on which to return results" as part of the "work request packet" it puts to the worker thread's queue. I've presented much more detail about this recommended architecture in the threading chapter of "Python in a Nutshell" 2nd edition (and why, as the book's author, I would of course never recommend you perform an illegal download of a free pirate copy of my book, I can however mention there's plenty of sites offering such pirate copies for download -- the legal way to read my book for free is to sign up for a trial offer to O'Reilly's "safari" online books website).
This does not address the specific problem you're observing, since it's happening when you only have one thread around. I notice that thread is trying to perform lots of I/O on standard input and standard output, which is possibly problematic from a thread -- consider doing the input for a thread before you start it (in the main thread) and for needed output use Python's standard logging module, which is guaranteed to be thread-safe. Do you still observe problems then? If that's the case, then the next step is to pepper your code with logging.info calls so that you can pinpoint exactly where it's stalling -- and tell us about that, so we can try to help from there!
Okay, I have an application written with cherrypy, and I want to build a wxpython gui for it. The problem is that both modules use a close loop for event handling, which (I assume) means while one is running the other will be locked.
I asked for some advice and it was suggested that I merge the two event loops rather than using the stock entrypoints (quickloop() for cherrypy and MainLoop() for wx)
The problem is I have no idea how to do this. Any advice would be greatly appreciated.
You already asked the same question here: cherrypy and wxpython, and I gave you the best response you're going to find anywhere there, which was voted up and you approved, apparently. Why are you asking again?
In the case of cherrypy, you have the source. Look in the code what quickloop() does and then try to merge this code with the MainLoop() of WX.
Both loops will probably look like this:
while (true) {
if (pendingEvents()) processEvents ();
else waitForEvents ();
}
You must find a way to merge the two waiting calls into one (so the code continues if either event source had pending events). For WX, look at Dispatch(), Pending() and ProcessIdle().
Or you can look at wxIdleEvent (see the docs) and process all cherrypy events in there.
Another solution might be to run the two loops in different threads. In this case, you can't call WX methods from cherrypy code and vice versa. To solve this, you must find a way to send messages to the other thread with all the information which method to call. This makes sure that WX methods get executed in the WX thread and cherrypy methods get executed in the cherrypy thread.