Here is my code:
async def runTaskWrapped(options):
layoutz = [[sg.Text("Running...", key="runstatus")]];
windowz = sg.Window("New Task", layoutz);
x = threading.Thread(target=runTask, args=(options,));
x.start();
startTime = time.time();
while True:
eventz, valuesz = windowz.read(timeout=100)
if eventz == sg.WIN_CLOSED:
if x.is_alive():
continue
break
if x.is_alive() == False:
x.join()
windowz.FindElement('runstatus').Update(value='Done! Check the new log.txt for more info.');
break;
else:
windowz.FindElement('runstatus').Update(value='Running... (' + str(math.floor(time.time()-startTime)) + ')')
asyncio.run(runTaskWrapped(options));
I have tried everything and it still seems that execution pauses after asyncio.run(runTaskWrapped(options));
Any idea why this may be?
EDIT:
I tried threading.Thread, and although it didn't pause execution, pysimplegui (imported as sg) didnt do anything and no window showed up for it like it does when called synchronously.
I tried trio too, but trio paused execution.
trio.run(runTaskWrapped, options);
When you call asyncio.run(some_function()), your program will not go to the next line until some_function() returns. In your case, runTaskWrapped doesn't return until you execute one of its "break" statements.
We deal with this sort of thing all the time. If you call any function f(), your program won't continue until f() returns. That's a familiar concept.
What's different about asyncio is that it creates a loop of its own, called the event loop, and launches some_function() from inside that loop. That allows you to start other tasks from within some_function(), and those other tasks get a chance to execute when some_function() encounters an "await" statement. That's a powerful concept if that's what you need. But it's only useful if you have two or more tasks that need to wait on external resources, like a network or a serial communications link, and one of the tasks can proceed while the other is waiting.
Your function runTaskWrapped does not contain any "await" statements. So asyncio creates an event loop, hands control to runTaskWrapped. That's a blind alley. It is more-or-less an infinite loop and doesn't "await" anything. Therefore there is no way out of runTaskWrapped, and your program is effectively dead at that point.
In order to make use of asyncio you must structure your program to have more than one task containing "await"s.
You are writing a GUI program, which typically means that it already has an event loop of its own. In some cases it is possible to run the GUI's event loop and the asyncio event loop together, but unless you have a specific need to do this it doesn't gain you anything.
You are also trying to use asyncio with multiple threads, and although this is possible it needs to be done quite carefully. You can start other threads just as in any other Python program, but the presence of those other threads doesn't change what happens in your main thread. You must specifically write code to synchronize events between threads.
No matter what you do in those other threads, asyncio.run(some_function()) will not return until its argument is finished.
Related
Maybe it's a very simple question, but I'm new in concurrency. I want to do a python script to run foo.py 10 times simultaneously with a time limit of 60 sec before automatically abort. The script is a non deterministic algorithm, hence all executions takes different times and one will be finished before the others. Once the first ends, I would like to save the execution time, the output of the algorithm and after that kill the rest of the processes.
I have seen this question run multiple instances of python script simultaneously and it looks very similar, but how can I add time limit and the possibility of when the first one finishes the execution, kills the rest of processes?
Thank you in advance.
I'd suggest using the threading lib, because with it you can set threads to daemon threads so that if the main thread exits for whatever reason the other threads are killed. Here's a small example:
#Import the libs...
import threading, time
#Global variables... (List of results.)
results=[]
#The subprocess you want to run several times simultaneously...
def run():
#We declare results as a global variable.
global results
#Do stuff...
results.append("Hello World! These are my results!")
n=int(input("Welcome user, how much times should I execute run()? "))
#We run the thread n times.
for _ in range(n):
#Define the thread.
t=threading.Thread(target=run)
#Set the thread to daemon, this means that if the main process exits the threads will be killed.
t.setDaemon(True)
#Start the thread.
t.start()
#Once the threads have started we can execute tha main code.
#We set a timer...
startTime=time.time()
while True:
#If the timer reaches 60 s we exit from the program.
if time.time()-startTime>=60:
print("[ERROR] The script took too long to run!")
exit()
#Do stuff on your main thread, if the stuff is complete you can break from the while loop as well.
results.append("Main result.")
break
#When we break from the while loop we print the output.
print("Here are the results: ")
for i in results:
print(f"-{i}")
This example should solve your problem, but if you wanted to use blocking commands on the main thread the timer would fail, so you'd need to tweak this code a bit. If you wanted to do that move the code from the main thread's loop to a new function (for example def main(): and execute the rest of the threads from a primary thread on main. This example may help you:
def run():
pass
#Secondary "main" thread.
def main():
#Start the rest of the threads ( in this case I just start 1).
localT=threading.Thread(target=run)
localT.setDaemon(True)
localT.start()
#Do stuff.
pass
#Actual main thread...
t=threading.Thread(target=main)
t.setDaemon(True)
t.start()
#Set up a timer and fetch the results you need with a global list or any other method...
pass
Now, you should avoid global variables at all costs as sometimes they may be a bit buggy, but for some reason the threading lib doesn't allow you to return values from threads, at least i don't know any methods. I think there are other multi-processing libs out there that do let you return values, but I don't know anything about them so I can't explain you anything. Anyways, I hope that this works for you.
-Update: Ok, I was busy writing the code and I didn't read the comments in the post, sorry. You can still use this method but instead of writing code inside the threads, execute another script. You could either import it as a module or actually run it as a script, here's a question that may help you with that:
How to run one python file in another file?
I was attempting to create a thread class that could be terminated by an exception (since I am trying to have the thread wait on an event) when I created the following:
import sys
class testThread(threading.Thread):
def __init__(self):
super(testThread,self).__init__()
self.daemon = True
def run(self):
try:
print('Running')
while 1:
pass
except:
print('Being forced to exit')
test1 = testThread()
test2 = testThread()
print(test1.daemon)
test1.run()
test2.run()
sys.exit()
However, running the program will only print out one Running message, until the other is terminated. Why is that?
The problem is that you're calling the run method.
This is just a plain old method that you implement, which does whatever you put in its body. In this case, the body is an infinite loop, so calling run just loops forever.
The way to start a thread is the start method. This method is part of the Thread class, and what it does is:
Start the thread’s activity.
It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.
So, if you call this, it will start a new thread, make that new thread run your run() method, and return immediately, so the main thread can keep doing other stuff.1 That's what you want here.
1. As pointed out by Jean-François Fabre, you're still not going to get any real parallelism here. Busy loops are never a great idea in multithreaded code, and if you're running this in CPython or PyPy, almost all of that busy looping is executing Python bytecode while holding the GIL, and only one thread can hold the GIL at a time. So, from a coarse view, things look concurrent—three threads are running, and all making progress. But if you zoom in, there's almost no overlap where two threads progress at once, usually not even enough to make up for the small scheduler overhead.
The Story:
I am currently looking through the asyncio basic examples, in particular this one - the simplest possible HTTP client. The main function starts an event loop, runs until the data fetching is complete and closes the event loop:
def main():
loop = get_event_loop()
try:
body = loop.run_until_complete(fetch())
finally:
loop.close()
print(body.decode('latin-1'), end='')
But, the code also works if I omit the loop.close():
def main():
loop = get_event_loop()
body = loop.run_until_complete(fetch())
print(body.decode('latin-1'), end='')
The Question:
While there is an example, the question is a generic one - what can potentially go wrong if one would forget to close the asyncio event loop? Is the event loop going to be always implicitly closed?
.close() can be used by different event loop implementations to free up system resources allocated by the loop (or do anything else). If you'll take a look at the code of _UnixSelectorEventLoop, which is the (default) IOLoop used in Linux, you would find the following code:
def close(self):
super().close()
for sig in list(self._signal_handlers):
self.remove_signal_handler(sig)
Here, for example, close() removes signal handlers registered with loop.add_signal_handler().
As multiple IOLoops can be started on different threads, or new IOLoops can be created after an old one is closed, (see asyncio.new_event_loop()), closing them should be considered as a good habit.
Update
Starting with Python 3.7 it is recommended to use asyncio.run instead of run_until_complete():
# Python 3.7+
def main():
body = asyncio.run(fetch())
print(body.decode('latin-1'), end='')
Among other things, asyncio.run takes care of finally close()ing the loop.
I wrote this code to lock a mouse in the middle of the screen
def lockmouse():
print "here"
while True:
win32api.SetCursorPos((GetSystemMetrics(0)/2,GetSystemMetrics(1)/2))
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTDOWN,GetSystemMetrics(0)/2,GetSystemMetrics(1)/2,0,0)
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTUP,GetSystemMetrics(0)/2,GetSystemMetrics(1)/2,0,0)
t = threading.Thread(target=lockmouse())
command = "lockmouse"
if "lockmouse" in command:
if t.is_alive==False:
t.start()
time.sleep(3)
t._Thread_stop()
and its not keep going after the t.start().I've been trying using different methods to stop the thread,but its even not make it after that line.anyone know what the problem?
It might be the fact that your function isn't indented properly. It should be:
def foo():
return 'bar'
Also, you seem to only be starting a single thread. What's the point?
EDIT:
I just realised that your function has an infinite loop. The program cannot carry on from t.start() because it has to wait for that function to finish execution, which it won't because there's a while loop. You either need to restructure your program somehow, or if you want to keep it how it is, see this answer for how to avoid waiting for a thread.
How to avoid waiting for a thread to finish execution - Python
This has been discussed many, many times, but I still don't have a good grasp on how to best accomplish this.
Suppose I have two threads: a main app thread and a worker thread. The main app thread (say it's a WXWidgets GUI thread, or a thread that is looping and accepting user input at the console) could have a reason to stop the worker thread - the user's closing the application, a stop button was clicked, some error occurred in the main thread, whatever.
Commonly suggested is to setup a flag that the thread checks frequently to determine whether to exit. I have two problems with the suggested ways to approach this, however:
First, writing constant checks of a flag into my code makes my code really ugly, and it's very, very prone to problems due to the huge amount of code duplication. Take this example:
def WorkerThread():
while (True):
doOp1() # assume this takes say 100ms.
if (exitThread == True):
safelyEnd()
return
doOp2() # this one also takes some time, say 200ms
if (exitThread == True):
safelyEnd()
return
if (somethingIsTrue == True):
doSomethingImportant()
if (exitThread == True): return
doSomethingElse()
if (exitThread == True): return
doOp3() # this blocks for an indeterminate amount of time - say, it's waiting on a network respond
if (exitThread == True):
safelyEnd()
return
doOp4() # this is doing some math
if (exitThread == True):
safelyEnd()
return
doOp5() # This calls a buggy library that might block forever. We need a way to detect this and kill this thread if it's stuck for long enough...
saveSomethingToDisk() # might block while the disk spins up, or while a network share is accessed...whatever
if (exitThread == True):
safelyEnd()
return
def safelyEnd():
cleanupAnyUnfinishedBusiness() # do whatever is needed to get things to a workable state even if something was interrupted
writeWhatWeHaveToDisk() # it's OK to wait for this since it's so important
If I add more code or change code, I have to make sure I'm adding those check blocks all over the place. If my worker thread is a very lengthy thread, I could easily have tens or even hundreds of those checks. Very cumbersome.
Think of the other problems. If doOp4() does accidentally deadlock, my app will spin forever and never exit. Not a good user experience!
Using daemon threads isn't really a good option either because it denies me the opportunity to execute the safelyEnd() code. This code might be important - flushing disk buffers, writing log data for debugging purposes, etc.
Second, my code might call functions that block where I don't have the opportunity to check frequently. Let's say this function exists but it's in code that I don't have access to - say part of a library:
def doOp4():
time.sleep(60) # imagine that this is a network thread, that waits for 60 seconds for a reply before returning.
If that timeout is 60 seconds, even if my main thread gives the signal for the thread to end, it still might sit there for 60 seconds, when it would be perfectly reasonable for it to just stop waiting for a network response and exit. If that code is part of a library I didn't write, however, I have no control over how that works.
Even if I did write the code for a network check, I'd basically have to refactor it so that rather than waiting 60 seconds, it loops 60 times and waits 1 second before checking the exit thread! Again, very messy!
The upshot of all of this, is it feels like a good way to be able to implement this easily would be to somehow cause an exception on a specific thread. If I could do that, I could wrap the entire worker thread's code in a try block, and put the safelyEnd() code in the exception handler, or even a finally block.
Is there a way to either accomplish this, or refactor this code with a different technique that will make things work? The thing is, ideally, when the user requests a quit, we want to make them wait the minimum possible amount. It seems that there has to be a simple way to accomplish this, as this is a very common thing in apps!
Most of the thread communication objects don't allow for this type of setup. They might allow for a cleaner way to have an exit flag, but it still doesn't eliminate the need to constantly check that exit flag, and it still won't deal with the thread blocking because of an external call or because it's simply in a busy loop.
The biggest thing for me is really that if I have a long worker thread procedure I have to litter it with hundreds of checks of the flag. This just seems way too messy and doesn't feel like it's very good coding practice. There has to be a better way...
Any advice would be greatly appreciated.
First, you can make this a lot less verbose and repetitive by using an exception, without needing the ability to raise exceptions into the thread from outside, or any other new tricks or language features:
def WorkerThread():
class ExitThreadError(Exception):
pass
def CheckEnd():
if exitThread:
raise ExitThreadError()
try:
while True:
doOp1() # assume this takes say 100ms.
CheckEnd()
doOp2() # this one also takes some time, say 200ms
CheckEnd()
# etc.
except ExitThreadError:
safelyEnd()
Note that you really ought to be guarding exitThread with a Lock or Condition—which is another good reason to wrap up the check, so you only need to fix that in one place.
Anyway, I've taken out some excessive parentheses, == True checks, etc. that added nothing to the code; hopefully you can still see how it's equivalent to the original.
You can take this even farther by restructuring your function into a simple state machine; then you don't even need an exception. I'll show a ridiculously trivial example, where every state always implicitly transitions to the next state no matter what. For this case, the refactor is obviously reasonable; whether it's reasonable for your real code, only you can really tell.
def WorkerThread():
states = (doOp1, doOp2, doOp3, doOp4, doOp5)
current = 0
while not exitThread:
states[current]()
current += 1
safelyEnd()
Neither of these does anything to help you interrupt in the middle of one of your steps.
If you have some function that takes 60 seconds and there's not a damn thing you can do about it, then there's no way to cancel your thread during those 60 seconds and there's not a damn thing you can do about it. That's just the way it is.
But usually, things that take 60 seconds are really doing something like blocking on a select, and there is something you can do about that—create a pipe, stick its read end in the select, and write on the other end to wake up the thread.
Or, in you're feeling hacky, often just closing/deleting/etc. a file or other object that the function is waiting on/processing/otherwise using will often guarantee that it fails quickly with an exception. Of course sometimes it guarantees a segfault, or corrupted data, or a 50% chance of exiting and a 50% chance of hanging forever, or… So, even if you can't control that doOp4 function, you'd better be able to analyze its source and/or whitebox test it.
If worst comes to worst, then yes, you do have to either change that one 60-second timeout into 60 1-second timeouts. But usually it won't come to that.
Finally, if you really do need to be able to kill a thread, don't use a thread, use a child process. Those are killable.
Just make sure that your process is always in a state where it's safe to kill it—or, if you only care about Unix, use a USR signal and mask it out when the process isn't in a safe-to-kill state.
But if it's not safe to kill your process in the middle of that 60-second doOp4 call, this isn't really going to help you, because you still won't be able to kill it during those 60 seconds.
In some cases, you can have the child process arrange for the parent to clean up for it if it gets killed unexpectedly, or even arrange for it to be cleaned up on the next run (e.g., think of a typical database journal).
But ultimately, what you're asking for is ultimately a contradiction: You want to hard-kill a thread without giving it a chance to finish what it's doing, but you want to guarantee that it finishes what it's doing, and you don't want to rewrite the code to make that possible. So, you need to rethink your design so that it requires something that isn't impossible.
If you do not mind your code running about ten times slower, you can use the Thread2 class implemented below. An example follows that shows how calling the new stop method should kill the thread on the next bytecode instruction. Implementing a cleanup system is left as an exercise for the reader to accomplish.
import threading
import sys
class StopThread(StopIteration): pass
threading.SystemExit = SystemExit, StopThread
class Thread2(threading.Thread):
def stop(self):
self.__stop = True
def _bootstrap(self):
if threading._trace_hook is not None:
raise ValueError('Cannot run thread with tracing!')
self.__stop = False
sys.settrace(self.__trace)
super()._bootstrap()
def __trace(self, frame, event, arg):
if self.__stop:
raise StopThread()
return self.__trace
class Thread3(threading.Thread):
def _bootstrap(self, stop_thread=False):
def stop():
nonlocal stop_thread
stop_thread = True
self.stop = stop
def tracer(*_):
if stop_thread:
raise StopThread()
return tracer
sys.settrace(tracer)
super()._bootstrap()
################################################################################
import time
def main():
test = Thread2(target=printer)
test.start()
time.sleep(1)
test.stop()
test.join()
def printer():
while True:
print(time.time() % 1)
time.sleep(0.1)
if __name__ == '__main__':
main()
The Thread3 class appears to run code approximately 33% faster than the Thread2 class.