Python Curses while Multithreading - python

I have a multithreaded program written in Python where I have a number of things happening at the same time:
reading raw data from an external source
organizing such data into different lists (parsing)
saving the post-parsed data into mass storage
some threads being used to flush some buffers from time to time and other optimizations
displaying parts of such data for real-time monitoring, using Curses.
The program is latency-sensitive, so I really need this to be multithreaded.
I got the curses thread to display correctly what I want to.
The problem is that while I had everything working without the curses thread, I had a "killswitch" in the main() function that terminated all activity at the press of a key.
I have this global variable called "killThreads" that goes into all functions who are called as threads, and all these functions only work as:
def oneThread():
while (not killThreads):
doStuff()
...
And then the main function defines the killThread as False, initializes all threads and turns the killThread as True after a raw_input():
killThreads=False
thisThread=threading.Thread(target=oneThread)
otherThread=threading.Thread(target=twoThread)
thisThread.setDaemon(True)
otherThread.setDaemon(True)
thisThread.start()
otherThread.start()
raw_input('Press to end the program')
killThreads=True
Everything ran fine until I ran a thread with the Curses module to display data.
It seems that while the Curses thread is on, it takes over all input commands. I tried to use getch() with no success. All I could do to keep everything running was to establish a timer within the Curses function:
def displayData():
screen=curses.initscr()
screen.nodelay(1)
timeKill=0
while (timeKill<80):
#stuff is drawn#
time.sleep(0.25)
timeKill+=1
Could anyone tell me how to go over Curses and get my keyboard input to "reach" the main function and kill all threads? Or do I always have to input to Curses and then make the Curses function alter the killThreads variable? If so, how do I do it (or where do I find the documentation for that)?
Thank you so much for your help.

Nice to meet you.
I'm trying to accomplish the same today. Look at this solution:
killThreads=False
thisThread=threading.Thread(target=oneThread)
otherThread=threading.Thread(target=twoThread)
thisThread.setDaemon(True)
otherThread.setDaemon(True)
thisThread.start()
otherThread.start()
raw_input('Press "q" to end the program')
key = ''
while key != ord('q'):
key = screen.getch()
killThreads=True
curses.nocbreak(); screen.keypad(0); curses.echo()
curses.endwin()
See, while will looping very fastly and waiting for q button be pressed, before switch your var killThreads to True.
It is pretty common practice. However, this while loop making thousands of idle loops in second, may be there can be more elegant way or better to embed into this while loop time.sleep(0.1) at least.

Related

I created a python bind for CS:GO with pyautogui but it doesnt work

I created a simple bind script. It works on IDLE Python but it doesn't work in CS:GO. Do you know why?
Mayby it must be on background to work?
import keyboard
import pyautogui
import time
def EventListen():
while True:
try:
if keyboard.is_pressed('n'):
pyautogui.press('`')
pyautogui.typewrite('say EZ')
pyautogui.press('enter')
pyautogui.press('`')
EventListen()
except:
EventListen()
EventListen()
I don't see the need to use pyautogui since you are already using keyboard which is sufficient to perform the tasks you need. I have made some changes to your code
import time
import keyboard
def EventListen():
while True:
try:
if keyboard.is_pressed('n'):
keyboard.press('`')
keyboard.write('say EZ')
keyboard.press('enter')
keyboard.press('`')
elif keyboard.is_pressed('/'): #add something to end the process
break
except:
EventListen()
time.sleep(0.001)
EventListen()
There is no need to call the function in the while loop, as it will anyway be executed infinitely unless you kill the process. I don't see why the script wouldn't run in the background, in fact I am typing this
n`say EZ
`
using the script. What might be possible is that your previous program ran continuously, causing high CPU usage which might have competed with the game's demand. I recomend you to add a small delay before every iteration of the while loop, in this case I have added 1 ms delay, which will cause significant reduction in CPU usage. I am not sure if that solved your problem as I am unable to reproduce your exact case, let me know if it helped.
EDIT : I forgot to mention, I have added another binding of keyboard.is_pressed('/') which will make the program break out of the loop and hence terminate it when / key is pressed. You can change this as you like. If you don't want any other binding as such (which I don't recommend) then you can rely on manually killing the task.
you should make an exe with pyinstaller and you run it background

Send a variable from outside the python console OR stop a script from outside

I wrote a Python script that executes an optimization and runs days to get a solution (due to the costly objective function). In all days work it will be sufficient to just stop the calculation at some point because the solution is good enough for me (but not for the optimization algorithm).
The problem is, I can always abort hitting Ctrl+C. But then there is no chance to nicely output the current best parameters, plot the data, save it etc. It would be great to stop the script in a controlled way after the next calculation of the objective function. So my thought was so question some variable (if user_stop=True) and programatically stop the optimization. But how to set such a variable? The python console is blocked during execution.
I thought about setting the content of a text file and reading it in each iteration but it's more than poor and hard to explain for other users of the script. Theoretically, I could also ask the user for an input but than the script won't run automatically (which it should until someone decides to stop).
Any ideas for my problem?
Basically that's it - stop the loop at some point but execute the print:
a = 0
while True:
a = a + 1
print(a)
If you poll your "variable" infrequently (say at most once every 20 seconds) then the overhead of testing for a file is negligible. Something like
import os
QUITFILE = "/home/myscript/quit_now.txt"
# and for convenience, delete any old QUITFILE that may exist at init time
... # days later
if os.path.isfile( QUITFILE)
# tidy up, delete QUITFILE, and exit
Then just echo please > home/myscript/quit_now.txt to tell your program to exit.
maybe you can use a do-while loop. holding your target in a varible
outside the loop and start looping the calculatio while <= your target calculation.
For Windows, I would use msvcrt.getch()
For example, this script will loop until a key is pressed, then, if it is q, prompt for the user to quit: (Note that the if statement uses 'short circuiting' to only evaluate the getch() - which is blocking - when we know that a key has been pressed.)
import msvcrt, time
while True: #This is your optimization loop
if msvcrt.kbhit() and msvcrt.getch() == 'q':
retval = raw_input('Quit? (Y/N) >')
if retval.lower() == 'y':
print 'Quitting'
break #Or set a flag...
else:
time.sleep(1)
print('Processing...')
If you place this if block at a point in the optimization loop where it will be frequently run, it will allow you to sop at a convenient point, or at least set a flag which you can check for at the end of each optimization run.
If you cannot place it somewhere where it will be frequently checked, then you can look at handling the KeyboardInterrupt raised by Ctrl-C
If you are running on Linux, or need cross-platform capability, have a look at this answer for getting the keypress.

Performing an action upon unexpected exit python

I was wandering if there was a way to perform an action before the program closes. I am running a program over a long time and I do want to be able to close it and have the data be saved in a text file or something but there is no way of me interfering with the while True loop I have running, and simply saving the data each loop would be highly ineffective.
So is there a way that I can save data, say a list, when I hit the x or destroy the program? I have been looking at the atexit module but have had no luck, except when I set the program to finish at a certain point.
def saveFile(list):
print "Saving List"
with open("file.txt", "a") as test_file:
test_file.write(str(list[-1]))
atexit.register(saveFile(list))
That is my whole atexit part of the code and like I said, it runs fine when I set it to close through the while loop.
Is this possible, to save something when the application is terminated?
Your atexit usage is wrong. It expects a function and its arguments, but you're just calling your function right away and passing the result to atexit.register(). Try:
atexit.register(saveFile, list)
Be aware that this uses the list reference as it exists at the time you call atexit.register(), so if you assign to list afterwards, those changes will not be picked up. Modifying the list itself without reassigning should be fine, though.
You could use the handle_exit context manager from this ActiveState recipe:
http://code.activestate.com/recipes/577997-handle-exit-context-manager/
It handles SystemExit, KeyboardInterrupt, SIGINT, and SIGTERM, with a simple interface:
def cleanup():
print 'do some cleanup here'
def main():
print 'do something'
if __name__ == '__main__':
with handle_exit(cleanup):
main()
There's nothing you can in reaction to a SIGKILL. It kills your process immediately, without any allowed cleanup.
Catch the SystemExit exception at the top of your application, then rethrow it.
There are a a couple of approaches to this. As some have commented you could used signal handling ... your [Ctrl]+[C] from the terminal where this is running in the foreground is dispatching a SIGHUP signal to your process (from the terminal's drivers).
Another approach would be to use a non-blocking os.read() on sys.stdin.fileno such that you're polling your keyboard one during every loop to see if an "exit" keystroke or sequence has been entered.
A similarly non-blocking polling approach can be implemented using the select module's functionality. I've see that used with the termios and tty modules. (Seems inelegant that it needs all those to save, set changes to, and restore the terminal settings, and I've also seen some examples using os and fcntl; and I'm not sure when or why one would prefer one over the other if os.isatty(sys.stdin.fileno())).
Yet another approach would be to use the curses module with window.nodelay() or window.timeout() to set your desired input behavior and then either window.getch() or window.getkey() to poll for any input.

Avoiding raw_input to take keys pressed while in a loop for windows

I am trying to make a program which has a raw_input in a loop, if anyone presses a key while the long loop is running the next raw_input takes that as input, how do I avoid that?
I don't know what else to add to this simple question. Do let me know if more is required.
EDIT
Some code
for i in range(1000):
var = raw_input("Enter the number")
#.... do some long magic and stuff here which takes afew seconds
print 'Output is'+str(output)
So if someone presses something inside the magic phase, that is take as the input for the next loop. That is where the problem begins. (And yes the loop has to run for 1000 times).
This works for me with Windows 7 64bit, python 2.7.
import msvcrt
def flush_input():
while msvcrt.kbhit():
msvcrt.getch()
I put the OS in the title, window 7 64 bit to be specific. I saw the
answers there. They do apply but by god they are so big. Aren't there
other n00b friendly and safer ways to take inputs?
Let me try to explain why you need to do such an elaborate process. When you press a key it is stored in a section of computer memory called keyboard buffer (not to be confused with stdin buffer). This buffer stores the key's pressed until it is processed by your program. Python doesn't provide any platform independent wrapper to do this task. You have to rely on OS specific system calls to access this buffer, and flush it, read it or query it. msvcrt is a MS VC++ Runtime Library and python msvcrt provides a wrapper over it. Unless you wan't a platform independent solution, it is quite straight forward.
Use msvcrt getch to read a character from console. msvcrt.kbhit() to test if a key press is present in the keyboard buffer and so on. So as MattH has shown, it just a couple of lines code. And if you think you are a noob take this opportunity to learn something new.
Just collect your input outside of the loop (before you enter the loop). Do you really want the user to enter 1000 numbers? well maybe you do. but just include a loop at the top and collect the 1000 numbers at the start, and store them in an array.
then on the bottom half change your loop so it just does all the work. then if someone enters something no the keyboard, it doesn't really matter anymore.
something like this:
def getvars(top=1000):
vars = []
for i in range(0,top):
anum = int(raw_input('%d) Please enter another number: ' % i))
vars.append(anum)
return vars
def doMagic(numbers):
top = len(numbers)
for number in numbers:
# do magic number stuff
print 'this was my raw number %s' % number
if __name__ == "__main__":
numbers = getvars(top=10)
doMagic(numbers)
presented in a different sort of way and less os dependent
There is another way to do it that should work. I don't have a windows box handy to test it out on but its a trick i used to use and its rather undocumented. Perhaps I'm giving away secrets... but its basically like this: trick the os into thinking your app is a screensaver by calling the api that turns on the screensaver function at the start of your magic calculations. at the end of your magic calculations or when you are ready to accept input again, call the api again and turn off the screensaver functionality.
That would work.
There is another way to do it as well. Since you are in windows this will work too. but its a fair amount of work but not really too much. In windows, the window that is foreground (at the top of the Z order) that window gets the 'raw input thread'. The raw input thread receives the mouse and keyboard input. So to capture all input all you need to do is create a function that stands up a transparent or (non transparent) window that sits at the top of the Z order setWindowPos would do the trick , have it cover the entire screen and perhaps display a message such as Even Geduld or Please wait
when you are ready to prompt the user for more input, you use showwindow() to hide the window, show the previous results, get the input and then reshow the window and capture the keys/mouse all over again.
Of course all these solutions tie you to a particular OS unless you implement some sort of try/except handling and/or wrapping of the low level windows SDK calls.

Python: How to quit CLI when stuck in blocking raw_input?

I have a GUI program which should also be controllable via CLI (for monitoring). The CLI is implemented in a while loop using raw_input.
If I quit the program via a GUI close button, it hangs in raw_input and does not quit until it gets an input.
How can I immediately abort raw_input without entering an input?
I run it on WinXP but I want it to be platform independent, it should also work within Eclipse since it is a developer tool. Python version is 2.6.
I searched stackoverflow for hours and I know there are many answers to that topic, but is there really no platform independent solution to have a non-blocking CLI reader?
If not, what would be the best way to overcome this problem?
Thanks
That's not maybe the best solution but you could use the thread module which has a function thread.interrupt_main(). So can run two thread : one with your raw_input method and one which can give the interruption signal. The upper level thread raise a KeyboardInterrupt exception.
import thread
import time
def main():
try:
m = thread.start_new_thread(killable_input, tuple())
while 1:
time.sleep(0.1)
except KeyboardInterrupt:
print "exception"
def killable_input():
w = thread.start_new_thread(normal_input, tuple())
i = thread.start_new_thread(wait_sometime, tuple())
def normal_input():
s = raw_input("input:")
def wait_sometime():
time.sleep(4) # or any other condition to kill the thread
print "too slow, killing imput"
thread.interrupt_main()
if __name__ == '__main__':
main()
Depending on what GUI toolkit you're using, find a way to hook up an event listener to the close window action and make it call win32api.TerminateProcess(-1, 0).
For reference, on Linux calling sys.exit() works.

Categories