I have a python script that uses threads and makes lots of HTTP requests. I think what's happening is that while a HTTP request (using urllib2) is reading, it's blocking and not responding to CtrlC to stop the program. Is there any way around this?
On Windows, the only sure way is to use CtrlBreak. Stops every python script instantly!
(Note that on some keyboards, "Break" is labeled as "Pause".)
Pressing Ctrl + c while a python program is running will cause python to raise a KeyboardInterrupt exception. It's likely that a program that makes lots of HTTP requests will have lots of exception handling code. If the except part of the try-except block doesn't specify which exceptions it should catch, it will catch all exceptions including the KeyboardInterrupt that you just caused. A properly coded python program will make use of the python exception hierarchy and only catch exceptions that are derived from Exception.
#This is the wrong way to do things
try:
#Some stuff might raise an IO exception
except:
#Code that ignores errors
#This is the right way to do things
try:
#Some stuff might raise an IO exception
except Exception:
#This won't catch KeyboardInterrupt
If you can't change the code (or need to kill the program so that your changes will take effect) then you can try pressing Ctrl + c rapidly. The first of the KeyboardInterrupt exceptions will knock your program out of the try block and hopefully one of the later KeyboardInterrupt exceptions will be raised when the program is outside of a try block.
If it is running in the Python shell use Ctrl + Z, otherwise locate the python process and kill it.
The interrupt process is hardware and OS dependent. So you will have very different behavior depending on where you run your python script. For example, on Windows machines we have Ctrl+C (SIGINT) and Ctrl+Break (SIGBREAK).
So while SIGINT is present on all systems and can be handled and caught, the SIGBREAK signal is Windows specific (and can be disabled in CONFIG.SYS) and is really handled by the BIOS as an interrupt vector INT 1Bh, which is why this key is much more powerful than any other. So if you're using some *nix flavored OS, you will get different results depending on the implementation, since that signal is not present there, but others are. In Linux you can check what signals are available to you by:
$ kill -l
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP
6) SIGABRT 7) SIGEMT 8) SIGFPE 9) SIGKILL 10) SIGBUS
11) SIGSEGV 12) SIGSYS 13) SIGPIPE 14) SIGALRM 15) SIGTERM
16) SIGURG 17) SIGSTOP 18) SIGTSTP 19) SIGCONT 20) SIGCHLD
21) SIGTTIN 22) SIGTTOU 23) SIGIO 24) SIGXCPU 25) SIGXFSZ
26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGPWR 30) SIGUSR1
31) SIGUSR2 32) SIGRTMAX
So if you want to catch the CTRL+BREAK signal on a linux system you'll have to check to what POSIX signal they have mapped that key. Popular mappings are:
CTRL+\ = SIGQUIT
CTRL+D = SIGQUIT
CTRL+C = SIGINT
CTRL+Z = SIGTSTOP
CTRL+BREAK = SIGKILL or SIGTERM or SIGSTOP
In fact, many more functions are available under Linux, where the SysRq (System Request) key can take on a life of its own...
Ctrl+D Difference for Windows and Linux
It turns out that as of Python 3.6, the Python interpreter handles Ctrl+C differently for Linux and Windows. For Linux, Ctrl+C would work mostly as expected however on Windows Ctrl+C mostly doesn't work especially if Python is running blocking call such as thread.join or waiting on web response. It does work for time.sleep, however. Here's the nice explanation of what is going on in Python interpreter. Note that Ctrl+C generates SIGINT.
Solution 1: Use Ctrl+Break or Equivalent
Use below keyboard shortcuts in terminal/console window which will generate SIGBREAK at lower level in OS and terminate the Python interpreter.
Mac OS and Linux
Ctrl+Shift+\ or Ctrl+</kbd>
Windows:
General: Ctrl+Break
Dell: Ctrl+Fn+F6 or Ctrl+Fn+S
Lenovo: Ctrl+Fn+F11 or Ctrl+Fn+B
HP: Ctrl+Fn+Shift
Samsung: Fn+Esc
Solution 2: Use Windows API
Below are handy functions which will detect Windows and install custom handler for Ctrl+C in console:
#win_ctrl_c.py
import sys
def handler(a,b=None):
sys.exit(1)
def install_handler():
if sys.platform == "win32":
import win32api
win32api.SetConsoleCtrlHandler(handler, True)
You can use above like this:
import threading
import time
import win_ctrl_c
# do something that will block
def work():
time.sleep(10000)
t = threading.Thread(target=work)
t.daemon = True
t.start()
#install handler
install_handler()
# now block
t.join()
#Ctrl+C works now!
Solution 3: Polling method
I don't prefer or recommend this method because it unnecessarily consumes processor and power negatively impacting the performance.
import threading
import time
def work():
time.sleep(10000)
t = threading.Thread(target=work)
t.daemon = True
t.start()
while(True):
t.join(0.1) #100ms ~ typical human response
# you will get KeyboardIntrupt exception
This post is old but I recently ran into the same problem of Ctrl+C not terminating Python scripts on Linux. I used Ctrl+\ (SIGQUIT).
On Mac press Ctrl+\ to quit a python process attached to a terminal.
Capture the KeyboardInterrupt (which is launched by pressing ctrl+c) and force the exit:
from sys import exit
try:
# Your code
command = input('Type your command: ')
except KeyboardInterrupt:
# User interrupt the program with ctrl+c
exit()
On a mac / in Terminal:
Show Inspector (right click within the terminal window or Shell >Show Inspector)
click the Settings icon above "running processes"
choose from the list of options under "Signal Process Group" (Kill, terminate, interrupt, etc).
Forcing the program to close using Alt+F4 (shuts down current program)
Spamming the X button on CMD for e.x.
Taskmanager (first Windows+R and then "taskmgr") and then end the task.
Those may help.
You can open your task manager (ctrl + alt + delete, then go to task manager) and look through it for python and the server is called (for the example) _go_app (naming convention is: _language_app).
If I end the _go_app task it'll end the server, so going there in the browser will say it "unexpectedly ended", I also use git bash, and when I start a server, I cannot break out of the server in bash's shell with ctrl + c or ctrl + pause, but once you end the python task (the one using 63.7 mb) it'll break out of the server script in bash, and allow me to use the git bash shell.
For the record, what killed the process on my Raspberry 3B+ (running raspbian) was Ctrl+'. On my French AZERTY keyboard, the touch ' is also number 4.
Related
Currently whenever I press CTRL + Z on a lengthy script I was given, it immediately terminates the script ([1+] stopped(SIGTSTP) ./test.py) which is what I want, but it also leaves the python2 process running (when I type ps to look at processes), which forces me to use killall -9 python2, which I do not want to do every time. Is there a way to immediately terminate a script that doesn't leave the python2 process running in the background?
There is no SIGTSTP currently in the code that I see but I did try using the following code with no luck. It didn't even exit the script when I pressed CTRL + Z.
def handler(signum, frame):
sys.exit("CTRL+Z pressed. Exiting Test")
signal.signal(signal.SIGTSTP, handler)
SIGSTP is a signal to suspend a process, it sounds like you want to terminate a process. You can try sending Ctrl-C or CTRL-D instead, which should send a SIGINT signal.
I believe you could also try CTRL-\ which sends SIGQUIT.
Use Ctrl + C. SIGTSTP suspends the process, hence why it keeps it open, but does not terminate it.
(Note: On a Linux terminal use Ctrl + \, otherwise use Ctrl + C or Ctrl + D)
Or just use sys.exit()
I'm writing some code in Python 3 on Windows that looks like this:
try:
do something that takes a long time
(training a neural network in TensorFlow, as it happens)
except KeyboardInterrupt:
print('^C')
print a summary of results
still useful even if the training was cut short early
This works perfectly if run directly from the console with python foo.py.
However, if the call to Python was within a batch file, it ends up doing all the above but then still spamming the console with the 'terminate batch job' prompt.
Is there a way to stop that happening? By fully eating the ^C within Python, jumping all the way out of the batch file or otherwise?
Use the break (More info here) command in the batch file, which will disable CTRL+C halting the file
EDIT: According to this site of the break command
Newer versions of Windows (Windows ME, Windows 2000, Windows XP, and higher) only include this command for backward compatibility and turning the break off has no effect.
I personally tested this, and can confirm, I will edit when I find a workaround
EDIT #2: If you could have a second batch script that runs start "" /b /wait cmd /c "yourfile.bat" although, this is known to cause glitches with other nested batch files
The flag to disable Ctrl+C is inherited by child processes, so Python will no longer raise a KeyboardInterrupt. Plus we still have bugs here in Python if reading from the console gets interrupted by Ctrl+C without getting a SIGINT from the CRT. The Python script should manually enable Ctrl+C via ctypes. Use import ctypes; kernel32 = ctypes.WinDLL('kernel32', use_last_error=True); success = kernel32.SetConsoleCtrlHandler(None, False)
EDIT #3 As pointed by Eryksyn (in the comments), you can use cytpes to ENABLE it;
import ctypes;
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True); success = kernel32.SetConsoleCtrlHandler(None, False)
EDIT #4: I think I found it, try this (Although it may not work) Can you use the threading import?
import time
from threading import Thread
def noInterrupt():
for i in xrange(4):
print i
time.sleep(1)
a = Thread(target=noInterrupt)
a.start()
a.join()
print "done"
I have this simple python script called myMain.py to execute another python program automatically with incremental number, and I'm running it on CentOS 7:
#!/usr/bin/python
import os
import sys
import time
def main():
step_indicator = ""
arrow = ">"
step = 2
try:
for i in range(0,360, step):
step_percentage = float(i)/360.0 * 100
if i % 10 == 0:
step_indicator += "="
os.system("python myParsePDB.py -i BP1.pdb -c 1 -s %s" % step)
print("step_percentage%s%s%.2f" % (step_indicator,arrow,step_percentage)+"%")
except KeyboardInterrupt:
print("Stop me!")
sys.exit(0)
if __name__ == "__main__":
main()
For now I only know this script is single thread safe, but I can't terminate it with Ctrl+C keyboard interruption.
I have read some relative questions: such as Cannot kill Python script with Ctrl-C and Stopping python using ctrl+c I realized that Ctrl+Z does not kill the process, it only pauses the process and keep the process in background. Ctrl+Break does work for my case either, I think it only terminates my main thread but keeps the child process.
I also noticed that calling os.system() will spawn a child process from the current executing process. At the same time, I also have os file I/O functions and os.system("rm -rf legacy/*") will be invoked in myParsePDB.py which means this myParsePDB.py child process will spawn child process as well. Then, if I want to catch Ctrl+C in myMain.py, should I daemon only myMain.py or should I daemon each process when they spawn?
This is a general problem that could raise when dealing with signal handling. Python signal is not an exception, it's a wrapper of operating system signal. Therefore, signal processing in python depends on operating system, hardware and many conditions. However, how to deal with these problem is similar.
According to this tutorial, I'll quote the following paragraphs: signal – Receive notification of asynchronous system events
Signals are an operating system feature that provide a means of
notifying your program of an event, and having it handled
asynchronously. They can be generated by the system itself, or sent
from one process to another. Since signals interrupt the regular flow
of your program, it is possible that some operations (especially I/O)
may produce error if a signal is received in the middle.
Signals are identified by integers and are defined in the operating
system C headers. Python exposes the signals appropriate for the
platform as symbols in the signal module. For the examples below, I
will use SIGINT and SIGUSR1. Both are typically defined for all Unix
and Unix-like systems.
In my code:
os.system("python myParsePDB.py -i BP1.pdb -c 1 -s %s" % step) inside the for loop will be executed for a bit of time and will spend some time on I/O files. If the keyboard interrupt is passing too fast and do not catch asynchronously after writing files, the signal might be blocked in operating system, so my execution will still remain the try clause for loop. (Errors detected during execution are called exceptions and are not unconditionally fatal: Python Errors and Exceptions).
Therefore the simplest way to make them asynchonous is wait:
try:
for i in range(0,360, step):
os.system("python myParsePDB.py -i BP1.pdb -c 1 -s %s" % step)
time.sleep(0.2)
except KeyboardInterrupt:
print("Stop me!")
sys.exit(0)
It might hurt performance but it guaranteed that the signal can be caught after waiting the execution of os.system(). You might also want to use other sync/async functions to solve the problem if better performance is required.
For more unix signal reference, please also look at: Linux Signal Manpage
Please don't consider it a duplicate before reading, There are a lot of questions about multithreading and keyboard interrupt, but i didn't find any considering os.system and it looks like it's important.
I have a python script which makes some external calls in worker threads.
I want it to exit if I press ctrl+c But it look like the main thread ignores it.
Something like this:
from threading import Thread
import sys
import os
def run(i):
while True:
os.system("sleep 10")
print i
def main():
threads=[]
try:
for i in range(0, 3):
threads.append(Thread(target=run, args=(i,)))
threads[i].daemon=True
threads[i].start()
for i in range(0, 3):
while True:
threads[i].join(10)
if not threads[i].isAlive():
break
except(KeyboardInterrupt, SystemExit):
sys.exit("Interrupted by ctrl+c\n")
if __name__ == '__main__':
main()
Surprisingly, it works fine if I change os.system("sleep 10") to time.sleep(10).
I'm not sure what operating system and shell you are using. I describe Mac OS X and Linux with zsh (bash/sh should act similar).
When you hit Ctrl+C, all programs running in the foreground in your current terminal receive the signal SIGINT. In your case it's your main python process and all processes spawned by os.system.
Processes spawned by os.system then terminate their execution. Usually when python script receives SIGINT, it raises KeyboardInterrupt exception, but your main process ignores SIGINT, because of os.system(). Python os.system() calls the Standard C function system(), that makes calling process ignore SIGINT (man Linux / man Mac OS X).
So neither of your python threads receives SIGINT, it's only children processes who get it.
When you remove os.system() call, your python process stops ignoring SIGINT, and you get KeyboardInterrupt.
You can replace os.system("sleep 10") with subprocess.call(["sleep", "10"]). subprocess.call() doesn't make your process ignore SIGINT.
I've had this same problem more times than I could count back when i was first learning python multithreading.
Adding the sleep call within the loop makes your main thread block, which will allow it to still hear and honor exceptions. What you want to do is utilize the Event class to set an event in your child threads that will serve as an exit flag to break execution upon. You can set this flag in your KeyboardInterrupt exception, just put the except clause for that in your main thread.
I'm not entirely certain what is going on with the different behaviors between the python specific sleep and the os called one, but the remedy I am offering should work for what your desired end result is. Just offering a guess, the os called one probably blocks the interpreter itself in a different way?
Keep in mind that generally in most situations where threads are required the main thread is going to keep executing something, in which case the "sleeping" in your simple example would be implied.
http://docs.python.org/2/library/threading.html#event-objects
I am currently working on a wrapper for a dedicated server running in the shell. The wrapper spawns the server process via subprocess and observes and reacts to its output.
The dedicated server must be explicitly given a command to shut down gracefully. Thus, CTRL-C must not reach the server process.
If I capture the KeyboardInterrupt exception or overwrite the SIGINT-handler in python, the server process still receives the CTRL-C and stops immediately.
So my question is:
How to prevent subprocesses from receiving CTRL-C / Control-C / SIGINT?
Somebody in the #python IRC-Channel (Freenode) helped me by pointing out the preexec_fn parameter of subprocess.Popen(...):
If preexec_fn is set to a callable
object, this object will be called in
the child process just before the
child is executed. (Unix only)
Thus, the following code solves the problem (UNIX only):
import subprocess
import signal
def preexec_function():
# Ignore the SIGINT signal by setting the handler to the standard
# signal handler SIG_IGN.
signal.signal(signal.SIGINT, signal.SIG_IGN)
my_process = subprocess.Popen(
["my_executable"],
preexec_fn = preexec_function
)
Note: The signal is actually not prevented from reaching the subprocess. Instead, the preexec_fn above overwrites the signal's default handler so that the signal is ignored. Thus, this solution may not work if the subprocess overwrites the SIGINT handler again.
Another note: This solution works for all sorts of subprocesses, i.e. it is not restricted to subprocesses written in Python, too. For example the dedicated server I am writing my wrapper for is in fact written in Java.
Combining some of other answers that will do the trick - no signal sent to main app will be forwarded to the subprocess.
import os
from subprocess import Popen
def preexec(): # Don't forward signals.
os.setpgrp()
Popen('whatever', preexec_fn = preexec)
you can do something like this to make it work in windows and unix:
import subprocess
import sys
def pre_exec():
# To ignore CTRL+C signal in the new process
signal.signal(signal.SIGINT, signal.SIG_IGN)
if sys.platform.startswith('win'):
#https://msdn.microsoft.com/en-us/library/windows/desktop/ms684863(v=vs.85).aspx
#CREATE_NEW_PROCESS_GROUP=0x00000200 -> If this flag is specified, CTRL+C signals will be disabled
my_sub_process=subprocess.Popen(["executable"], creationflags=0x00000200)
else:
my_sub_process=subprocess.Popen(["executable"], preexec_fn = pre_exec)
After an hour of various attempts, this works for me:
process = subprocess.Popen(["someprocess"], creationflags=subprocess.DETACHED_PROCESS | subprocess.CREATE_NEW_PROCESS_GROUP)
It's solution for windows.
Try setting SIGINT to be ignored before spawning the subprocess (reset it to default behavior afterward).
If that doesn't work, you'll need to read up on job control and learn how to put a process in its own background process group, so that ^C doesn't even cause the kernel to send the signal to it in the first place. (May not be possible in Python without writing C helpers.)
See also this older question.