After reading A LOT of data on the subject I still couldn't find any actual solution to my problem (there might not be any).
My problem is as following:
In my project I have multiple drivers working with various hardware's (IO managers, programmable loads, power supplies and more).
Initializing connection to these hardware's is costly (in time), and I cant open and then close the connection for every communication iteration between us.
Meaning I cant do this (Assuming programmable load implements enter / exit):
start of code...
with programmable_load(args) as program_instance:
programmable_load_instance.do_something()
rest of code...
So I went for a different solution :
class programmable_load():
def __init__(self):
self.handler = handler_creator()
def close_connection(self):
self.handler.close_connection()
self.handler = None
def __del__(self):
if (self.handler != None):
self.close_connection()
For obvious reasons I dont 'trust' the destructor to actually get called so I explicitly call close_connection() when I want to end my program (for all drivers).
The problem happens when I abruptly terminate the process, for example when I run via debug mode and quit debugging.
In these cases the process terminates without running through any destructors.
I understand that the OS will clear all memory unused at this point, but is there any way to clear the memory in an organized manner?
and if not, is there a way to make the quit debugging function pass through a certain set of functions? Does the python process know it got a quite debugging event or does it treat it as a normal termination?
Operating system: Windows
According to this documentation:
If a process is terminated by TerminateProcess, all threads of the
process are terminated immediately with no chance to run additional
code.
(Emphasis mine.) This implies that there is nothing you can do in this case.
As detailed here, signals don't work very well on ms-windows.
As was mentioned in a comment, you could use atexit to do the cleanup. But that only works if the process is asked to close (e.g. QUIT signal on Linux) and not just killed (as is likely the case when stopping the debugging session). Similarily if you force your computer to turn off (e.g. long press power button or remove power) then it won't be called either. There is no 'solution' to that for obvious reasons. Your program can't expect to be called when the power suddenly goes off or when it is forcefully killed. The point of forcefully killing is to definitely kill the process now. If it first called your clean-up code then you could delay that which defeats the purpose. That is why there are signals such as to ask your process to stop. This is not Python specific. The same concept also applies across operating systems.
Bonus (design suggestion, not a solution): I would argue that you can still make use of the context manager (using with). Your problem is not unique. Database connections are usually kept alive for longer as well. It is a question of the scope. Move the context further up to the application level. Then it is clear what the boundary is and you don't need any magic (you are probably also aware of #contextmanager to make that a breeze).
I haven't tested properly as I don't have wingide installed over here so I can't grant you this will work but what about using setconsolectrlhandler? For instance, try something like this:
import os
import sys
import win32api
if __name__ == "__main__":
def callback(sig, func=None):
print("Exit handler called!")
try:
win32api.SetConsoleCtrlHandler(callback, True)
except Exception as e:
print("Captured exception", e)
sys.exit(1)
print("Press to quit")
input()
print("Bye!")
It'll be able to handle CTRL+C and CTRL+BREAK signals:
Related
I must ensure that the end_event() function is executed at the end of the program. I tried to implement this as Python's atexit. However, when the .py file was converted to an exe file with a PyInstaller and closed with a click, it did not work. I would appreciate it if you could tell me a solution that always works. Have a good day.
import atexit
import signal
import pyupbit
def end_event():
for keys in buy_list.keys():
order = upbit.get_order(keys)
if "state" in order:
if(order['state'] == 'wait'):
upbit.cancel_order(keys)
exit(1)
atexit.register(end_event)
signal.signal(signal.SIGINT, end_event)
There is no way to guarantee that code will run when the process exits, because there are ways to exit which do not permit any code to run.
User may pull the power cable. No code can run because there is no power.
External events (power outage, lightning strike), same as above.
Kill -9 or equivalent, e.g. Windows' TerminateProcess
Patterns for emergency cleanup.
The best pattern is: Never be in a state which requires emergency cleanup.
There is no guarantee that your cleanup will run because you could have a kill -9 or a power outage.
Therefore you need to be able to recover in that scenario.
See also: What does the POSIX standard say about thread stacks in atexit() handlers? What's the OS practice?
I am attempting to convert code from Summerfield's article on (old-style) PyQt Signals/Slots to new-style PySide code. One example is a pure console application, which I have never worked with before. Unfortunately, when I try to run it multiple times, I am told that the previous application is still running.
It is a simple app: it basically lets you set a number, and reports back if the number is new:
from PySide import QtCore
import sys
class TaxRate(QtCore.QObject):
rateChangedSig=QtCore.Signal(float)
def __init__(self):
QtCore.QObject.__init__(self)
self.rate = 17.5
def getRate(self):
return self.rate
def setRate(self, newRate):
if newRate != self.rate:
self.rate = newRate
self.rateChangedSig.emit(self.rate) #was self.emit(SIGNAL("rateChanged"), self.rate)
#QtCore.Slot() #technically not really needed
def rateChangedSlot(value):
print("Tax rate changed to {0:.2f} %".format(value))
if __name__=="__main__":
qtApp = QtCore.QCoreApplication(sys.argv) #origional had QtGui.QApplication, but there is no GUI
vat = TaxRate()
vat.rateChangedSig.connect(rateChangedSlot) #was vat.connect(vat, SIGNAL("rateChanged"), rateChanged)
vat.setRate(8.5) # A change will occur (new rate is different)
qtApp.quit()
sys.exit(qtApp.exec_())
Overall, it works as expected, except the final two lines do not kill the process. When I try to run the program twice, the second time my IDE (Spyder) always tells me that it is already running in a separate process. If I try running it from the command line, the window just hangs.
Strangely, when I comment out the last two lines I do not get this warning. This is the opposite of what I expect (based on previous experience with PySide GUI applications and the documentation for quit()).
Following the Closing a window example at Zetcode, I tried replacing qtApp.quit() with qtApp.instance().quit(), which yielded the same non-killing result.
So, how do I kill this thing?
One idea is that I shouldn't have even started it in the first place (as suggested here). Even though it is a pure console app, Summerfield's original program initializes with app=QtGui.QApplication(sys.argv), and it does not contain the last two lines. Things run fine, multiple times. However, isn't there a concern that each run would create a new process, so his program seems to be effectively multiplying processes without warning? (Note in practice I don't think this is happening on my system, so the answer seems to be 'No' for reasons I don't understand).
What is the correct way to control/initialize/kill a console app using PySide?
(This is ignoring the question, for the time being, why one would ever use PySide for a pure console application in Python as has been pointed out previously. But if anyone were to be interested in answering that separate question, I could start a separate question for it).
Potentially relevant post:
Pyside applications not closing properly
The problem is because you call QCoreApplication.quit() before you call QCoreApplication.exec_(). The call to quit is not queued up in the event loop, it happens immediately. The call to QCoreApplication.exec_() starts the event loop which only ends when a call to QCoreApplication.exit() (or QCoreApplication.quit()) is made while the event loop is running.
This is somewhat explained in the Qt documentation of QCoreApplication but it is very easy to miss.
I imagine you don't really need to call exec_() as you aren't using any events in your current code (most events are to do with window/mouse/keyboard though you might conceivably use some in the future like those generated by QTimer). It really depends what you want to do with the program in the future.
If you don't call exec_(), then your script will exit as you would normally expect any Python script to do (the only blocking function in your code is the call to exec_(), remove that and nothing keeps it running.)
I'm using COM integration to drive MS Excel from Python 2.7. I noticed a peculiar thing: when I run the following bit of code:
import win32com.client
excelApp = win32com.client.dynamic.Dispatch('Excel.Application')
an EXCEL.EXE process appears on the processes list (which view using the Windows Task Manager or subprocess.Popen('tasklist')) as expected. I then do all the stuff I need to do no problem. However, when I close Excel:
excelApp.Application.Quit()
The process persists, even if I close the Python interpreter which started it (this kind of makes sense as Excel runs in a different process but just to be sure). The only way I've found to terminate this process is either by hand, using the Task Manager, or by calling:
subprocess.Popen("taskkill /F /im EXCEL.EXE",shell=True)
the forceful /F flag is necessary, otherwise the process doesn't terminate.
This isn't really a problem (I hope) but I wanted to ask whether this could cause issues when I first edit the documents "normally", then when calling Excel from python and then "normally" again? Potentially many (couple dozens) times in a row? What I'm worried about is creating conflicting versions of documents etc. Or should I just terminate the EXCEL.EXE process each time just to be safe?
Also I noticed that subprocess.Popen("taskkill") doesn't return any exceptions that I can catch and anylse (or am I doing something worng here?). I'm particularly interested in distinguishing between the "non-existent process" kill attempt and a failed attempt to terminate the process.
try closing any open books, telling the app to quit and delete any references to the app. I usually wrap my com objects in a class. This is what my quit method looks like.
def quit(self):
self.xlBook.Close(SaveChanges=0)
self.xlApp.Quit()
del self.xlApp
Are you calling Dispatch from the main thread? If not be sure to call
pythoncom.CoInitialize()
before Dispatch, and
pythoncom.CoUninitialize()
after Quit
I'm stuck at a tricky problem to test whether a daemon thread is running. The daemon thread I made should run at the background to keep the service running, so I do the following to create it and keep it alive:
Creation:
ASThread = threading.Thread(target = initAirserv, args=[],)
ASThread.setDaemon(True)
ASThread.start()
Inside the initAirserv() method:
def initAirserv(self, channel="15"):
interface = self.execAirmon(options="start", interface=self.interface)
port = self.plug_port
if interface != "removed":
if channel=="15":
command = "airserv-ng -d " +str(interface)+" -p "+str(port)
else:
command = "airserv-ng -d " +str(interface)+" -p "+str(port)+" -c"+str(channel)
else:
return None
AServConn=self.init_Plug()
if AServConn:
(stdin, stdout, stderr) = AServConn.exec_command(command)
serv_op = stdout
serv_er = stderr
##### keep the daemon thread run persistently ####
a = 0
while 1:
a += 1
else:
logging.debug( "SSH Error" )
The purpose of the last several lines is to keep the thread busy using a stupid way. However, after starting this daemon thread and I did something else, when I came back and check the thread as follows:
if ASThread.is_alive() == 1:
# do something
the if body never gets executed. Can someone explain to me why does this happen? What's the best way to run a thread which executes something that needs to be busy all the time? Thanks very much.
The posted code doesn't add up. initAirserv as posted is a method on a class, but the initAirserv passed to the Thread constructor is not.
It's also hard to say anything concrete without knowing what execAirmon and init_Plug does, and what else happens in your application.
In general, I'd say you have it right. This should work. The fact that it doesn't means your assumptions are wrong.
Are you sure execAirmon returns something that is not equal to "removed"?
Are you sure init_Plug returns a non-false object?
Are you sure no exceptions are thrown? (I assume you would notice a spurious stacktrace, so are there other parts of your application that could swallow them up unnoticed?)
Some of my information is a few months old, and things may have changed, so please bear with me.
If you are using standard C based Python and are writing a multi-threaded application, you need to be aware of the Global Interpreter Lock (GIL) restriction. That is only one thread can run at a time. If you are willing to use one of the Python C interface packages and write a lot of your code in C, the C portion of your function call can be threaded and is not subject to the GIL restriction.
Python has excellent multi-process support and libraries, and because you are synchronizing processes, the GIL restriction does not apply.
There is talk about fixing the GIL restriction, but for now this is a problem you have to accept.
IMHO, I chose Python to write software in Python, not in C, unless a very specific problem had to be addressed. Python is an excellent language for a lot of things, but the GIL restriction encouraged me to learn a language that would support better event synchronization, aka a multi-threaded environment.
I hope this helps.
I have a server script that I need to be able to shutdown cleanly. While testing the usual try..except statements I realized that Ctrl-C didn't work the usual way. Normally I'd wrap long running tasks like this
try:
...
except KeyboardInterrupt:
#close the script cleanly here
so the task could be shutdown cleanly on Ctrl-C. I have never ran into any problems with this before, but somehow when I hit Ctrl-C when this particular script is running the script just exits without catching the Ctrl-C.
The initial version was implemented using Process from multiprocessing. I rewrote the script using Thread from threading, but same issue there. I have used threading many times before, but I am new to the multiprocessing library. Either way, I have never experienced this Ctrl-C behavior before.
Normally I have always implemented sentinels etc to close down Queues and Thread instances in an orderly fashion, but this script just exits without any response.
Last, I tried overriding signal.SIGINT as well like this
def handler(signal, frame):
print 'Ctrl+C'
signal.signal(signal.SIGINT, handler)
...
Here Ctrl+C was actually caught, but the handler doesn't execute, it never prints anything.
Besides the threading / multiprocessing aspect, parts of the script contains C++ SWIG objects. I don't know if that has anything to do with it. I am running Python 2.7.2 on OS X Lion.
So, a few questions:
What's going on here?
How can I debug this?
What do I need to learn in order to understand the root cause?
PLEASE NOTE: The internals of the script is proprietary so I can't give code examples. I am however very willing to receive pointers so I could debug this myself. I am experienced enough to be able to figure it out if someone could point me in the right direction.
EDIT: I started commenting out imports etc to see what caused the weird behavior, and I narrowed it down to an import of a C++ SWIG library. Any ideas why importing a C++ SWIG library 'steals' Ctrl-C? I am not the author of the guilty library however and my SWIG experience is limited so don't really know where to start...
EDIT 2: I just tried the same script on a windows machine, and in Windows 7 the Ctrl-C is caught as expected. I'm not really going to bother with the OS X part, the script will be run in an Windows environment anyway.
This might have to do with the way Python manages threads, signals and C calls.
In short - Ctrl-C cannot interrupt C calls, since the implementation requires that a python thread will handle the signal, and not just any thread, but the main thread (often blocked, waiting for other threads).
In fact, long operations can block everything.
Consider this:
>>> nums = xrange(100000000)
>>> -1 in nums
False (after ~ 6.6 seconds)
>>>
Now, Try hitting Ctrl-C (uninterruptible!)
>>> nums = xrange(100000000)
>>> -1 in nums
^C^C^C (nothing happens, long pause)
...
KeyboardInterrupt
>>>
The reason Ctrl-C doesn't work with threaded programs is that the main thread is often blocked on an uninterruptible thread-join or lock (e.g, any 'wait', 'join' or just a plain empty 'main' thread, which in the background causes python to 'join' on any spawned threads).
Try to insert a simple
while True:
time.sleep(1)
in your main thread.
If you have a long running C function, do signal handling in C-level (May the Force be with you!).
This is largely based on David Beazley's video on the subject.
It exits because something else is likely catching the KeyboardInterupt and then raising some other exception, or simply returning None. You should still get a traceback to help debug. You need to capture the stderr output or run your script with the -i commandline option so you can see traceback. Also, add another except block to catch all other exceptions.
If you suspect the C++ function call to be catching the CTRL+C try catching it's output. If the C function is not returning anything then there isn't much you can do except ask the author to add some exception handling, return codes, etc.
try:
#Doing something proprietary ...
#catch the function call output
result = yourCFuncCall()
#raise an exception if it's not what you expected
if result is None:
raise ValueError('Unexpected Result')
except KeyboardInterupt:
print('Must be a CTRL+C')
return
except:
print('Unhandled Exception')
raise
how about atexit?
http://docs.python.org/library/atexit.html#module-atexit