I'm stuck at a tricky problem to test whether a daemon thread is running. The daemon thread I made should run at the background to keep the service running, so I do the following to create it and keep it alive:
Creation:
ASThread = threading.Thread(target = initAirserv, args=[],)
ASThread.setDaemon(True)
ASThread.start()
Inside the initAirserv() method:
def initAirserv(self, channel="15"):
interface = self.execAirmon(options="start", interface=self.interface)
port = self.plug_port
if interface != "removed":
if channel=="15":
command = "airserv-ng -d " +str(interface)+" -p "+str(port)
else:
command = "airserv-ng -d " +str(interface)+" -p "+str(port)+" -c"+str(channel)
else:
return None
AServConn=self.init_Plug()
if AServConn:
(stdin, stdout, stderr) = AServConn.exec_command(command)
serv_op = stdout
serv_er = stderr
##### keep the daemon thread run persistently ####
a = 0
while 1:
a += 1
else:
logging.debug( "SSH Error" )
The purpose of the last several lines is to keep the thread busy using a stupid way. However, after starting this daemon thread and I did something else, when I came back and check the thread as follows:
if ASThread.is_alive() == 1:
# do something
the if body never gets executed. Can someone explain to me why does this happen? What's the best way to run a thread which executes something that needs to be busy all the time? Thanks very much.
The posted code doesn't add up. initAirserv as posted is a method on a class, but the initAirserv passed to the Thread constructor is not.
It's also hard to say anything concrete without knowing what execAirmon and init_Plug does, and what else happens in your application.
In general, I'd say you have it right. This should work. The fact that it doesn't means your assumptions are wrong.
Are you sure execAirmon returns something that is not equal to "removed"?
Are you sure init_Plug returns a non-false object?
Are you sure no exceptions are thrown? (I assume you would notice a spurious stacktrace, so are there other parts of your application that could swallow them up unnoticed?)
Some of my information is a few months old, and things may have changed, so please bear with me.
If you are using standard C based Python and are writing a multi-threaded application, you need to be aware of the Global Interpreter Lock (GIL) restriction. That is only one thread can run at a time. If you are willing to use one of the Python C interface packages and write a lot of your code in C, the C portion of your function call can be threaded and is not subject to the GIL restriction.
Python has excellent multi-process support and libraries, and because you are synchronizing processes, the GIL restriction does not apply.
There is talk about fixing the GIL restriction, but for now this is a problem you have to accept.
IMHO, I chose Python to write software in Python, not in C, unless a very specific problem had to be addressed. Python is an excellent language for a lot of things, but the GIL restriction encouraged me to learn a language that would support better event synchronization, aka a multi-threaded environment.
I hope this helps.
Related
I am new to python and having some problems.
I wrote an update_manager class that can communicate with the user via Tcp and preform installations of different components.
My update_manager class uses 2 other classes(they are his members) to accomplish this. The first is used for TCP communication and the second for actual installation. the installation class runs from the main thread and the communication is run by using Threading.thread() function.
my main locks like this:
if __name__ == "__main__":
new_update = UpdateManager()
#time.sleep(10)
new_update.run()
and the run functions is:
def run(self):
comm_thread = threading.Thread(target=
self._comm_agent.start_server_tcp_comunication)
comm_thread.start()
while True:
if (False == self.is_recovery_required()):
self.calculate_free_storage_for_update_zip_extraction()
self.start_communication_with_client_in_state_machine()
self._comm_agent.disable_synchronized_communication()
self.start_update_install()
self._comm_agent.enable_synchronized_communication()
if (True == self.is_dry_run_requested()):
self.preform_cleanup_after_dry_run()
else:
self.reset_valid_states()
self.preform_clean_up_after_update_cycle()
I use 2 multiprocessing.Queue() to sync between the threads and between the user. One for incoming messages and one for outgoing messages.
At first TCP communication is synchronous, user provides installation file and few other things.
Once installation begins TCP communication is no longer synchronous.
During the installation I use 4 different install methods. and all but one work just fine with no problem(user can pool the update_manager process and ask progress questions and get immediate a reply)
The problematic one is the instantiation of rpm files. for this I tried calling for os.system() and subprocess.run() and it works but for big rpm files I notices the entire process with my threads freezes until
the call finishes(I can see the progress bar of rpm installation on my screen during this freeze).
What I noticed and tried:
1.There is no freeze during other installation methods which use python.
libraries.
2.Once user connects via TCP there are only 2 threads for the update_manager, once first request is sent and a reply is send back 2 more threads appear (I assume it have something to do with the queues I use).
3.I created third thread that prints time(and has nothing to do with the queues), and I start it as soon as update_manager process starts. When the 2 threads freeze this one keeps going.
4.On some rare occasions process will unfreeze just for a message to go throw from client to update_manager and freeze back.
Edit: I forgot one more important point
5. The freeze occurs when calling:
os.system("rpm --nodeps --force -ivh rpm_file_name")
But does not happen when calling:
os.system("sleep 5")
I would really appreciate some indigent, Thanks.
The problem was with the incoming queue.
I used:
if (True == self._outgoing_message_queue.empty()):
temp: dict = self._outgoing_message_queue.get()
This is a simple bug, thread just got stuck on an empty queue.
But even If the code is changed to
if (False == self._outgoing_message_queue.empty()):
temp: dict = self._outgoing_message_queue.get()
it might cause the same behavior because between the moment the if statement is evaluated and the moment the get() is called a contact switch might occur and the queue might become empty and thread will get stuck on .get() as in my original code.
A better solution is to use get_nowait()
try:
temp = self._outgoing_message_queue.get_nowait()
except:
temp = None
After reading A LOT of data on the subject I still couldn't find any actual solution to my problem (there might not be any).
My problem is as following:
In my project I have multiple drivers working with various hardware's (IO managers, programmable loads, power supplies and more).
Initializing connection to these hardware's is costly (in time), and I cant open and then close the connection for every communication iteration between us.
Meaning I cant do this (Assuming programmable load implements enter / exit):
start of code...
with programmable_load(args) as program_instance:
programmable_load_instance.do_something()
rest of code...
So I went for a different solution :
class programmable_load():
def __init__(self):
self.handler = handler_creator()
def close_connection(self):
self.handler.close_connection()
self.handler = None
def __del__(self):
if (self.handler != None):
self.close_connection()
For obvious reasons I dont 'trust' the destructor to actually get called so I explicitly call close_connection() when I want to end my program (for all drivers).
The problem happens when I abruptly terminate the process, for example when I run via debug mode and quit debugging.
In these cases the process terminates without running through any destructors.
I understand that the OS will clear all memory unused at this point, but is there any way to clear the memory in an organized manner?
and if not, is there a way to make the quit debugging function pass through a certain set of functions? Does the python process know it got a quite debugging event or does it treat it as a normal termination?
Operating system: Windows
According to this documentation:
If a process is terminated by TerminateProcess, all threads of the
process are terminated immediately with no chance to run additional
code.
(Emphasis mine.) This implies that there is nothing you can do in this case.
As detailed here, signals don't work very well on ms-windows.
As was mentioned in a comment, you could use atexit to do the cleanup. But that only works if the process is asked to close (e.g. QUIT signal on Linux) and not just killed (as is likely the case when stopping the debugging session). Similarily if you force your computer to turn off (e.g. long press power button or remove power) then it won't be called either. There is no 'solution' to that for obvious reasons. Your program can't expect to be called when the power suddenly goes off or when it is forcefully killed. The point of forcefully killing is to definitely kill the process now. If it first called your clean-up code then you could delay that which defeats the purpose. That is why there are signals such as to ask your process to stop. This is not Python specific. The same concept also applies across operating systems.
Bonus (design suggestion, not a solution): I would argue that you can still make use of the context manager (using with). Your problem is not unique. Database connections are usually kept alive for longer as well. It is a question of the scope. Move the context further up to the application level. Then it is clear what the boundary is and you don't need any magic (you are probably also aware of #contextmanager to make that a breeze).
I haven't tested properly as I don't have wingide installed over here so I can't grant you this will work but what about using setconsolectrlhandler? For instance, try something like this:
import os
import sys
import win32api
if __name__ == "__main__":
def callback(sig, func=None):
print("Exit handler called!")
try:
win32api.SetConsoleCtrlHandler(callback, True)
except Exception as e:
print("Captured exception", e)
sys.exit(1)
print("Press to quit")
input()
print("Bye!")
It'll be able to handle CTRL+C and CTRL+BREAK signals:
I currently have a python script that does exactly what I need it to do, however every now and then the script will hang and the only way to restart it is by killing the script and relaunching it.
I was wondering if there was a way to put in a few commands that will restart it lets say everytime it hangs or when a specific message appears or even just restart it on a timer eg:every 50 seconds.
I cannot provide the code through here, but I can provide it if we talk in private.
I am willing to pay you a bit of money if your fix does work.
please email me at stackoverflow1#shaw.ca
Thanks!
Edit: I see, ok - then is it possible to provide me with some codes which it will restart on a specific timer?
Edit2: Ok thanks everyone for their comments - I will get in touch with the person who built it to see if they can rewrite it from scratch to include a timer.
Cheers.
Feel free to pay me if you want, although it is by no means necessary.
Here:
import time
import threading
import os
def restart():
time.sleep(50)
os.execv('/full/path/to/this/script', ['second argument', 'third argument'])
def main():
t = threading.Thread(target=restart, args=(), name='reset')
t.start()
# ... The rest of your code.
If you have any buffers open that you care about (such as stdout) you'll want to flush them right before the call to execv up there.
I haven't tested this code, because I don't have a python interpreter handy at the moment, but I'd be surprised if it didn't work. That call to execv replaces the current context, so you don't get an increasingly deep hierarchy of child processes. All I'm doing, in case you're curious and want to know what magic phrase to google, is setting a "timer interrupt handler". For the pedants, no, I recognize this thing isn't directly handling any interrupts.
The numeric argument to sleep is in seconds. I would simply request that you not use my code in malware, unless it is for research purposes. I'm particular that way.
edit: Additionally, a lot of it was taken from here.
I'm using COM integration to drive MS Excel from Python 2.7. I noticed a peculiar thing: when I run the following bit of code:
import win32com.client
excelApp = win32com.client.dynamic.Dispatch('Excel.Application')
an EXCEL.EXE process appears on the processes list (which view using the Windows Task Manager or subprocess.Popen('tasklist')) as expected. I then do all the stuff I need to do no problem. However, when I close Excel:
excelApp.Application.Quit()
The process persists, even if I close the Python interpreter which started it (this kind of makes sense as Excel runs in a different process but just to be sure). The only way I've found to terminate this process is either by hand, using the Task Manager, or by calling:
subprocess.Popen("taskkill /F /im EXCEL.EXE",shell=True)
the forceful /F flag is necessary, otherwise the process doesn't terminate.
This isn't really a problem (I hope) but I wanted to ask whether this could cause issues when I first edit the documents "normally", then when calling Excel from python and then "normally" again? Potentially many (couple dozens) times in a row? What I'm worried about is creating conflicting versions of documents etc. Or should I just terminate the EXCEL.EXE process each time just to be safe?
Also I noticed that subprocess.Popen("taskkill") doesn't return any exceptions that I can catch and anylse (or am I doing something worng here?). I'm particularly interested in distinguishing between the "non-existent process" kill attempt and a failed attempt to terminate the process.
try closing any open books, telling the app to quit and delete any references to the app. I usually wrap my com objects in a class. This is what my quit method looks like.
def quit(self):
self.xlBook.Close(SaveChanges=0)
self.xlApp.Quit()
del self.xlApp
Are you calling Dispatch from the main thread? If not be sure to call
pythoncom.CoInitialize()
before Dispatch, and
pythoncom.CoUninitialize()
after Quit
I have a server script that I need to be able to shutdown cleanly. While testing the usual try..except statements I realized that Ctrl-C didn't work the usual way. Normally I'd wrap long running tasks like this
try:
...
except KeyboardInterrupt:
#close the script cleanly here
so the task could be shutdown cleanly on Ctrl-C. I have never ran into any problems with this before, but somehow when I hit Ctrl-C when this particular script is running the script just exits without catching the Ctrl-C.
The initial version was implemented using Process from multiprocessing. I rewrote the script using Thread from threading, but same issue there. I have used threading many times before, but I am new to the multiprocessing library. Either way, I have never experienced this Ctrl-C behavior before.
Normally I have always implemented sentinels etc to close down Queues and Thread instances in an orderly fashion, but this script just exits without any response.
Last, I tried overriding signal.SIGINT as well like this
def handler(signal, frame):
print 'Ctrl+C'
signal.signal(signal.SIGINT, handler)
...
Here Ctrl+C was actually caught, but the handler doesn't execute, it never prints anything.
Besides the threading / multiprocessing aspect, parts of the script contains C++ SWIG objects. I don't know if that has anything to do with it. I am running Python 2.7.2 on OS X Lion.
So, a few questions:
What's going on here?
How can I debug this?
What do I need to learn in order to understand the root cause?
PLEASE NOTE: The internals of the script is proprietary so I can't give code examples. I am however very willing to receive pointers so I could debug this myself. I am experienced enough to be able to figure it out if someone could point me in the right direction.
EDIT: I started commenting out imports etc to see what caused the weird behavior, and I narrowed it down to an import of a C++ SWIG library. Any ideas why importing a C++ SWIG library 'steals' Ctrl-C? I am not the author of the guilty library however and my SWIG experience is limited so don't really know where to start...
EDIT 2: I just tried the same script on a windows machine, and in Windows 7 the Ctrl-C is caught as expected. I'm not really going to bother with the OS X part, the script will be run in an Windows environment anyway.
This might have to do with the way Python manages threads, signals and C calls.
In short - Ctrl-C cannot interrupt C calls, since the implementation requires that a python thread will handle the signal, and not just any thread, but the main thread (often blocked, waiting for other threads).
In fact, long operations can block everything.
Consider this:
>>> nums = xrange(100000000)
>>> -1 in nums
False (after ~ 6.6 seconds)
>>>
Now, Try hitting Ctrl-C (uninterruptible!)
>>> nums = xrange(100000000)
>>> -1 in nums
^C^C^C (nothing happens, long pause)
...
KeyboardInterrupt
>>>
The reason Ctrl-C doesn't work with threaded programs is that the main thread is often blocked on an uninterruptible thread-join or lock (e.g, any 'wait', 'join' or just a plain empty 'main' thread, which in the background causes python to 'join' on any spawned threads).
Try to insert a simple
while True:
time.sleep(1)
in your main thread.
If you have a long running C function, do signal handling in C-level (May the Force be with you!).
This is largely based on David Beazley's video on the subject.
It exits because something else is likely catching the KeyboardInterupt and then raising some other exception, or simply returning None. You should still get a traceback to help debug. You need to capture the stderr output or run your script with the -i commandline option so you can see traceback. Also, add another except block to catch all other exceptions.
If you suspect the C++ function call to be catching the CTRL+C try catching it's output. If the C function is not returning anything then there isn't much you can do except ask the author to add some exception handling, return codes, etc.
try:
#Doing something proprietary ...
#catch the function call output
result = yourCFuncCall()
#raise an exception if it's not what you expected
if result is None:
raise ValueError('Unexpected Result')
except KeyboardInterupt:
print('Must be a CTRL+C')
return
except:
print('Unhandled Exception')
raise
how about atexit?
http://docs.python.org/library/atexit.html#module-atexit