Can someone tell me where i can put the lock inside a custom thread in python?
import threading
lock = threading.Lock()
class WorkerThread(threading.Thread):
def __init__(self,lock):
super(WorkerThread,self).__init__()
self.lock = lock
def run(self):
self.lock.acquire()
print "Hello World"
self.lock.release()
worker = WorkerThread(lock)
Error Traceback:
Traceback (most recent call last):
File "<buffer>", line 14, in <module>
File "<buffer>", line 11, in __init__
AssertionError: release() of un-acquire()d lock
You've mixed tabs and spaces. Most of the definition of run is actually nested inside __init__, and the self.lock.release() is actually outside run and inside __init__. Your thread ends up trying to release the unlocked lock on thread creation.
Don't mix tabs and spaces. Stick to spaces. Turn on "show whitespace" in your editor to make the problem more visible, and get a better editor if your editor can't do that. Running Python with the -tt flag can also help catch these errors.
Related
When my program hits an assertion failure, I don't want the assertion library to do things that cause the program to progress further than it would have without the assertion failure. But that is exactly what the built-in assert seems to do: It raises an exception, which releases locks.
For example, consider the following program.
import threading
import time
lock = threading.Lock()
class Thread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.start()
def run(self):
with lock:
time.sleep(0.2)
print "ASSERTION ON NEXT LINE"
assert False
# Were it not for the assert,
# This thread would hold the lock for a while longer.
time.sleep(1.0)
Thread()
time.sleep(0.1)
with lock:
print "*** Main Thread ***"
The assertion causes the lock to be released, which causes the main thread to acquire the lock in between the assertion failure and the traceback. As a result, the output is:
ASSERTION ON NEXT LINE
*** Main Thread ***
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "so.py", line 14, in run
assert False
AssertionError
which is very undesirable because *** Main Thread *** is printed before the traceback. This can easily mislead one into thinking that the *** Main Thread *** acquired the lock before the exception was generated.
I know we could write our own assertion function that calls os._exit() instead of raising an assertion. What I'm wondering is if there is something like that already in the standard packages.
If you don't want to release the lock you have to catch the exception inside the with lock::
with lock:
try:
time.sleep(0.2)
print "ASSERTION ON NEXT LINE"
assert False
except Exception:
print "there was an exception"
while True:
pass
if we use assert and that fails its gonna raise an assertion error. Because asserts are designed to do this only ( break/stop the flow). I could think of synchronise this with Queue.
import time
import threading
from Queue import Queue
q = Queue()
# code here
class MyThread(threading.Thread):
def run(self):
# do stuff
print "ASSERTION ON NEXT LINE"
q.put(True)
# do other stuff
# code here
mt = MyThread()
mt.start()
try:
assert q.get()
finally:
mt.join()
Thank you everyone for your contributions. I am going to mostly build on Daniel's answer. I believe that the best answer depends on how much code is inside the lock. If you have a big function call, then
with lock:
try:
deep_complicated_function()
except Exception:
print "there was an exception"
print '\n'.join(traceback.format_stack()[:-1])
os._exit(1)
while True:
pass
is best because it gives you a chance to provide a handler for a large number of exception cases in one construct.
However, this is too much typing for one-off assertions like:
with lock:
quick_update()
assert property()
Those cases call for something more convenient, like:
def Assert(cond, msg = "Assertion failed."):
if not cond:
print msg
print '\n'.join(traceback.format_stack()[:-1])
os._exit(1)
This will stop the process, which simplifies debugging (as discussed in the question). To use in nose and other testing tools, launch your main program through a multiprocessing API and generate an exception when the exit code is not 0.
Here's some slimmed down code that demonstrates my use of threading:
import threading
import Queue
import time
def example():
""" used in MainThread as the example generator """
while True:
yield 'asd'
class ThreadSpace:
""" A namespace to be shared among threads/functions """
# set this to True to kill the threads
exit_flag = False
class MainThread(threading.Thread):
def __init__(self, output):
super(MainThread, self).__init__()
self.output = output
def run(self):
# this is a generator that contains a While True
for blah in example():
self.output.put(blah)
if ThreadSpace.exit_flag:
break
time.sleep(0.1)
class LoggerThread(threading.Thread):
def __init__(self, output):
super(LoggerThread, self).__init__()
self.output = output
def run(self):
while True:
data = self.output.get()
print data
def main():
# start the logging thread
logging_queue = Queue.Queue()
logging_thread = LoggerThread(logging_queue)
logging_thread.daemon = True
logging_thread.start()
# launch the main thread
main_thread = MainThread(logging_queue)
main_thread.start()
try:
while main_thread.isAlive():
time.sleep(0.5)
except KeyboardInterrupt:
ThreadSpace.exit_flag = True
if __name__ == '__main__':
main()
I have one main thread which gets data yielded to it from a blocking generator. In the real code, this generator yields network related data it sniffs out over the socket.
I then have a logging, daemon, thread which prints the data to screen.
To exit the program cleanly, I'm catching a KeyboardInterrupt which will set an exit_flag to try - This tells the main thread to return.
9 times out of 10, this will work fine. The program will exit cleanly. However, there's a few occasions when I'll receive the following two errors:
Error 1:
^CTraceback (most recent call last):
File "demo.py", line 92, in <module>
main('')
File "demo.py", line 87, in main
time.sleep(0.5)
KeyboardInterrupt
Error 2:
Exception KeyboardInterrupt in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
I've run this exact sample code a few times and haven't been able to replicate the errors. The only difference between this and the real code is the example() generator. This, like I said, yields network data from the socket.
Can you see anything wrong with how I'm handling the threads?
KeyboardInterrupts are received by arbitrary threads. If the receiver isn't the main thread, it dies, the main thread is unaffected, ThreadSpace.exit_flag remains false, and the script keeps running.
If you want sigint to work, you can have each thread catch KeyboardInterrupt and call thread.interrupt_main() to get Python to exit, or use the signal module as the official documentation explains.
HI, guys,
I am developing a GUI with python 2.4.3 and wxpython. Everything works fine except when I exit main program(close the main window of GUI). The wierd thing is that sometimes there is such error, sometimes there is no error at all. Although I found the same error reports from the python mailing list(the link is http://bugs.python.org/issue1722344, I am not sure if my case is the same as this one). I do not know how it is finally solved and what I should do to overcome this problem.
The error message from the console is as follows.
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
File "/opt/company/workspace/username/application/src/mainwidget.py", line 1066, in run
File "/usr/lib/python2.4/Queue.py", line 89, in put
File "/usr/lib/python2.4/threading.py", line 237, in notify
exceptions.TypeError: exceptions must be classes, instances, or strings (deprecated), not NoneType
Unhandled exception in thread started by
Error in sys.excepthook:
Original exception was:
The following is part of my code(the thread related code is complete, I extract the main operations for the rest). when I use the GUI to launch an external subprocess, at the same time, a wx.TextCtrl object is created. This wx.TextCtrl object is used to give input and print output of the external subprocess
class BashProcessThread(threading.Thread):
def __init__(self, readlineFunc):
threading.Thread.__init__(self)
self.readlineFunc = readlineFunc
self.lines = []
self.outputQueue = Queue.Queue()
self.setDaemon(True)
def run(self):
while True:
line = self.readlineFunc()
self.outputQueue.put(line)
if (line==""):
break
return ''.join(self.lines)
def getOutput(self):
""" called from other thread """
while True:
try:
line = self.outputQueue.get_nowait()
lines.append(line)
except Queue.Empty:
break
return ''.join(self.lines)
class ExternalProcWindow(wx.Window):
def __init__(self, parent, externapp):
wx.Window.__init__(self, parent, -1, pos=wx.DefaultPosition, size = wx.Size(1200, 120))
self.externapp=externapp
self.prompt = externapp.name.lower() + '>>'
self.textctrl = wx.TextCtrl(self, -1, '', size= wx.Size(1200, 120), style=wx.TE_PROCESS_ENTER|wx.TE_MULTILINE)
self.default_txt = self.textctrl.GetDefaultStyle()
self.textctrl.AppendText(self.prompt)
self.outputThread = BashProcessThread(self.externapp.sub_process.stdout.readline)
self.outputThread.start()
self.textctrl.SetFocus()
self.__bind_events()
self.Fit()
def __bind_events(self):
self.Bind(wx.EVT_TEXT_ENTER, self.__enter)
def __enter(self, e):
nl=self.textctrl.GetNumberOfLines()
ln = self.textctrl.GetLineText(nl-1)
ln = ln[len(self.prompt):]
self.externapp.sub_process.stdin.write(ln+"\n")
time.sleep(.3)
self.textctrl.AppendText(self.outputThread.getOutput())
class ExternApp:
def launch(self):
self.sub_process = subprocess.Popen(launchcmd, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
The problem is caused by the use of threading.Thread.setDaemon. Threads set daemonic don't prevent the Python intepreter from exiting, but they still keep running. Because Python cleans up the environment before the process is terminated, the threads can run into trouble when stuff is removed from under them. That raises an exception, which the thread class tries to print for your convenience -- but that, then, too fails because the process is exiting.
You could try to silence the exception, but that's tricky (and if the thread does anything substantial, it might hide a real problem. Not the case here, though.) Or you could ask the thread to stop before exiting, and not set the thread daemonic. Or you can simply avoid using threads altogether. I do not remember if wxPython has a convenient mechanism for getting a process's output or even of doing asynchronous I/O, but many GUI toolkits do. And there's always Twisted, which does it all for you.
How can you have a function or something that will be executed before your program quits? I have a script that will be constantly running in the background, and I need it to save some data to a file before it exits. Is there a standard way of doing this?
Check out the atexit module:
http://docs.python.org/library/atexit.html
For example, if I wanted to print a message when my application was terminating:
import atexit
def exit_handler():
print 'My application is ending!'
atexit.register(exit_handler)
Just be aware that this works great for normal termination of the script, but it won't get called in all cases (e.g. fatal internal errors).
If you want something to always run, even on errors, use try: finally: like this -
def main():
try:
execute_app()
finally:
handle_cleanup()
if __name__=='__main__':
main()
If you want to also handle exceptions you can insert an except: before the finally:
If you stop the script by raising a KeyboardInterrupt (e.g. by pressing Ctrl-C), you can catch that just as a standard exception. You can also catch SystemExit in the same way.
try:
...
except KeyboardInterrupt:
# clean up
raise
I mention this just so that you know about it; the 'right' way to do this is the atexit module mentioned above.
If you have class objects, that exists during the whole lifetime of the program, you can also execute commands from the classes with the __del__(self) method:
class x:
def __init__(self):
while True:
print ("running")
sleep(1)
def __del__(self):
print("destructuring")
a = x()
this works on normal program end as well if the execution is aborted, for sure there will be some exceptions:
running
running
running
running
running
Traceback (most recent call last):
File "x.py", line 14, in <module>
a = x()
File "x.py", line 8, in __init__
sleep(1)
KeyboardInterrupt
destructuring
This is a version adapted from other answers.
It should work (not fully tested) with graceful exits, kills, and PyCharm stop button (the last one I can confirm).
import signal
import atexit
def handle_exit(*args):
try:
... do computation ...
except BaseException as exception:
... handle the exception ...
atexit.register(handle_exit)
signal.signal(signal.SIGTERM, handle_exit)
signal.signal(signal.SIGINT, handle_exit)
I'm trying to create a COM Object from a dll in a new thread in Python - so I can run a message pump in that thread:
from comtypes.client import CreateObject
import threading
class MessageThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.daemon = True
def run(self):
print "Thread starting"
connection = CreateObject("IDMessaging.IDMMFileConnection")
print "connection created"
a = CreateObject("IDMessaging.IDMMFileConnection")
print "aConnection created"
t = MessageThread()
t.start()
this is the error trace I get:
aConnection created
Thread starting
>>> Exception in thread Thread-1:
Traceback (most recent call last):
File "c:\python26\lib\threading.py", line 532, in __bootstrap_inner
self.run()
File "fred.py", line 99, in run
self.connection = CreateObject("IDMessaging.IDMMFileConnection")
File "c:\python26\lib\site-packages\comtypes\client\__init__.py", line 235, in CreateObject
obj = comtypes.CoCreateInstance(clsid, clsctx=clsctx, interface=interface)
File "c:\python26\lib\site-packages\comtypes\__init__.py", line 1145, in CoCreateInstance
_ole32.CoCreateInstance(byref(clsid), punkouter, clsctx, byref(iid), byref(p))
File "_ctypes/callproc.c", line 925, in GetResult
WindowsError: [Error -2147221008] CoInitialize has not been called
Any ideas?
You need to have called CoInitialize() (or CoInitializeEx()) on a thread before you can create COM objects on that thread.
from win32com.client.pythoncom import CoInitialize
CoInitialize()
As far as I remember (long time ago I'e programmed a lot with COM Components) you have to call CoInitialize on each thread if your COM Object uses STA.
http://msdn.microsoft.com/en-us/library/ms678543(VS.85).aspx
But I've no idea how to call that function in python.
Here is the MSDN Doc
http://msdn.microsoft.com/en-us/library/ms678543(VS.85).aspx
Just to update with current experience using PyCharm and Python 2.7:
You need to import:
from pythoncom import CoInitializeEx
from pythoncom import CoUninitialize
then for running the thread:
def run(self):
res = CoInitializeEx(0)
#<your code>
CoUninitialize()
PyCharm get confused with STA apartment, you need to enable true multithreading.
It is important that each CoInitialize() is terminated with a CoUninitialize(), so be sure your code follows this rule in case of errors, too.
As another answer has said you need to run
CoInitialize()
However it is possible that the COMObject cannot just be passed to the threads directly. You will have to use CoMarshalInterThreadInterfaceInStream() and CoGetInterfaceAndReleaseStream() to pass instance between threads
https://stackoverflow.com/a/27966218/18052428