We Keep Coding

Python thread class wont exit

I want to exit the crawler script which is using threading, but it doesn't detect Ctrl+c. I tried to try and except inside the main thread and in newly created threads, but it doesn't raise the exception, signal in main thread, but main thread isn't active, and it doesn't detect it.
import threading
import signal
class Crawler(threading.Thread):
def __init__(self, num):
threading.Thread.__init__(self)
# Other stuff
def run(self, ):
try:
self.crawl()
except KeyboardInterrupt:
self.exit()
def crawl(self):
for url in self.links_list:
try:
# Getting the website and all links on it and adding it to links_list
pass
except (KeyboardInterrupt, SystemExit):
self.exit()
except AssertionError:
self.exit()
def exit(self):
# Saving progress
raise SystemExit()
def handler(signum, frame):
exit()
if __name__ == "__main__":
signal.signal(signal.SIGINT, handler)
for i in range(8):
crawler = Crawler(i)
crawler.start()

Instead of trying to do stuff with signals, use a threading.Event() to signal all of the other threads to stop:
import threading
import time
stop_signal = threading.Event()
class Crawler(threading.Thread):
def __init__(self, num):
threading.Thread.__init__(self)
# Other stuff
def run(self):
try:
for url in self.links_list:
# Between every url, see if we should stop
if stop_signal.is_set():
break
# Getting the website and all links on it and adding it to links_list
finally:
self.save_progress()
def save_progress(self):
pass
def main():
crawlers = [Crawler(x) for x in range(8)]
for crawler in crawlers:
crawler.start()
# Do nothing for a whole long while in the main loop
# (you could exit the loop when all jobs are done, though)
while True:
try:
time.sleep(1)
except KeyboardInterrupt:
stop_signal.set()
break
for crawler in crawlers:
crawler.join()
if __name__ == "__main__":
main()

Properly terminate a process

I have a subclass of multiprocessing.Process and I want to terminate it correctly.
class MyProcess(Process):
def __init__(self, target, input_queue, output_queue, database):
super().__init__()
self.input_queue = input_queue
self.output_queue = output_queue
self.target = target
self.database = database
self.db_session = database.create_session()
# ...
def run(self):
signal.signal(signal.SIGINT, signal.SIG_IGN)
while True:
try:
# doing some stuff here
except Empty:
break
except Exception as err:
logger.error(str(err))
try:
self.db_session.commit()
except:
logger.error(str(err))
I want to close self.db_session (which is an SQLAlchemy Session) when the process is terminated. But as the Python documentation says "exit handlers and finally clauses, etc., will not be executed". How can I correctly terminate a process and close the things it uses?

I found a nice solution, by using multiprocessing.Event(). I added an Event object to the constructor of my MyProcess class, and the process loop looks like that now:
def run(self):
while True:
try:
if self.stop_event.is_set():
break # breaks the loop if a stop event is set
# doing some stuff here
except Empty:
break
except Exception:
logger.error(str(err))
try:
self.db_session.commit()
except:
logger.error(str(err))
Instead of calling terminate() when I need to terminate my processes, I just call stop_event.set(), which will break the loop and close everything gracefully.

downloading a file in python and cancel

I am trying to make a simple function to download file in python
The code is something like
def download(url , dest):
urllib.urlretrieve(url, dest)
My issue is that if I want to cancel the download process in the middle of downloading how do I approach???
This function runs in the background of app and is triggered by a button. Now I am trying to trigger it off with another button.
The platform is XBMC.

A simple class to do the same as your download function:
import urllib
import threading
class Downloader:
def __init__(self):
self.stop_down = False
self.thread = None
def download(self, url, destination):
self.thread = threading.Thread(target=self.__down, args=(url, destination))
self.thread.start()
def __down(self, url, dest):
_continue = True
handler = urllib.urlopen(url)
self.fp = open(dest, "w")
while not self.stop_down and _continue:
data = handler.read(4096)
self.fp.write(data)
_continue = data
handler.close()
self.fp.close()
def cancel(self):
self.stop_down = True
So, when someone clicks the "Cancel" button you have to call the cancel() method.
Please note that this will not remove the partially downloaded file if you cancel it, but that should not be hard to achieve using os.unlink(), for example.
The following example script shows how to use it, starting the download of a ~20Mb file and cancelling it after 5 seconds:
import time
if __name__ == "__main__":
url = "http://ftp.postgresql.org/pub/source/v9.2.3/postgresql-9.2.3.tar.gz"
down = Downloader()
down.download(url, "file")
print "Download started..."
time.sleep(5)
down.cancel()
print "Download canceled"

If you are canceling by pressing CTRL+C, then you can use this built in exception and proceed with what you think the best move should be.
In this case, if I cancel in the middle of a download, I simply want that partial file to be deleted:
def download(url , dest):
try:
urllib.urlretrieve(url, dest)
except KeyboardInterrupt:
if os.path.exists(dest):
os.remove(dest)
except Exception, e:
raise

simple thread management within python classes

Im trying to write a module for Python that prints out text for my program and displays a progress bar while i do something in the background. Im using the 'threading' module currently but open to suggestions if something else will make it easier.
what i want to know is two fold, how should i call this class elegantly and how should i stop these threads im creating?
this is what im currently doing:
tmp1 = textprint("hello this is text")
tmp1.start()
# do something
tmp1.stop()
these are the options ive considered and looked into so far:
using thread.name to find the name of the thread or having the thread
return a name to kill afterwards. OR passing a number for similar
thread identification afterwards. (a bit cumbersome and not my
favourite solution.)
sending a thread.event ? - from reading the docs i see an event can
be sent, perhaps that can be used to stop it?
or a with statement but im unclear how to use it in this context, plus i find most of the python docs extremely confusing and not written for me at all.
what i would like to do is something like this:
echo('hello') (prints progress bar etc)
- and then when i want to stop it echo.stop()
the obv. problem there though is that the stop function doesnt know which thread it is trying to stop.
Here is a skeleton of what im trying to do:
import time
import string
import threading
class print_text(threading.Thread):
def __init__(self,arg=None):
super(print_text,self).__init__()
self._stop = False
self.arg=arg
def run (self):
# start thread for text
print self.txt
while not self._stop:
print "rude words"
def echo (self,txt):
self.txt=txt
self.start()
def stop(self):
self._stop = True
def stopped(self):
return self._stop == True
def __enter__(self):
print "woo"
return thing
def __exit__(self, type, value, traceback):
return isinstance(value, TypeError)
if __name__ == '__main__':
print_text.start.echo('this is text') # dunt werk
with print_text.echo('this is text'):
time.sleep(3)
print "done"
and then call it like so:
echo('this is text')
i also guess to do this i would have to
import echo from print_text
the WITH way of doing things suggests putting an __enter__ and __exit__ bit in. i tried them and they didnt work and also, i didnt know what i was doing, really appreciate any help, thanks.

You were very close to having working code. There just needed to be a few minor fixups:
print_text is a class. It should be instantiated with print_text()
The start method returns an instance of print_text, you need to save that
in order to call stop and echo on it: t = print_text()
The enter method needs to return self instead of thing.
The exit method should either set _stop or call stop().
The echo method should return self so that it can be used with the with-statement.
Here is some working code that includes those minor edits:
import time
import string
import threading
class print_text(threading.Thread):
def __init__(self, arg=None):
super(print_text,self).__init__()
self._stop = False
self.arg=arg
def run (self):
# start thread for text
print self.txt
while not self._stop:
print "rude words"
def echo (self, txt):
self.txt=txt
self.start()
return self
def stop(self):
self._stop = True
def stopped(self):
return self._stop == True
def __enter__(self):
print "woo"
return self
def __exit__(self, type, value, traceback):
self._stop = True
return isinstance(value, TypeError)
if __name__ == '__main__':
t = print_text()
t.echo('this is text')
time.sleep(3)
t.stop()
with print_text().echo('this is text'):
time.sleep(3)
print "done"

The best way to stop a thread in Python is to politely ask it to stop. The best way to pass new data to a thread is with the Queue module.
Both are used in the code in this post, which demonstrates socket communication from a Python thread but is otherwise relevant to your question. If you read the code carefully you'll notice:
Using threading.Event() which is set by a method call from outside, and which the thread periodically checks to know if it was asked to die.
Using Queue.Queue() for both passing commands to the thread and receiving responses from it.

A thread name is useful if you could potentially have multiple subthreads running the same target at once and want to ensure that all of them are stopped. It seems like a useful generalization and doesn't seem too cumbersome to me, but beauty is in the eye of the beholder :-). The following:
starts a subthread to print a message and start a progressbar
stops the subthread using a name given when it was started.
It is much simpler code. Does it do what you want?
import time, threading
class print_text:
def __init__(self):
pass
def progress(self):
while not self._stop: # Update progress bar
print(".", sep="", end="")
time.sleep(.5)
def echo(self, arg="Default"): # Print message and start progress bar
print(arg)
self._stop = False
threading.Thread(target=self.progress, name="_prog_").start()
def stop(self):
self._stop = True
for t in threading.enumerate():
if t.name == "_prog_":
t.join()
tmp1 = print_text()
tmp1.echo("hello this is text")
time.sleep(10)
tmp1.stop()
print("Done")

wxpython event not triggering

I'm following the example given in http://www.blog.pythonlibrary.org/2010/05/22/wxpython-and-threads/
I have a thread which is checking an sftp server for new files every 30 seconds. If it finds files, it uploads them to a db, and then it should trigger an update of certain GUI elements which will reload from the db.
The custom event code:
EVT_RESULT_ID = wx.NewId()
def EVT_RESULT(win, func):
"""Define Result Event."""
win.Connect(-1, -1, EVT_RESULT_ID, func)
class ResultEvent(wx.PyEvent):
"""Simple event to carry arbitrary result data."""
def __init__(self, data):
"""Init Result Event."""
wx.PyEvent.__init__(self)
self.SetEventType(EVT_RESULT_ID)
self.data = data
The ftp thread:
class FTPThread(threading.Thread):
def __init__(self,wxObject):
"""Init Worker Thread Class."""
threading.Thread.__init__(self)
self.wxObject = wxObject
self._stop = threading.Event()
self._stop.set()
self.start() # start the thread
def run(self):
while True:
time.sleep(30)
if not self._stop.isSet():
wx.CallAfter(self.parseFTP)
def stop(self):
self._stop.set()
def resume(self):
self._stop.clear()
def parseFTP(self):
#connect to db
...
#connect to sftp site
...
files_found=False
#process each file and delete
for file in dirlist:
files_found=True
...#process into db
sftp.remove(file)
sftp.close()
t.close()
#trigger update event if files found
if files_found==True:
wx.PostEvent(self.wxObject, ResultEvent("Files found"))
One of the GUI elements:
class MyGrid(wx.grid.Grid):
def __init__(self, parent):
wx.grid.Grid.__init__(self, parent,-1,style=wx.EXPAND)
self.parent=parent
...
self.update()
EVT_RESULT(self, self.updateFromEvent)
def updateFromEvent(self,event):
self.update()
def update(self):
...
Following debugging, the wx.PostEvent is being created, but not triggering any response in the grid.
The only difference I can find between the example and my code is that in the example the EVT_RESULT is in the main Frame, and not a GUI element - is this required?

Events don't propagate to its children so if MyGrid is a child of your main frame, events posted in the main won't make it through to MyGrid. What you can do instead is bind the event handler directly to your function within the instance of MyGrid like so:
"""from MainWindow"""
self._workerthread = FtpThread(...)
self._mygrid = MyGrid(...)
# Bind event
EVT_RESULT(self, self._mygrid.updateFromEvent)
I'm not too familiar with this kind of binding as I typically use wx.Bind.

I'm not sure, but that example was based on something in the wiki: http://wiki.wxpython.org/LongRunningTasks
I suspect that since it says "win" as an argument, it is probably referring to a Top Level Window, so the wx.Frame is probably required. You can still update the grid from the frame though.
EDIT: Manny has a good point. That would probably work too. And pubsub rocks!