We Keep Coding

how to terminate a thread from within another thread [duplicate]

How can I start and stop a thread with my poor thread class?
It is in loop, and I want to restart it again at the beginning of the code. How can I do start-stop-restart-stop-restart?
My class:
import threading
class Concur(threading.Thread):
def __init__(self):
self.stopped = False
threading.Thread.__init__(self)
def run(self):
i = 0
while not self.stopped:
time.sleep(1)
i = i + 1
In the main code, I want:
inst = Concur()
while conditon:
inst.start()
# After some operation
inst.stop()
# Some other operation

You can't actually stop and then restart a thread since you can't call its start() method again after its run() method has terminated. However you can make one pause and then later resume its execution by using a threading.Condition variable to avoid concurrency problems when checking or changing its running state.
threading.Condition objects have an associated threading.Lock object and methods to wait for it to be released and will notify any waiting threads when that occurs. Here's an example derived from the code in your question which shows this being done. In the example code I've made the Condition variable a part of Thread subclass instances to better encapsulate the implementation and avoid needing to introduce additional global variables:
from __future__ import print_function
import threading
import time
class Concur(threading.Thread):
def __init__(self):
super(Concur, self).__init__()
self.iterations = 0
self.daemon = True # Allow main to exit even if still running.
self.paused = True # Start out paused.
self.state = threading.Condition()
def run(self):
self.resume()
while True:
with self.state:
if self.paused:
self.state.wait() # Block execution until notified.
# Do stuff...
time.sleep(.1)
self.iterations += 1
def pause(self):
with self.state:
self.paused = True # Block self.
def resume(self):
with self.state:
self.paused = False
self.state.notify() # Unblock self if waiting.
class Stopwatch(object):
""" Simple class to measure elapsed times. """
def start(self):
""" Establish reference point for elapsed time measurements. """
self.start_time = time.time()
return self
#property
def elapsed_time(self):
""" Seconds since started. """
try:
return time.time() - self.start_time
except AttributeError: # Wasn't explicitly started.
self.start_time = time.time()
return 0
MAX_RUN_TIME = 5 # Seconds.
concur = Concur()
stopwatch = Stopwatch()
print('Running for {} seconds...'.format(MAX_RUN_TIME))
concur.start()
while stopwatch.elapsed_time < MAX_RUN_TIME:
concur.resume()
# Can also do other concurrent operations here...
concur.pause()
# Do some other stuff...
# Show Concur thread executed.
print('concur.iterations: {}'.format(concur.iterations))

This is David Heffernan's idea fleshed-out. The example below runs for 1 second, then stops for 1 second, then runs for 1 second, and so on.
import time
import threading
import datetime as DT
import logging
logger = logging.getLogger(__name__)
def worker(cond):
i = 0
while True:
with cond:
cond.wait()
logger.info(i)
time.sleep(0.01)
i += 1
logging.basicConfig(level=logging.DEBUG,
format='[%(asctime)s %(threadName)s] %(message)s',
datefmt='%H:%M:%S')
cond = threading.Condition()
t = threading.Thread(target=worker, args=(cond, ))
t.daemon = True
t.start()
start = DT.datetime.now()
while True:
now = DT.datetime.now()
if (now-start).total_seconds() > 60: break
if now.second % 2:
with cond:
cond.notify()

The implementation of stop() would look like this:
def stop(self):
self.stopped = True
If you want to restart, then you can just create a new instance and start that.
while conditon:
inst = Concur()
inst.start()
#after some operation
inst.stop()
#some other operation
The documentation for Thread makes it clear that the start() method can only be called once for each instance of the class.
If you want to pause and resume a thread, then you'll need to use a condition variable.

Python threaded class calls another threaded class (queue help)

I am trying to control a 3-axis printer using an x-box controller. To get inputs from the x-box I have borrowed code from martinohanlon https://github.com/martinohanlon/XboxController/blob/master/XboxController.py
I have also created code that reads a text file line by line (G-code) to move the printer.
I would like to be able to use the X-Box controller to select a G-code file and run it, then as the printer is running continue to listen for a cancel button just in case the print goes wrong. The controller is a threaded class, and my readGcode is a threaded class.
The problem I'm having is that when I use the controller to start the readGcode class I cant communicate with the controller until that thread finished.
My temporary solution is to use the controller to select a file then pass that files path to the readGcode class. In the readGcode class it keeps trying to open a file using a try block and fails until the filepath is acceptable. Then it changes a bool which makes it skip further reading until its done.
Code:
import V2_Controller as Controller
import V2_ReadFile as Read
import time
import sys
# file daialogue
import tkinter as tk
from tkinter import filedialog
# when X is selected on the x-box controller
def X(xValue):
if not bool(xValue):
try:
f=selectfile()
rf.setfilename(f)
except:
print("failed to select file")
def selectfile():
try:
root = tk.Tk() # opens tkinter
root.withdraw() # closes the tkinter window
return filedialog.askopenfilename()
except Exception:
print("no file")
# setup xbox controller
xboxCont = Controller.XboxController(controlCallBack, deadzone=30,
scale=100, invertYAxis=True)
# init the readfile class
rf = Read.Readfile()
# set the custom function for pressing X
xboxCont.setupControlCallback(xboxCont.XboxControls.X, X)
try:
# start the controller and readfile threads
xboxCont.start()
rf.start()
xboxCont.join()
rf.join()
while True:
time.sleep(1)
# Ctrl C
except KeyboardInterrupt:
print("User cancelled")
# error
except:
print("Unexpected error:", sys.exc_info()[0])
raise
finally:
# stop the controller
xboxCont.stop()
rf.stop()
V2_Readfile
# Main class for reading the script
class Readfile(threading.Thread):
# supports all variables needed to read a script
class readfile:
fileselect = True
linecount = 0
currentline = 0
commands = []
# setup readfile class
def __init__(self):
# setup threading
threading.Thread.__init__(self)
# persist values
self.running = False
self.reading = False
def setfilename(self,filename):
self.filename = filename
# called by the thread
def run(self):
self._start()
# start reading
def _start(self):
self.running = True
while self.running:
time.sleep(1)
if not self.reading:
try:
self.startread()
except:
pass
def startread(self):
try:
with open(self.filename, "r") as f: # read a local file
f1 = f.readlines()
# run through each line and extract the command from each line
linecount = 0
line = []
for x in f1:
# read each line into an array
line.append(x.split(";")[0])
linecount += 1
# Store the variables for later use
self.readfile.linecount = linecount
self.readfile.commands = line
self.reading = True
except Exception:
pass
i = 0
while i < self.readfile.linecount and self.reading:
self.readfile.currentline = i + 1
self.readline(i)
i += 1
# the following stops the code from reading again
self.reading = False
self.filename = ""
def readline(self,line):
Sort.sortline(self.readfile.commands[line])
# stops the controller
def stop(self):
self.running = False

You could use a syncronization primitive like threading.Event.
To do so, you need to modify your Readfile class like this:
from threading import Event
class Readfile(threading.Thread):
# setup readfile class
def __init__(self):
# setup threading
threading.Thread.__init__(self)
# persist values
self.running = False
self.reading = False
self.reading_file = Event() # Initialize your event here it here
def setfilename(self,filename):
self.filename = filename
self.reading_file.set() # This awakens the reader
def _start(self):
self.running = True
self.reading_file.wait() # Thread will be stopped until readfilename is called
self.startread()
Another syncronization primitive worth exploring is queue.Queue. It could be useful if you want to process more than one filename.
The pattern you describe in your question is called Busy Waiting, and should be avoided when possible.

Send file pointer to python thread and update file pointer

I have a python program with a thread and the thread should write into a file. I will spawn a thread from the main program. Now on new day trigger I will change the file pointer in the main program and I want the thread also to take the new file to write the data to the file.
I have a code which will take global variable and do this task. But is there any other better way of doing this?
#!/usr/bin/env python
import sys
import threading
import time
filePtr = None
import time
def fileWriteTh():
global filePtr
time.sleep(2)
filePtr.write("from the thrread this should in file 2")
def main():
global filePtr
filePtr = open("test1.txt","ab")
fileThread = threading.Thread(target=fileWriteTh)
fileThread.start()
if new_day_trigger:
filePtr.close()
filePtr = open("test2.txt","ab")
fileThread.join()
if __name__ == "__main__":
main()
This is the new code that is written:
#!/usr/bin/env python
import sys
import threading
import time
class SendPacket(object):
fileDesc = None
def __init__(self, fd):
super(SendPacket, self).__init__()
SendPacket.fileDesc = fd
def printFromInstance(self,var):
print var
SendPacket.fileDesc.write(var)
time.sleep(3)
print var
SendPacket.fileDesc.write(var)
def startabc(self, someVar):
self.printFromInstance(someVar)
#classmethod
def printVar(cls, printStr):
print printStr
cls.fileDesc.write(printStr)
#classmethod
def changeClsFile(cls, newFd):
cls.fileDesc = newFd
def main():
filePtr = open("test1.txt","ab")
sendPack_inst = SendPacket(filePtr)
fileThread = threading.Thread(target=sendPack_inst.startabc, args=("test1",))
fileThread.start()
time.sleep(2)
filePtr.close()
filePtr = open("test2.txt","ab")
SendPacket.changeClsFile(filePtr)
fileThread.join()
filePtr.close()
if __name__ == "__main__":
main()

Like this:
#!/usr/bin/env python
import sys
import thread
import time
class _fileACT :
def __init__(self):
self.trigger = 0
self.flag = True
self.msg = ""
self.files = (open("test1.txt","ab"),open("test2.txt","ab"))
def run(self,pssrg):
while self.flag :
if self.msg != "" :
self.files[self.trigger].write(self.msg)
self.msg = ""
def test(self,pssrg):
for i in range(20):
time.sleep(1)
if i %2 != 0 :
self.trigger = 0
elif i %2 != 1:
self.trigger = 1
self.msg = "%0.3d test-1,asdasdasd\n"%i
time.sleep(0.5)
print "wait..."
self.flag = False
for e in self.files : e.close()
print "can exit !"
if __name__ == "__main__":
fileACT = _fileACT()
thread.start_new_thread(fileACT.run,(None,))
thread.start_new_thread(fileACT.test,(None,))
We have three variables, filename, last opened file name and message. Two files, only False and True will be sufficient (of course you can use index for multiple files). We've written a test function into the class because we don't want our main cycle to freeze. The file selection is done with ' trigger ', but the previous and next file name is not the same, the previous closes.
The important point in the thread is that the time delay is strictly unavailable! The time delay is always applied to the trigger. The time delay cannot be placed in the main loop. An instance of access from outside the class is also attached. I hope it helps.

Background Process Locking up GUI Python

I have a background Process (using Process from multiprocessing) that is pushing objects to my GUI, however this background process keeps locking up the GUI and the changes being pushed are never being displayed. The objects are being put in to my queue, however the update method in my GUI isn't being called regularly. What can I do make the GUI update more regularly? My GUI is written in Tkinter.
My background process has a infinite loop within it because I always need to keep reading the USB port for more data, so basically my code looks like this:
TracerAccess.py
import usb
from types import *
import sys
from multiprocessing import Process, Queue
import time
__idVendor__ = 0xFFFF
__idProduct__ = 0xFFFF
END_POINT = 0x82
def __printHEXList__(list):
print ' '.join('%02x' % b for b in list)
def checkDeviceConnected():
dev = usb.core.find(idVendor=__idVendor__, idProduct=__idProduct__)
if dev is None:
return False
else:
return True
class LowLevelAccess():
def __init__(self):
self.rawIn = []
self.tracer = usb.core.find(idVendor=__idVendor__, idProduct=__idProduct__)
if self.tracer is None:
raise ValueError("Device not connected")
self.tracer.set_configuration()
def readUSB(self):
"""
This method reads the USB data from the simtracer.
"""
try:
tmp = self.tracer.read(END_POINT, 10000,None, 100000).tolist()
while(self.checkForEmptyData(tmp)):
tmp = self.tracer.read(END_POINT, 10000,None, 100000).tolist()
self.rawIn = tmp
except:
time.sleep(1)
self.readUSB()
def checkForEmptyData(self, raw):
if(len(raw) == 10 or raw[10] is 0x60 or len(raw) == 11):
return True
else:
return False
class DataAbstraction:
def __init__(self, queue):
self.queue = queue
self.lowLevel = LowLevelAccess()
def readInput(self):
while True:
self.lowLevel.readUSB()
raw = self.lowLevel.rawIn
self.queue.put(raw)
ui.py
from Tkinter import *
import time
import TracerAccess as io
from multiprocessing import Process, Queue
from Queue import Empty
from math import ceil
def findNumberOfLines(message):
lines = message.split("\n")
return len(lines)
class Application(Frame):
def addTextToRaw(self, text, changeColour=False, numberOfLines=0):
self.rawText.config(state=NORMAL)
if changeColour is True:
self.rawText.insert(END,text, 'oddLine')
else:
self.rawText.insert(END,text)
self.rawText.config(state=DISABLED)
def updateRaw(self, text):
if(self.numberOfData() % 2 is not 0):
self.addTextToRaw(text, True)
else:
self.addTextToRaw(text)
def startTrace(self):
self.dataAbstraction = io.DataAbstraction(self.queue)
self.splitProc = Process(target=self.dataAbstraction.readInput())
self.stopButton.config(state="normal")
self.startButton.config(state="disabled")
self.splitProc.start()
def pollQueue(self):
try:
data = self.queue.get(0)
self.dataReturned.append(data)
self.updateRaw(str(data).upper())
self.rawText.tag_config("oddLine", background="#F3F6FA")
except Empty:
pass
finally:
try:
if(self.splitProc.is_alive() is False):
self.stopButton.config(state="disabled")
self.startButton.config(state="normal")
except AttributeError:
pass
self.master.after(10, self.pollQueue)
def stopTrace(self):
self.splitProc.join()
self.stopButton.config(state="disabled")
self.startButton.config(state="normal")
def createWidgets(self):
self.startButton = Button(self)
self.startButton["text"] = "Start"
self.startButton["command"] = self.startTrace
self.startButton.grid(row = 0, column=0)
self.stopButton = Button(self)
self.stopButton["text"] = "Stop"
self.stopButton["command"] = self.stopTrace
self.stopButton.config(state="disabled")
self.stopButton.grid(row = 0, column=1)
self.rawText = Text(self, state=DISABLED, width=82)
self.rawText.grid(row=1, columnspan=4)
def __init__(self, master):
Frame.__init__(self, master)
self.queue = Queue()
self.master.after(10, self.pollQueue)
self.pack()
self.dataReturned = []
self.createWidgets()
def numberOfData(self):
return len(self.dataReturned)
Main.py
import ui as ui
if __name__ == "__main__":
root = Tk()
root.columnconfigure(0,weight=1)
app = ui.Application(root)
app.mainloop()
So the background thread never finishes, however when I end the process the UI starts to be displayed before closing. The problem could have appeared because of my design for the TracerAccess.py module as I developed this after moving straight form java and little to no design experience for python.

What multiprocess.Process does, internally, is really a fork(), which effectively duplicated your process. You can perhaps visualize it as:
/ ["background" process] -------------\
[main process] --+ +-- [main process]
\ [main process continued] -----------/
p.join() attempts to "join" the two processes back to one. This effectively means: waiting until the background process is finished. Here's the actual (full) code from the .join() function:
def join(self, timeout=None):
'''
Wait until child process terminates
'''
assert self._parent_pid == os.getpid(), 'can only join a child process'
assert self._popen is not None, 'can only join a started process'
res = self._popen.wait(timeout)
if res is not None:
_current_process._children.discard(self)
Note how self._popen.wait is called.
This is obviously not what you want.
What you probably want, in the context of TKinter, is use the tk event loop, for example like this (Python 3, but the concept also works on Python 2)
from multiprocessing import Process, Queue
import time, tkinter, queue, random, sys
class Test:
def __init__(self, root):
self.root = root
txt = tkinter.Text(root)
txt.pack()
self.q = Queue()
p = Process(target=self.bg)
p.start()
self.checkqueue()
print('__init__ done', end=' ')
def bg(self):
print('Starting bg loop', end=' ')
n = 42
while True:
# Burn some CPU cycles
(int(random.random() * 199999)) ** (int(random.random() * 1999999))
n += 1
self.q.put(n)
print('bg task finished', end=' ')
def checkqueue(self):
try:
print(self.q.get_nowait(), end=' ')
except queue.Empty:
print('Queue empty', end=' ')
sys.stdout.flush()
# Run myself again after 1 second
self.root.after(1000, self.checkqueue)
root = tkinter.Tk()
Test(root)
root.mainloop()
You don't call the .join(), and instead use the .after() method, which schedules a function to run after n microseconds (if you've ever used Javascript, then think setTimeout()) to read the queue.
Depending on the actual content of your bg() function, you may not event need multiprocesing at all, just scheduling a function with .after() may be enough.
Also see:
http://tkinter.unpythonic.net/wiki/UsingTheEventLoop

Creating interruptible process in python

I'm creating a python script of which parses a large (but simple) CSV.
It'll take some time to process. I would like the ability to interrupt the parsing of the CSV so I can continue at a later stage.
Currently I have this - of which lives in a larger class: (unfinished)
Edit:
I have some changed code. But the system will parse over 3 million rows.
def parseData(self)
reader = csv.reader(open(self.file))
for id, title, disc in reader:
print "%-5s %-50s %s" % (id, title, disc)
l = LegacyData()
l.old_id = int(id)
l.name = title
l.disc_number = disc
l.parsed = False
l.save()
This is the old code.
def parseData(self):
#first line start
fields = self.data.next()
for row in self.data:
items = zip(fields, row)
item = {}
for (name, value) in items:
item[name] = value.strip()
self.save(item)
Thanks guys.

If under linux, hit Ctrl-Z and stop the running process. Type "fg" to bring it back and start where you stopped it.

You can use signal to catch the event. This is a mockup of a parser than can catch CTRL-C on windows and stop parsing:
import signal, tme, sys
def onInterupt(signum, frame):
raise Interupted()
try:
#windows
signal.signal(signal.CTRL_C_EVENT, onInterupt)
except:
pass
class Interupted(Exception): pass
class InteruptableParser(object):
def __init__(self, previous_parsed_lines=0):
self.parsed_lines = previous_parsed_lines
def _parse(self, line):
# do stuff
time.sleep(1) #mock up
self.parsed_lines += 1
print 'parsed %d' % self.parsed_lines
def parse(self, filelike):
for line in filelike:
try:
self._parse(line)
except Interupted:
print 'caught interupt'
self.save()
print 'exiting ...'
sys.exit(0)
def save(self):
# do what you need to save state
# like write the parse_lines to a file maybe
pass
parser = InteruptableParser()
parser.parse([1,2,3])
Can't test it though as I'm on linux at the moment.

The way I'd do it:
Puty the actual processing code in a class, and on that class I'd implement the Pickle protocol (http://docs.python.org/library/pickle.html ) (basically, write proper __getstate__ and __setstate__ functions)
This class would accept the filename, keep the open file, and the CSV reader instance as instance members. The __getstate__ method would save the current file position, and setstate would reopen the file, forward it to the proper position, and create a new reader.
I'd perform the actuall work in an __iter__ method, that would yeld to an external function after each line was processed.
This external function would run a "main loop" monitoring input for interrupts (sockets, keyboard, state of an specific file on the filesystem, etc...) - everything being quiet, it would just call for the next iteration of the processor. If an interrupt happens, it would pickle the processor state to an specific file on disk.
When startingm the program just has to check if a there is a saved execution, if so, use pickle to retrieve the executor object, and resume the main loop.
Here goes some (untested) code - the iea is simple enough:
from cPickle import load, dump
import csv
import os, sys
SAVEFILE = "running.pkl"
STOPNOWFILE = "stop.now"
class Processor(object):
def __init__(self, filename):
self.file = open(filename, "rt")
self.reader = csv.reader(self.file)
def __iter__(self):
for line in self.reader():
# do stuff
yield None
def __getstate__(self):
return (self.file.name, self.file.tell())
def __setstate__(self, state):
self.file = open(state[0],"rt")
self.file.seek(state[1])
self.reader = csv.reader(self.File)
def check_for_interrupts():
# Use your imagination here!
# One simple thing would e to check for the existence of an specific file
# on disk.
# But you go all the way up to instantiate a tcp server and listen to
# interruptions on the network
if os.path.exists(STOPNOWFILE):
return True
return False
def main():
if os.path.exists(SAVEFILE):
with open(SAVEFILE) as savefile:
processor = load(savefile)
os.unlink(savefile)
else:
#Assumes the name of the .csv file to be passed on the command line
processor = Processor(sys.argv[1])
for line in processor:
if check_for_interrupts():
with open(SAVEFILE, "wb") as savefile:
dump(processor)
break
if __name__ == "__main__":
main()

My Complete Code
I followed the advice of #jsbueno with a flag - but instead of another file, I kept it within the class as a variable:
I create a class - when I call it asks for ANY input and then begins another process doing my work. As its looped - if I were to press a key, the flag is set and only checked when the loop is called for my next parse. Thus I don't kill the current action.
Adding a process flag in the database for each object from the data I'm calling means I can start this any any time and resume where I left off.
class MultithreadParsing(object):
process = None
process_flag = True
def f(self):
print "\nMultithreadParsing has started\n"
while self.process_flag:
''' get my object from database '''
legacy = LegacyData.objects.filter(parsed=False)[0:1]
if legacy:
print "Processing: %s %s" % (legacy[0].name, legacy[0].disc_number)
for l in legacy:
''' ... Do what I want it to do ...'''
sleep(1)
else:
self.process_flag = False
print "Nothing to parse"
def __init__(self):
self.process = Process(target=self.f)
self.process.start()
print self.process
a = raw_input("Press any key to stop \n")
print "\nKILL FLAG HAS BEEN SENT\n"
if a:
print "\nKILL\n"
self.process_flag = False
Thanks for all you help guys (especially yours #jsbueno) - if it wasn't for you I wouldn't have got this class idea.