We Keep Coding

Multithreading - One long thread vs many little threads

I have an application that has 3 unique tasks that need to be executed once per second and need to run in parallel. I made the application to have 3 long running threads, basically a thread with a while True and it only breaks when the application is stopped.
I have a Java developer friend that suggested instead to make the task perform then stop and basically execute the same function over and over again, essentially creating a new thread each second. The while True is outside of the threads and creates them over and over. The idea being that if for some reason one dies or hangs, it doesn't cause issues. Any unforeseen errors, the next second it moves on and tries again. I like this idea but it's slightly harder to implement.
Curious what is the best approach for Python:
3 long running threads that loop inside the thread
1 while loop that creates new threads each second
One last note, I need to have some data that carries over each run, so for #2 I would have a class created and a class method would be called as a thread.
#2 would look something like this:
import concurrent.futures
import time
class Test:
def __init__(self) -> None:
super().__init__()
self.counter = 0
def run(self):
print('Test')
self.counter += 1
test1 = Test()
test2 = Test()
with concurrent.futures.ThreadPoolExecutor() as executer:
while True:
t1 = executer.submit(test1.run)
t2 = executer.submit(test2.run)
time.sleep(1.0)

Schedule at given time meanwhile a function runs - Python

I have been searching for different schedule in Python such as Sched (Im a Windows user) etc. However I can't really get a grip on it and I don't know if it is possible. My plan is to make like the picture below:
We can see at Time:00.21 is etc the time I want the program to do the function 2 BUT the function 1 should be add into a list I have made as many as possible in the list as it works in 2 minutes before the timer hits. Basically...
The function 1 is doing its function 2 minutes before the timer. When it hits 00:21 then stop the function 1 and do the function 2 where it takes the List and uses it in its own function and when its done then its done.
However I don't know how to do this or to start. I was thinking to do a own timer but it feels like that is not the solution. What do you guys suggest?

I think I would approach a problem like this by creating a class that subclasses threading.Thread. From there, you override the run method with the function that you want to perform, which in this case will put stuff in a list. Then, in main, you start that thread followed by a call to sleep. The class would look like this:
class ListBuilder(threading.Thread):
def__init__(self):
super().__init__()
self._finished = False
self.lst = []
def get_data():
# This is the data retrieval function
# It could be imported in, defined outside the class, or made static.
def run(self):
while not self._finished:
self.lst.append(self.get_data())
def stop(self):
self._finished = True
Your main would then look something like
import time
if __name__ == '__main__':
lb = ListBuilder()
lb.start()
time.sleep(120) # sleep for 120 seconds, 2 minutes
lb.stop()
time.sleep(.1) # A time buffer to make sure the final while loop finishes
# Depending on how long each while loop iteration takes,
# it may not be necessary or it may need to be longer
do_stuf(lb.lst) # performs actions on the resulting list
Now, all you have to do is use the Windows Task Scheduler to run it at 00:19 and you should be set.

python, calling method on main thread from timer callback

I'm very new to python development, I need to call a function every x seconds.
So I'm trying to use a timer for that, something like:
def start_working_interval():
def timer_tick():
do_some_work() // need to be called on the main thread
timer = threading.Timer(10.0, timer_tick)
timer.start()
timer = threading.Timer(10.0, timer_tick)
timer.start()
the do_some_work() method need to be called on the main thread, and I think using the timer causing it to execute on different thread.
so my question is, how can I call this method on the main thread?

I'm now sure what you trying to achive but i played with your code and did this:
import threading
import datetime
def do_some_work():
print datetime.datetime.now()
def start_working_interval():
def timer_tick():
do_some_work()
timer = threading.Timer(10.0, timer_tick)
timer.start()
timer_tick()
start_working_interval()
So basically what i did was to set the Time inside the timer_tick() so it will call it-self after 10 sec and so on, but i removed the second timer.

I needed to do this too, here's what I did:
import time
MAXBLOCKINGSECONDS=5 #maximum time that a new task will have to wait before it's presence in the queue gets noticed.
class repeater:
repeatergroup=[] #our only static data member it holds the current list of the repeaters that need to be serviced
def __init__(self,callback,interval):
self.callback=callback
self.interval=abs(interval) #because negative makes no sense, probably assert would be better.
self.reset()
self.processing=False
def reset(self):
self.nextevent=time.time()+self.interval
def whennext(self):
return self.nextevent-time.time() #time until next event
def service(self):
if time.time()>=self.nextevent:
if self.processing=True: #or however you want to be re-entrant safe or thread safe
return 0
self.processing==True
self.callback(self) #just stuff all your args into the class and pull them back out?
#use this calculation if you don't want slew
self.nextevent+=self.interval
#reuse this calculation if you do want slew/don't want backlog
#self.reset()
#or put it just before the callback
self.processing=False
return 1
return 0
#this the transition code between class and classgroup
#I had these three as a property getter and setter but it was behaving badly/oddly
def isenabled(self):
return (self in self.repeatergroup)
def start(self):
if not (self in self.repeatergroup):
self.repeatergroup.append(self)
#another logical place to call reset if you don't want backlog:
#self.reset()
def stop(self):
if (self in self.repeatergroup):
self.repeatergroup.remove(self)
#group calls in c++ I'd make these static
def serviceall(self): #the VB hacker in me wants to name this doevents(), the c hacker in me wants to name this probe
ret=0
for r in self.repeatergroup:
ret+=r.service()
return ret
def minwhennext(self,max): #this should probably be hidden
ret=max
for r in self.repeatergroup:
ret=min(ret,r.whennext())
return ret
def sleep(self,seconds):
if not isinstance(threading.current_thread(), threading._MainThread): #if we're not on the main thread, don't process handlers, just sleep.
time.sleep(seconds)
return
endtime=time.time()+seconds #record when caller wants control back
while time.time()<=endtime: #spin until then
while self.serviceall()>0: #service each member of the group until none need service
if (time.time()>=endtime):
return #break out of service loop if caller needs control back already
#done with servicing for a while, yield control to os until we have
#another repeater to service or it's time to return control to the caller
minsleeptime=min(endtime-time.time(),MAXBLOCKINGPERIOD) #smaller of caller's requested blocking time, and our sanity number (1 min might be find for some systems, 5 seconds is good for some systems, 0.25 to 0.03 might be better if there could be video refresh code waiting, 0.15-0.3 seems a common range for software denouncing of hardware buttons.
minsleeptime=self.minwhennext(minsleeptime)
time.sleep(max(0,minsleeptime))
###################################################################
# and now some demo code:
def handler1(repeater):
print("latency is currently {0:0.7}".format(time.time()-repeater.nextevent))
repeater.count+=repeater.interval
print("Seconds: {0}".format(repeater.count))
def handler2(repeater): #or self if you prefer
print("Timed message is: {0}".format(repeater.message))
if repeater.other.isenabled():
repeater.other.stop()
else:
repeater.other.start()
repeater.interval+=1
def demo_main():
counter=repeater(handler1,1)
counter.count=0 #I'm still new enough to python
counter.start()
greeter=repeater(handler2,2)
greeter.message="Hello world." #that this feels like cheating
greeter.other=counter #but it simplifies everything.
greeter.start()
print ("Currently {0} repeaters in service group.".format(len(repeater.repeatergroup)))
print("About to yield control for a while")
greeter.sleep(10)
print("Got control back, going to do some processing")
time.sleep(5)
print("About to yield control for a while")
counter.sleep(20) #you can use any repeater to access sleep() but
#it will only service those currently enabled.
#notice how it gets behind but tries to catch up, we could add repeater.reset()
#at the beginning of a handler to make it ignore missed events, or at the
#end to let the timing slide, depending on what kind of processing we're doing
#and what sort of sensitivity there is to time.
#now just replace all your main thread's calls to time.sleep() with calls to mycounter.sleep()
#now just add a repeater.sleep(.01) or a while repeater.serviceall(): pass to any loop that will take too long.
demo_main()
There's a couple of odd things left to consider:
Would it be better to sort handlers that you'd prefer to run on main thread from handlers that you don't care? I later went on to add a threadingstyle property, which depending on it's value would run on main thread only, on either main thread or a shared/group thread, or stand alone on it's own thread. That way longer or more time-sensitive tasks, could run without causing the other threads to be as slowed down, or closer to their scheduled time.
I wonder whether, depending on the implementation details of threading: is my 'if not main thread: time.sleep(seconds); return' effectively make it sufficiently more likely to be the main thread's turn, and I shouldn't worry about the difference.
(It seems like adding our MAXBLOCKINGPERIOD as the 3rd arg to the sched library could fix it's notorious issue of not servicing new events after older longer in the future events have already hit the front of the queue.)

How to schedule tasks without exiting existing loop?

I have struggled with this question for about a week -- time to ask someone who can bang out an answer in a couple minutes.
I am trying to run a python program once every 10 seconds. There are a lot of questions of this sort : Use sched module to run at a given time, Python threading.timer - repeat function every 'n' seconds, How to execute a function asynchronously every 60 seconds in Python?
Normally the solutions using sched or time.sleep would work, but I am trying to start a scheduled process from within cmd2, which is already running in a while False loop. (When you exit cmd2, it exits this loop).
Because of this, when I start a function to repeat every 10 seconds, I enter another loop nested within cmd2 and I am unable to enter cmd2 commands. I can only get back to cmd2 by exiting the sub-loop that is repeating the function, and thus the function stops repeating.
Evidently threading will solve this problem. I have tried threading.Timer without success. Perhaps the real problem is that I do not understand threads or multiprocessing.
Here is an example of code that is roughly isomorphic to the code I'm using, using sched module, which I got to work:
import cmd2
import repeated
class prompt(cmd2.Cmd):
"""this lets you enter commands"""
def default(self, line):
return cmd2.Cmd.default(self, line)
def do_exit(self, line):
return True
def do_repeated(self, line):
repeated.func_1()
Where repeated.py looks like this:
import sched
import time
def func_2(sc):
print 'doing stuff'
sc.enter(10, 0, func_2, (sc,))
def func_1():
s = sched.scheduler(time.time, time.sleep)
s.enter(0, 0, func_2, (s,))
s.run()

http://docs.python.org/2/library/queue.html?highlight=queue#Queue
Can you instance a Queue object outside of cmd2? There can be one thread that watches the queue and takes jobs from it at periodic intervals; while cmd2 is free to run or not run. The thread that processes the queue, and the queue object itself need to be in the outer scope, of course.
To schedule something at a particular time, you can insert a tuple which has the target time in it. Or you can have the thread just check at regular intervals, if that's good enough.
[Edit, if you have a process that is intended to repeat, you can have it requeue itself at the end of it's operation.]

As soon as I asked the question I was able to figure it out. Don't know why that happens sometimes.
This code
def f():
# do something here ...
# call f() again in 60 seconds
threading.Timer(60, f).start()
# start calling f now and every 60 sec thereafter
f()
From here: How to execute a function asynchronously every 60 seconds in Python?
Actually works for what I was trying to do. There are evidently some subtleties in how the function is called as an argument in threading.Timer. Before when I was including the arguments and even the parentheses after the function I was getting recursive depth errors --i.e. the function was calling itself without delay constantly.
So anyone else who has a problem like this, pay attention to how you call the function in threading.Timer(60, f).start(). If you write threading.Timer(60, f()).start() or something similar it will probably not work.

timer interrupt thread python

I've been trying to make a precise timer in python, or as precise a OS allows it to be. But It seems to be more complicated than I initially thought.
This is how I would like it to work:
from time import sleep
from threading import Timer
def do_this():
print ("hello, world")
t = Timer(4, do_this)
t.start()
sleep(20)
t.cancel()
Where during 20 seconds I would execute 'do_this' every fourth second. However 'do_this' executes once then the script terminates after 20 seconds.
Another way would be to create a thread with a while loop.
import time
import threading
import datetime
shutdown_event = threading.Event()
def dowork():
while not shutdown_event.is_set():
print(datetime.datetime.now())
time.sleep(1.0)
def main():
t = threading.Thread(target=dowork, args=(), name='worker')
t.start()
print("Instance started")
try:
while t.isAlive():
t.join(timeout=1.0)
except (KeyboardInterrupt, SystemExit):
shutdown_event.set()
pass
if __name__ == '__main__':
main()
This thread executes as expected but I get a timing drift. In this case have to compensate for the time it takes to execute the code in the while loop by adjusting the sleep accordingly.
Is there a simple way in python to execute a timer every second (or any interval) without introducing a drift compared to the system time without having to compensate the sleep(n) parameter?
Thanks for helping,
/Anders

If dowork() always runs in less time than your intervals, you can spawn a new thread every 4 seconds in a loop:
def dowork():
wlen = random.random()
sleep(wlen) # Emulate doing some work
print 'work done in %0.2f seconds' % wlen
def main():
while 1:
t = threading.Thread(target=dowork)
time.sleep(4)
If dowork() could potentially run for more than 4 seconds, then in your main loop you want to make sure the previous job is finished before spawning a new one.
However, time.sleep() can itself drift because no guarantees are made on how long the thread will actually be suspended. The correct way of doing it would be to figure out how long the job took and sleep for the remaining of the interval. I think this is how UI and game rendering engines work, where they have to display fixed number of frames per second at fixed times and rendering each frame could take different length of time to complete.