I have a function def act(obs) that returns a float and is computationally expensive (takes some time to run).
import time
import random
def act(obs):
time.sleep(5) # mimic computation time
action = random.random()
return action
I regularly keep calling this function in a script faster than how long it takes for it to execute. I do not want any waiting time when calling the function. Rather I prefer using the returned value from an earlier computation. How do I achieve this?
Something I have thought of is having a global variable that updated in the function and I keep reading the global variable although I am not sure if that is the best way to achieve it.
This is what I ended up using based on this answer
class MyClass:
def __init__(self):
self.is_updating = False
self.result = -1
def _act(self, obs):
self.is_updating = True
time.sleep(5)
self.result = obs
self.is_updating = False
def act(self, obs):
if not self.is_updating:
threading.Thread(target=self._act, args=[obs]).start()
return self.result
agent = MyClass()
i = 0
while True:
agent.act(obs=i)
time.sleep(2)
print(i, agent.result)
i += 1
The global variable way should work, Also you can have a class that has a private member let's say result and a flag isComputing and a method getResult which would call a method compute()(via a thread) if it is not computing currently, and returns the previous result. The compute() method should update the flag isComputing properly.
Related
I am currently having issues to do with my understanding of threading or possibly my understanding of how variables are passed/assigned thru threads in python. I have this simple program that takes in a list of current stocks that are displayed on a screen and grabs the stock information related to those. I am using threads so that I can constantly update the screen and constantly collect data. I am having two issues:
Inside dataCollector_thread() i understand that if i append to the stocksOnScreenListInfo that the variable (stocksOnScreenListInfo) inside main is updated.
However I don't want to append to the list but rather just reassign the list like the following but this does not work?.
def dataCollector_thread(stocksOnScreenListInfo, stocksOnScreen):
while(True):
placeholder = []
for stock in stocksOnScreen:
placeholer.append(RetrieveQuote(stock))
stocksOnScreenListInfo = placeholder
time.sleep(5)
Inside screenUpdate_thread i am wanting to update stocksOnScreen to the variable 'TSLA' defined by the function UpdateScreen. This does not seem to update its corresponding stocksOnScreen in main as when I print to check it continues to say 'AAPL'?
def main(args):
stocksOnScreen = ["AAPL"] # List of the stocks currently displayed on LED screen
stocksOnScreenListInfo = [] # The quote information list for each stock on screen
thread_data_collector = threading.Thread(target=dataCollector_thread, args=(stocksOnScreenListInfo,stocksOnScreen))
thread_data_collector.daemon = True
thread_data_collector.start()
thread_screen = threading.Thread(target=screenUpdate_thread, args=(stocksSearchArray,stocksOnScreen))
thread_screen.daemon = True
thread_screen.start()
def dataCollector_thread(stocksOnScreenListInfo, stocksOnScreen):
while(True):
for stock in stocksOnScreen:
stocksOnScreenListInfo.append(RetrieveQuote(stock))
time.sleep(5)
def screenUpdate_thread(stocksSearchArray, stocksOnScreen):
while(True):
stocksOnScreen = UpdateScreen(stocksSearchArray)
def UpdateScreen(stocksSearchArray):
pass
return ["TSLA"]
There are a couple of issues with this function:
def dataCollector_thread(stocksOnScreenListInfo, stocksOnScreen):
while(True):
placeholder = []
for stock in stocksOnScreen:
placeholer.append(RetrieveQuote(stock))
stocksOnScreenListInfo = placeholder
time.sleep(5)
you're assigning stocksOnScreenListInfo within this function to a new list placeholder. What you want to do is modify the contents in-place so that stocksOnScreenListInfo in main is updated. You can do that like this: stocksOnScreenListInfo[:] = placeholder (which means change contents from beginning to end with the new list).
stocksOnScreen could change while you're iterating it in the for loop since you're updating it in another thread. You should do this atomically. A lock
(that you pass as a parameter to the function) will help here: it's a synchronisation primitive that is designed to prevent data races when multiple threads share data and at least one of them modifies it.
I can't see stocksOnScreenListInfo being used anywhere else in your code. Is it used in another function? If so, you should think about having a lock around that.
I would modify the function like this:
def dataCollector_thread(stocksOnScreenListInfo, stocksOnScreen, lock):
while True:
placeholder = []
with lock: # use lock to ensure you atomically access stocksOnScreen
for stock in stocksOnScreen:
placeholder.append(RetrieveQuote(stock))
stocksOnScreenListInfo[:] = placeholder # modify contents of stocksOnScreenListInfo
time.sleep(5)
In your other thread function:
def screenUpdate_thread(stocksSearchArray, stocksOnScreen):
while(True):
stocksOnScreen = UpdateScreen(stocksSearchArray)
you're assigning stocksOnScreen to a new list within this function; it won't affect stocksOnScreen in main. Again you can do that using the notation stocksOnScreen[:] = new_list. I would lock before before updating stocksOnScreen to ensure your other thread function dataCollector_thread accesses stocksOnScreen atomically like so:
def screenUpdate_thread(stocksSearchArray, stocksOnScreen, lock):
while True:
updated_list = UpdateScreen() # build new list - doesn't have to be atomic
with lock:
stocksOnScreen[:] = updated_list # update contents of stocksOnScreen
time.sleep(0.001)
As you can see I put in a small sleep, otherwise the function will loop constantly and be too CPU-intensive. Plus it will give Python a chance to context switch between your thread functions.
Finally, in main create a lock:
lock = threading.Lock()
and pass lock to both functions as a parameter.
stocksOnScreen = ... changes the reference itself. Since the reference is passed to the function/thread as a parameter, the change is done to the copy of the original reference within the function/thread. (Both function/thread have their own copy.)
So instead you should manipulate the list object it refers to (e.g. list.clear() and list.extend()).
However, as you can see, it's now no longer an atomic action. So there are chances that dataCollector_thread would be working on an empty list (i.e. do nothing) and sleep 5 seconds. I provided a possible workaround/solution below as well. Not sure if it is supposed to work (perfectly) though:
def dataCollector_thread(stocksOnScreen):
while(True):
sos_copy = stocksOnScreen.copy() # *might* avoid the race?
for stock in sos_copy:
print(stock)
if (len(sos_copy) > 0): # *might* avoid the race?
time.sleep(5)
def UpdateScreen():
return ["TSLA"]
def screenUpdate_thread(stocksOnScreen):
while(True):
# manipulate the list object instead of changing the reference (copy) in the function
stocksOnScreen.clear()
# race condition: dataCollector_thread might work on an empty list and sleep 5 seconds
stocksOnScreen.extend(UpdateScreen())
def main():
stocksOnScreen = ["AAPL"] # List of the stocks currently displayed on LED screen
thread_data_collector = threading.Thread(target=dataCollector_thread, args=(stocksOnScreen,)) # note the comma
thread_data_collector.daemon = True
thread_data_collector.start()
thread_screen = threading.Thread(target=screenUpdate_thread, args=(stocksOnScreen,)) # note the comma
thread_screen.daemon = True
thread_screen.start()
Note: according to this answer, python lists are thread-safe, so the copy workaround should work.
You can probably make use of global instead of passing stocksOnScreen as parameter as well:
def dataCollector_thread():
global stocksOnScreen # superfluous if no re-assignment
while(True):
for stock in stocksOnScreen:
print(stock)
time.sleep(5)
def UpdateScreen():
return ["TSLA"]
def screenUpdate_thread():
global stocksOnScreen # needed for re-assignment
while(True):
stocksOnScreen = UpdateScreen()
def main():
global stocksOnScreen # Or create stocksOnScreen outside main(), which is a function itself
stocksOnScreen = ["AAPL"] # List of the stocks currently displayed on LED screen
thread_data_collector = threading.Thread(target=dataCollector_thread)
thread_data_collector.daemon = True
thread_data_collector.start()
thread_screen = threading.Thread(target=screenUpdate_thread)
thread_screen.daemon = True
thread_screen.start()
Ref.: https://docs.python.org/3/faq/programming.html#what-are-the-rules-for-local-and-global-variables-in-python
You have three options here since, python like java passes parameters by value & not reference.
First, use a global parameter.
def threadFunction():
globalParam = "I've ran"
global globalParam
threading.Thread(target=threadFunction)
Second, an Updater Function
def threadFunction(update):
update("I've ran")
threading.Thread(target=threadFunction, args=((lambda x: print(x)),))
Third, Expose global parameter holder
def threadFunction(param1, param2):
globalParams[0]= param1 + " Just Got access"
global globalParams
globalParams = ["Param1","Param2"]
threading.Thread(target=threadFunction, args=(*globalParams))
I hope this answered your question ;)
Note: the suggested answer doesn't work as I call the function every x seconds.
In python I have:
def inline_paid_generator(user_id, password):
login_thread = threading.Thread(target=paid_generator, name="Register", args={user_id,password})
login_thread.start()
How outside that function (from where I called it) how can I know if the thread is alive or not?
You can set the thread as a global variable and only start it in the function, as following:
import time
import threading
def wait():
time.sleep(5)
login_thread = threading.Thread(target=wait)
def start_inline_paid_generator():
login_thread.start()
print(login_thread.is_alive()) # False
start_inline_paid_generator()
print(login_thread.is_alive()) # True
Then you can check anywhere in your program with login_thread.is_alive()
Return the created thread from the function and check if it's alive:
def inline_paid_generator(user_id, password):
login_thread = threading.Thread(target=paid_generator, name="Register", args={user_id,password})
login_thread.start()
return login_thread
t = inline_paid_generator(...)
...
if not t.is_alive():
t = inline_paid_generator(...)
I'm trying to separate my logic (function or task) from main loop. This function usually takes long time to run. In following example calculate() function takes a lot of time to calculate self.time variable.
In other words I want calculate() to be called asynchronously. In every iteration first I would check if calculate() is running, if not then call it. Secondly I would call show() function in every iteration, no matter if calculate() changed self.time value.
import time
import datetime
class Clock:
def __init__(self):
self.time = None
def calculate(self):
time.sleep(3)
self.time = datetime.datetime.now()
def show(self):
print(f"{self.time.minute}:{self.time.second}")
def loop(self):
while True:
self.calculate() # this should be asynchronous
self.show() # this should be called every iteration
if __name__ == '__main__':
clock = Clock()
clock.loop()
Output:
36:9
36:12
36:15
36:18
This is not a wanted outcome. What I want:
36:9
...
36:9
36:9
36:12
...
36:12
36:15
36:15
36:15
...
36:15
36:18
36:18
...
36:18
How I've tried to solve this (this probably not the best solution, so future reader please check answers):
import time
import datetime
import threading
class Clock:
def __init__(self):
self.time = datetime.datetime.now()
self.__thread = None
def calculate(self):
time.sleep(3)
self.time = datetime.datetime.now() # race condition?
def show(self):
print(f"{self.time.minute}:{self.time.second}")
def loop(self):
while True:
if self.__thread is None or not self.__thread.is_alive():
self.__thread = threading.Thread(target=self.calculate)
self.__thread.start()
self.show()
if __name__ == '__main__':
clock = Clock()
clock.loop()
Program output is what I wanted. Is there any flaw in this solution? I can think only of race condition.
I am aware that's not the greatest description of this problem. If you
could help me with editing this post with more searchable and specific
vocabulary I would be grateful. If you have idea for better title please
leave it in comments.
I checked your code in my IDE, and it looks like exactly what you wanted using an acceptable approach. The only alternative solution I can think of is: 1) use a temp variable that goes up by one until it gets to 3 then reset. 2) similar idea but use time.nextTime = 3 to jump every 3 seconds and print current time.
Also, your first code solution seems to take some time to run is because it's waiting 3 seconds to show the time. And regarding race condition, that's not an issue since you're not using multiple threads at the same time in your second solution.
I have the following code in a class:
def __setattr__(self, key, value):
self.__dict__['d'][key] = value
...
self.saveToIni()
The saveToIni function saves all the dict's items to an ini file at every object's setattr call. If 80 setattr calls are made in the last 120ms, then the file will be written from scratch every time. The function also orders and sometimes deletes data from the dictonary, so I don't want to change it.
I want to limit the calls to once in, let's say, 5 seconds:
When the first setattr is triggered, a timer starts asynchronicly, still not running saveToIni.
If any calls are made to setattr and the timer is still counting, it will nor fire a timer nor run saveToIni.
When the timer times out, the saveToIni should launch.
Now, I'm not sure how to achieve this behavoir. I've thought about messing with threads, but still didn't found the idea about how to do it.
The way I would go, is to create a Timer Thread which runs a timedSaveToIni function once every 5 sec.
I would also have a tag isSaveRequested telling this function if it should actually write data to the disk or not.
The _setattr_ function would simply set this tag to true.
Your code would look like this:
class Dict:
def __init__(self):
...
self.isSaveRequested = False
self.timedSaveToIni()
def timedSaveToIni(self):
threading.Timer(5.0, self.timedSaveToIni).start()
if self.isSaveRequested:
self.saveToIni()
self.isSaveRequested = False
def __setattr__(self, key, value):
self.__dict__['d'][key] = value
...
self.isSaveRequested = True
def saveToInit()
...
Perhaps you'd just have the function remember when it was last called and until 5 seconds have passed since the last call just have it block (sleep). That won't really require any sort of threading. Here's an example - just an idea, not specific to your use case.
import time
def timed_func():
while (timed_func.last_use + 5) > time.time():
time.sleep(1)
print "I am working..." # Do the job
timed_func.last_use = time.time()
def main():
timed_func.last_use = time.time() - 5
while True:
timed_func()
if __name__ == '__main__':
main()
I have recently posted a question about how to postpone execution of a function in Python (kind of equivalent to Javascript setTimeout) and it turns out to be a simple task using threading.Timer (well, simple as long as the function does not share state with other code, but that would create problems in any event-driven environment).
Now I am trying to do better and emulate setInterval. For those who are not familiar with Javascript, setInterval allows to repeat a call to a function every x seconds, without blocking the execution of other code. I have created this example decorator:
import time, threading
def setInterval(interval, times = -1):
# This will be the actual decorator,
# with fixed interval and times parameter
def outer_wrap(function):
# This will be the function to be
# called
def wrap(*args, **kwargs):
# This is another function to be executed
# in a different thread to simulate setInterval
def inner_wrap():
i = 0
while i != times:
time.sleep(interval)
function(*args, **kwargs)
i += 1
threading.Timer(0, inner_wrap).start()
return wrap
return outer_wrap
to be used as follows
#setInterval(1, 3)
def foo(a):
print(a)
foo('bar')
# Will print 'bar' 3 times with 1 second delays
and it seems to me it is working fine. My problem is that
it seems overly complicated, and I fear I may have missed a simpler/better mechanism
the decorator can be called without the second parameter, in which case it will go on forever. When I say foreover, I mean forever - even calling sys.exit() from the main thread will not stop it, nor will hitting Ctrl+c. The only way to stop it is to kill python process from the outside. I would like to be able to send a signal from the main thread that would stop the callback. But I am a beginner with threads - how can I communicate between them?
EDIT In case anyone wonders, this is the final version of the decorator, thanks to the help of jd
import threading
def setInterval(interval, times = -1):
# This will be the actual decorator,
# with fixed interval and times parameter
def outer_wrap(function):
# This will be the function to be
# called
def wrap(*args, **kwargs):
stop = threading.Event()
# This is another function to be executed
# in a different thread to simulate setInterval
def inner_wrap():
i = 0
while i != times and not stop.isSet():
stop.wait(interval)
function(*args, **kwargs)
i += 1
t = threading.Timer(0, inner_wrap)
t.daemon = True
t.start()
return stop
return wrap
return outer_wrap
It can be used with a fixed amount of repetitions as above
#setInterval(1, 3)
def foo(a):
print(a)
foo('bar')
# Will print 'bar' 3 times with 1 second delays
or can be left to run until it receives a stop signal
import time
#setInterval(1)
def foo(a):
print(a)
stopper = foo('bar')
time.sleep(5)
stopper.set()
# It will stop here, after printing 'bar' 5 times.
Your solution looks fine to me.
There are several ways to communicate with threads. To order a thread to stop, you can use threading.Event(), which has a wait() method that you can use instead of time.sleep().
stop_event = threading.Event()
...
stop_event.wait(1.)
if stop_event.isSet():
return
...
For your thread to exit when the program is terminated, set its daemon attribute to True before calling start(). This applies to Timer() objects as well because they subclass threading.Thread. See http://docs.python.org/library/threading.html#threading.Thread.daemon
Maybe these are the easiest setInterval equivalent in python:
import threading
def set_interval(func, sec):
def func_wrapper():
set_interval(func, sec)
func()
t = threading.Timer(sec, func_wrapper)
t.start()
return t
Maybe a bit simpler is to use recursive calls to Timer:
from threading import Timer
import atexit
class Repeat(object):
count = 0
#staticmethod
def repeat(rep, delay, func):
"repeat func rep times with a delay given in seconds"
if Repeat.count < rep:
# call func, you might want to add args here
func()
Repeat.count += 1
# setup a timer which calls repeat recursively
# again, if you need args for func, you have to add them here
timer = Timer(delay, Repeat.repeat, (rep, delay, func))
# register timer.cancel to stop the timer when you exit the interpreter
atexit.register(timer.cancel)
timer.start()
def foo():
print "bar"
Repeat.repeat(3,2,foo)
atexit allows to signal stopping with CTRL-C.
this class Interval
class ali:
def __init__(self):
self.sure = True;
def aliv(self,func,san):
print "ali naber";
self.setInterVal(func, san);
def setInterVal(self,func, san):
# istenilen saniye veya dakika aralığında program calışır.
def func_Calistir():
func(func,san); #calışıcak fonksiyon.
self.t = threading.Timer(san, func_Calistir)
self.t.start()
return self.t
a = ali();
a.setInterVal(a.aliv,5);