We Keep Coding

Python sleep without blocking other processes

I am running a python script every hour and I've been using time.sleep(3600) inside of a while loop. It seems to work as needed but I am worried about it blocking new tasks. My research of this seems to be that it only blocks the current thread but I want to be 100% sure. While the hourly job shouldn't take more than 15min, if it does or if it hangs, I don't want it to block the next one that starts. This is how I've done it:
import threading
import time
def long_hourly_job():
# do some long task
pass
if __name__ == "__main__":
while True:
thr = threading.Thread(target=long_hourly_job)
thr.start()
time.sleep(3600)
Is this sufficient?
Also, the reason i am using time.sleep for this hourly job rather than a cron job is I want to do everything in code to make dockerization cleaner.

The code will work (ie: sleep does only block the calling thread), but you should be careful of some issues. Some of them have been already stated in the comments, like the possibility of time overlaps between threads. The main issue is that your code is slowly leaking resources. After creating a thread, the OS keeps some data structures even after the thread has finished running. This is necessary, for example to keep the thread's exit status until the thread's creator requires it. The function to clear these structures (conceptually equivalent to closing a file) is called join. A thread that has finished running and is not joined is termed a 'zombie thread'. The amount of memory required by these structures is very small, and your program should run for centuries for any reasonable amount of available RAM. Nevertheless, it is a good practice to join all the threads you create. A simple approach (if you know that 3600 s is more than enough time for the thread to finish) would be:
if __name__ == "__main__":
while True:
thr = threading.Thread(target=long_hourly_job)
thr.start()
thr.join(3600) # wait at most 3600 s for the thread to finish
if thr.isAlive(): # join does not return useful information
print("Ooops: the last job did not finish on time")
A better approach if you think that it is possible that sometimes 3600 s is not enough time for the thread to finish:
if __name__ == "__main__":
previous = []
while True:
thr = threading.Thread(target=long_hourly_job)
thr.start()
previous.append(thr)
time.sleep(3600)
for i in reversed(range(len(previous))):
t = previous[i]
t.join(0)
if t.isAlive():
print("Ooops: thread still running")
else:
print("Thread finished")
previous.remove(t)
I know that the print statement makes no sense: use logging instead.

Perhaps a little late. I tested the code from other answers but my main process got stuck (perhaps I'm doing something wrong?). I then tried a different approach. It's based on threading Timer class, but trying to emulate the QtCore.QTimer() behavior and features:
import threading
import time
import traceback
class Timer:
SNOOZE = 0
ONEOFF = 1
def __init__(self, timerType=SNOOZE):
self._timerType = timerType
self._keep = threading.Event()
self._timerSnooze = None
self._timerOneoff = None
class _SnoozeTimer(threading.Timer):
# This uses threading.Timer class, but consumes more CPU?!?!?!
def __init__(self, event, msec, callback, *args):
threading.Thread.__init__(self)
self.stopped = event
self.msec = msec
self.callback = callback
self.args = args
def run(self):
while not self.stopped.wait(self.msec):
self.callback(*self.args)
def start(self, msec: int, callback, *args, start_now=False) -> bool:
started = False
if msec > 0:
if self._timerType == self.SNOOZE:
if self._timerSnooze is None:
self._timerSnooze = self._SnoozeTimer(self._keep, msec / 1000, callback, *args)
self._timerSnooze.start()
if start_now:
callback(*args)
started = True
else:
if self._timerOneoff is None:
self._timerOneoff = threading.Timer(msec / 1000, callback, *args)
self._timerOneoff.start()
started = True
return started
def stop(self):
if self._timerType == self.SNOOZE:
self._keep.set()
self._timerSnooze.join()
else:
self._timerOneoff.cancel()
self._timerOneoff.join()
def is_alive(self):
if self._timerType == self.SNOOZE:
isAlive = self._timerSnooze is not None and self._timerSnooze.is_alive() and not self._keep.is_set()
else:
isAlive = self._timerOneoff is not None and self._timerOneoff.is_alive()
return isAlive
isAlive = is_alive
KEEP = True
def callback():
global KEEP
KEEP = False
print("ENDED", time.strftime("%M:%S"))
if __name__ == "__main__":
count = 0
t = Timer(timerType=Timer.ONEOFF)
t.start(5000, callback)
print("START", time.strftime("%M:%S"))
while KEEP:
if count % 10000000 == 0:
print("STILL RUNNING")
count += 1
Notice the while loop runs in a separate thread, and uses a callback function to invoke when the time is over (in your case, this callback function would be used to check if the long running process has finished).

Updating Popup.Animated to play gif until external task is completed (PYSimpleGUI)

I am looking to create a UI that displays an animated popup while another task is being carried out. That will exit upon completion. I am using PYSimpleGUI and am using the example listed here to base my work off. I can get a single frame of the animation to display once I start the code and exit upon completion of the task but can't get it to play the entire gif. Code:
import queue
import threading
import time
import PySimpleGUI as sg
# ############################# User callable CPU intensive code #############################
# Put your long running code inside this "wrapper"
# NEVER make calls to PySimpleGUI from this thread (or any thread)!
# Create one of these functions for EVERY long-running call you want to make
def long_function_wrapper(work_id, gui_queue):
# LOCATION 1
# this is our "long running function call"
#time.sleep(10) # sleep for a while as a simulation of a long-running computation
x = 0
while True:
print(x)
time.sleep(0.5)
x = x + 1
if x == 5:
break
# at the end of the work, before exiting, send a message back to the GUI indicating end
gui_queue.put('{} ::: done'.format(work_id))
# at this point, the thread exits
return
def the_gui():
gui_queue = queue.Queue() # queue used to communicate between the gui and long-running code
layout = [[sg.Text('Multithreaded Work Example')],
[sg.Text('This is a Test.', size=(25, 1), key='_OUTPUT_')],
[sg.Button('Go'), sg.Button('Exit')], ]
window = sg.Window('Multithreaded Window').Layout(layout)
# --------------------- EVENT LOOP ---------------------
work_id = 0
while True:
event, values = window.Read(timeout=100) # wait for up to 100 ms for a GUI event
if event is None or event == 'Exit':
#sg.PopupAnimated(None)
break
if event == 'Go': # clicking "Go" starts a long running work item by starting thread
window.Element('_OUTPUT_').Update('Starting long work %s'%work_id)
# LOCATION 2
# STARTING long run by starting a thread
thread_id = threading.Thread(target=long_function_wrapper, args=(work_id, gui_queue,), daemon=True)
thread_id.start()
#for i in range(200000):
work_id = work_id+1 if work_id < 19 else 0
#while True:
sg.PopupAnimated(sg.DEFAULT_BASE64_LOADING_GIF, background_color='white', time_between_frames=100)
#if message == None:
#break
# --------------- Read next message coming in from threads ---------------
try:
message = gui_queue.get_nowait() # see if something has been posted to Queue
except queue.Empty: # get_nowait() will get exception when Queue is empty
message = None # nothing in queue so do nothing
# if message received from queue, then some work was completed
if message is not None:
# LOCATION 3
# this is the place you would execute code at ENDING of long running task
# You can check the completed_work_id variable to see exactly which long-running function completed
completed_work_id = int(message[:message.index(' :::')])
sg.PopupAnimated(None)
#window['_GIF_'].update_animation(sg.DEFAULT_BASE64_LOADING_GIF, time_between_frames=100)
#window.read(timeout = 1000)
# if user exits the window, then close the window and exit the GUI func
window.Close()
############################# Main #############################
if __name__ == '__main__':
the_gui()
print('Exiting Program'
)

You've got your call to popup_animated inside of an "if" statement that is only executed once.
You must call popup_animated for every frame you wish to show. It's not spun off as a task that works in the background.
This change to your code will keep the animation going as long as there as background tasks running.
import queue
import threading
import time
import PySimpleGUI as sg
# ############################# User callable CPU intensive code #############################
# Put your long running code inside this "wrapper"
# NEVER make calls to PySimpleGUI from this thread (or any thread)!
# Create one of these functions for EVERY long-running call you want to make
def long_function_wrapper(work_id, gui_queue):
# LOCATION 1
# this is our "long running function call"
# time.sleep(10) # sleep for a while as a simulation of a long-running computation
x = 0
while True:
print(x)
time.sleep(0.5)
x = x + 1
if x == 5:
break
# at the end of the work, before exiting, send a message back to the GUI indicating end
gui_queue.put('{} ::: done'.format(work_id))
# at this point, the thread exits
return
def the_gui():
gui_queue = queue.Queue() # queue used to communicate between the gui and long-running code
layout = [[sg.Text('Multithreaded Work Example')],
[sg.Text('This is a Test.', size=(25, 1), key='_OUTPUT_')],
[sg.Text(size=(25, 1), key='_OUTPUT2_')],
[sg.Button('Go'), sg.Button('Exit')], ]
window = sg.Window('Multithreaded Window').Layout(layout)
# --------------------- EVENT LOOP ---------------------
work_id = 0
while True:
event, values = window.Read(timeout=100) # wait for up to 100 ms for a GUI event
if event is None or event == 'Exit':
# sg.PopupAnimated(None)
break
if event == 'Go': # clicking "Go" starts a long running work item by starting thread
window.Element('_OUTPUT_').Update('Starting long work %s' % work_id)
# LOCATION 2
# STARTING long run by starting a thread
thread_id = threading.Thread(target=long_function_wrapper, args=(work_id, gui_queue,), daemon=True)
thread_id.start()
# for i in range(200000):
work_id = work_id + 1 if work_id < 19 else 0
# while True:
# if message == None:
# break
# --------------- Read next message coming in from threads ---------------
try:
message = gui_queue.get_nowait() # see if something has been posted to Queue
except queue.Empty: # get_nowait() will get exception when Queue is empty
message = None # nothing in queue so do nothing
# if message received from queue, then some work was completed
if message is not None:
# LOCATION 3
# this is the place you would execute code at ENDING of long running task
# You can check the completed_work_id variable to see exactly which long-running function completed
completed_work_id = int(message[:message.index(' :::')])
window.Element('_OUTPUT2_').Update('Finished long work %s' % completed_work_id)
work_id -= 1
if not work_id:
sg.PopupAnimated(None)
if work_id:
sg.PopupAnimated(sg.DEFAULT_BASE64_LOADING_GIF, background_color='white', time_between_frames=100)
# window['_GIF_'].update_animation(sg.DEFAULT_BASE64_LOADING_GIF, time_between_frames=100)
# window.read(timeout = 1000)
# if user exits the window, then close the window and exit the GUI func
window.Close()
############################# Main #############################
if __name__ == '__main__':
the_gui()
print('Exiting Program')

How to exit from a generator at some specific time?

I'm reading tweets from Twitter Streaming API. After connecting to the API, I'm getting a generator.
I'm looping through each tweet received but I want to exit from the iterator, say, at 18PM. After receiving each tweet, I'm checking if it's later than the specified timestamp and stopping.
The issue is that I'm not receiving tweets frequently enough. So, I could receive one at 17:50 and the next one at 19PM. That's when I'll find out that the time has passed and I need to stop.
Is there a way to force the stop at 18PM exactly?
Here's a high-level view of my code:
def getStream(tweet_iter):
for tweet in tweet_iter:
#do stuff
if time_has_passed():
return
tweet_iter = ConnectAndGetStream()
getStream(tweet_iter)

Create a separate thread for the producer and use a Queue to communicate. I also had to use a threading.Event for stopping the producer.
import itertools, queue, threading, time
END_TIME = time.time() + 5 # run for ~5 seconds
def time_left():
return END_TIME - time.time()
def ConnectAndGetStream(): # stub for the real thing
for i in itertools.count():
time.sleep(1)
yield "tweet {}".format(i)
def producer(tweets_queue, the_end): # producer
it = ConnectAndGetStream()
while not the_end.is_set():
tweets_queue.put(next(it))
def getStream(tweets_queue, the_end): # consumer
try:
while True:
tweet = tweets_queue.get(timeout=time_left())
print('Got', tweet)
except queue.Empty:
print('THE END')
the_end.set()
tweets_queue = queue.Queue() # you might wanna use the maxsize parameter
the_end = threading.Event()
producer_thread = threading.Thread(target=producer,
args=(tweets_queue, the_end))
producer_thread.start()
getStream(tweets_queue, the_end)
producer_thread.join()

Your problem could be resolved by splitting the functionality of your design into two separated processes:
A twitter process that acts as wrapper to Twitter API and
A monitor process that is able to terminate the twitter process when the exit time is reached.
The following piece of code prototypes the functionality described above using Python's multiprocessing module:
import multiprocessing as mp
import time
EXIT_TIME = '12:21' #'18:00'
def twitter():
while True:
print 'Twittttttttttt.....'
time.sleep(5)
def get_time():
return time.ctime().split()[3][:5]
if __name__ == '__main__':
# Execute the function as a process
p = mp.Process( target=twitter, args=() )
p.start()
# Monitoring the process p
while True:
print 'Checking the hour...'
if get_time() == EXIT_TIME:
p.terminate()
print 'Current time:', time.ctime()
print 'twitter process has benn terminated...'
break
time.sleep(5)
Of course you can use p.join(TIMEOUT) instead of using the while True loop presented in my example as pointed here.

Here is an example with threading and python scheduler:
import threading
import time
import os
import schedule
def theKillingJob():
print("Kenny and Cartman die!")
os._exit(1)
schedule.every().day.at("18:00").do(theKillingJob,'It is 18:00')
def getStream(tweet_iter):
for tweet in tweet_iter:
#do stuff
def kenny():
while True:
print("Kenny alive..")
schedule.run_pending()
time.sleep(1)
def cartman():
while True:
print("Cartman alive..")
tweet_iter = ConnectAndGetStream()
getStream(tweet_iter)
# You can change whenever you want to check for tweets by changing sleep time here
time.sleep(1)
if __name__ == '__main__':
daemon_kenny = threading.Thread(name='kenny', target=kenny)
daemon_cartman = threading.Thread(name='cartman', target=cartman)
daemon_kenny.setDaemon(True)
daemon_cartman.setDaemon(True)
daemon_kenny.start()
daemon_cartman.start()
daemon_kenny.join()
daemon_cartman.join()

Calling While loop Function in __main__?

Here I have a program which polls the queue for an event, if found it executes an order to a REST API. In addition if an event is found, it prints the current price that I need to use as my stopLoss. This code runs exactly as I would like it to, however, the moment I try and call the function rates() inside the __main__ the program just stops running.
Remove the reference stopLoss = rates() and the program runs great just without a stopLoss, but I need the rate -.001 as my stopLoss.
Code as follows:
import Queue
import threading
import time
import json
import oandapy
from execution import Execution
from settings import STREAM_DOMAIN, API_DOMAIN, ACCESS_TOKEN, ACCOUNT_ID
from strategy import TestRandomStrategy
from streaming import StreamingForexPrices
#polls API for Current Price
def stop():
while True:
oanda = oandapy.API(environment="practice", access_token="xxxxxx")
response = oanda.get_prices(instruments="EUR_USD")
prices = response.get("prices")
asking_price = prices[0].get("ask")
s = asking_price - .001
return s
#Checks for events and executes order
def trade(events, strategy, execution):
while True:
try:
event = events.get(False)
except Queue.Empty:
pass
else:
if event is not None:
if event.type == 'TICK':
strategy.calculate_signals(event)
elif event.type == 'ORDER':
print
execution.execute_order(event)
def rates(events):
while True:
try:
event = events.get(False)
except Queue.Empty:
pass
else:
if event.type == 'TICK':
r = stop()
print r
if __name__ == "__main__":
heartbeat = 0 # Half a second between polling
events = Queue.Queue()
# Trade 1 unit of EUR/USD
instrument = "EUR_USD"
units = 1
stopLoss = rates() #Problem area!!!!!!>>>>>>>>>>>>>>//////////////////////////
prices = StreamingForexPrices(
STREAM_DOMAIN, ACCESS_TOKEN, ACCOUNT_ID,
instrument, events
)
execution = Execution(API_DOMAIN, ACCESS_TOKEN, ACCOUNT_ID)
strategy = TestRandomStrategy(instrument, units, events, stopLoss)
#Threads
trade_thread = threading.Thread(target=trade, args=(events, strategy, execution))
price_thread = threading.Thread(target=prices.stream_to_queue, args=[])
stop_thread = threading.Thread(target=rates, args=(events,))
# Start both threads
trade_thread.start()
price_thread.start()
stop_thread.start()

Okay no answers so far, so I'll try.
Your main problem seems to be, that you don't know how to interchange data between threads.
First the problem with the price.
The loop here:
while True:
oanda = oandapy.API(environment="practice", access_token="xxxxxx")
response = oanda.get_prices(instruments="EUR_USD")
prices = response.get("prices")
asking_price = prices[0].get("ask")
s = asking_price - .001
return s
Has no effect, because return s will automatically break out of it.
So what you need is a shared variable where you store s. You can protect the access to it by using threading.Lock. The easiest way would be to subclass Thread and make s an instance attribute like this (I named it price):
class PricePoller(threading.Thread):
def __init__(self, interval):
super(PricePoller, self).__init__()
# private attribute, will be accessed as property via
# threadsafe getter and setter
self._price = None
# lock guarding access to _price
self._dataLock = threading.Lock()
# polling interval
self.interval = interval
# set this thread as deamon, so it will be killed when
# the main thread dies
self.deamon = True
# create an event that allows us to exit the mainloop
# and terminate the thread safely
self._stopEvent = threading.Event()
def getPrice(self):
# use _dataLock to get threadsafe access to self._price
with self._dataLock:
return self._price
def setPrice(self, price)
# use _dataLock to get threadsafe access to self._price
with self._dataLock:
self._price = price
price = property(getPrice, setPrice, None)
def run(self):
while not self.stopEvent.isSet():
oanda = oandapy.API(environment="practice", access_token="xxxxxx")
response = oanda.get_prices(instruments="EUR_USD")
prices = response.get("prices")
asking_price = prices[0].get("ask")
self.price = asking_price - .001
time.sleep(self.interval) # don't spam the server
def stop(self):
self._stopEvent.set()
It can then be started with:
poller = PricePoller(heartbeat)
poller.start()
And you can get the price with poller.price wherever you want! You can even pass the poller on to other threads if you like.
BUT! If you try to get the price immediately after poller.start() you will certainly get a None. Why this? poller.start() does not block, therefore while your main thread is going on and tries to get the first price, your poller has not even finished starting!
How to solve this? Introduce another threading.Event and use its function wait to let the main thread wait until the poller thread has set it. I leave the implementation up to you.
I'm just guessing that this is what you want... looking only at your code you don't have to put the stop function in a thread at all and you can just replace stopLess = rates() with stopLess = stop(), because you're not updating the results from the price polling anywhere! But I think you want to do that at some point, otherwise it wouldn't make sense to put it into a thread.
Now to the queue and your 'event stream'.
This snippet:
try:
event = events.get(False)
except Queue.Empty:
pass
Can just as well be:
event = events.get()
You're doing nothing in the meantime anyway and it is better to let Queue deal with waiting for an event.
Then, as far as I can see, you have two threads calling Queue.get, but this function will delete the element from the queue after retrieving it! This means whoever obtains the event first, consumes it and the other thread will never see it. But with the above solution for the poller, I think you can get rid of the stop_thread, which also solves that problem.
Now a note on Threads in general.
A thread has its own 'chain' of calls that starts within its run method (or the method which you supply as target if you don't subclass).
That means whatever function is called by run is executed by this thread, and also all functions that are called in turn by this one (and so on). HOWEVER, it is perfectly possible that two threads execute the same function, at the same time! And there is no way to know which thread executes which part of the code at a certain time, if you do not use means of synchronisation (e.g. Events, Locks or Barriers).
This is no problem if all variables used in a called function are local ore were local in the calling function:
def onlyLocal(x, n):
if n == 0:
return x
return onlyLocal(x*2, n-1)
or are exclusively read:
def onlyRead(myarray):
t = time.time()
return t - sum(myarray)
But as soon as you do both read from and write to a variable from multiple threads, you need to secure access to those because otherwise if you pass objects which are known by more than one thread (for example self):
def setPrice(self, price):
self._price = price
or if your function uses variables from an outer scope which is acessed by multiple threads:
def variableFromOutside(y):
global x
x += y
return y
You can never be sure that there isn't a thread(2) changing a variable which you(1) have just read, while you are processing it and before you update it with a then invalid value.
global x ; Thread1 ; Thread2 ;
2 ; y = x ; z = x ;
2 ; y **= 3 ; x = z+1 ;
3 ; x = y-4 ; return ;
4 ; return ; ... ;
This is why you have to secure the access to those variables with locks. With a Lock (l):
global x ; Thread1 ; Thread2 ;
2 ; l.acqcuire() ; l.acquire() ;
2 ; y = x ; | ;
2 ; y **= 3 ; | ;
2 ; x = y-4 ; | ;
4 ; l.release() ; v ;
4 ; return ; z = x ;
4 ; ... ; x = z+1 ;
5 ; ... ; l.release() ;
Here Thread1 acquires the lock before Thread2. Thread2 therefore has to wait until Thread1 releases the lock again before its call to acquire returns.
acquire and release are automatically called when you use with lock:.
Note also that in this toy example it could have been the case that Thread2 aqcuires the lock before Thread1, but at least they would still not interfere with each other.
This was a brief introduction on a large topic, read a bit about thread parallelisation and play around with it. There is no better means for learning than practice.
I've written this code here in the browser and therefore it is not tested! If someone finds issues, please tell me so in the comments or feel free to change it directly.

Trouble with multithreading python

Having a lot of trouble adding an additional thread to this program'. As it sits it accesses an API and executes trades on an account. All fine and good until I try and pass a streaming rate to the stopLoss order. Once I run it this way the code runs a loop blocking out everything else and just prints the latest price. How can I set this thing to run concurrently with the other two threads?
Main trading program:
import Queue
import threading
import time
import json
import streamer
from execution import Execution
from settings import STREAM_DOMAIN, API_DOMAIN, ACCESS_TOKEN, ACCOUNT_ID
from strategy import TestRandomStrategy
from streaming import StreamingForexPrices
from event import TickEvent
from streamer import demo
stop = demo(0)
def trade(events, strategy, execution):
"""
Carries out an infinite while loop that polls the
events queue and directs each event to either the
strategy component of the execution handler. The
loop will then pause for "heartbeat" seconds and
continue.
"""
while True:
try:
event = events.get(False)
except Queue.Empty:
pass
else:
if event is not None:
if event.type == 'TICK':
strategy.calculate_signals(event)
elif event.type == 'ORDER':
print "Executing order!"
execution.execute_order(event)
elif event.type == 'stopLoss':
print "StOP LOSS HERE!!!!"
execution.execute_order(event)
time.sleep(heartbeat)
if __name__ == "__main__":
heartbeat = 0 # Half a second between polling
events = Queue.Queue()
# Trade 1000 unit of EUR/USD
instrument = "EUR_USD"
units = 1
stopLoss = stop
# Create the OANDA market price streaming class
# making sure to provide authentication commands
prices = StreamingForexPrices(
STREAM_DOMAIN, ACCESS_TOKEN, ACCOUNT_ID,
instrument, events
)
#handle stopLoss price
stop = demo(0)
# Create the execution handler making sure to
# provide authentication commands
execution = Execution(API_DOMAIN, ACCESS_TOKEN, ACCOUNT_ID)
# Create the strategy/signal generator, passing the
# instrument, quantity of units and the events queue
strategy = TestRandomStrategy(instrument, units, events, stopLoss)
# Create two separate threads: One for the trading loop
# and another for the market price streaming class
trade_thread = threading.Thread(target=trade, args=(events, strategy, execution))
price_thread = threading.Thread(target=prices.stream_to_queue, args=[])
stop_thread = threading.Thread(target=stop, args=[])
# Start both threads
trade_thread.start()
price_thread.start()
stop_thread.start()
Any help is greatly appreciated!
Edit:
Ok I have drilled down to the problem I think. This code here executes the trades and when I try and post "prices" to the stopLoss order the script gives me error. example:
stopLoss = prices NameError: name 'prices' is not defined
On the other hand, in trying to simplify the problem to post here I solved another one I was having!
code:
def trade(events, strategy, execution):
while True:
prices = demo(0)
print prices
while True:
try:
event = events.get(False)
except Queue.Empty:
pass
else:
if event is not None:
if event.type == 'TICK':
strategy.calculate_signals(event)
elif event.type == 'ORDER':
print "Executing order!"
execution.execute_order(event)
time.sleep(heartbeat)
if __name__ == "__main__":
heartbeat = 0 # Half a second between polling
events = Queue.Queue()
# Trade 1000 unit of EUR/USD
instrument = "EUR_USD"
units = 1
stopLoss = prices