We Keep Coding

Python sleep without blocking other processes

I am running a python script every hour and I've been using time.sleep(3600) inside of a while loop. It seems to work as needed but I am worried about it blocking new tasks. My research of this seems to be that it only blocks the current thread but I want to be 100% sure. While the hourly job shouldn't take more than 15min, if it does or if it hangs, I don't want it to block the next one that starts. This is how I've done it:
import threading
import time
def long_hourly_job():
# do some long task
pass
if __name__ == "__main__":
while True:
thr = threading.Thread(target=long_hourly_job)
thr.start()
time.sleep(3600)
Is this sufficient?
Also, the reason i am using time.sleep for this hourly job rather than a cron job is I want to do everything in code to make dockerization cleaner.

The code will work (ie: sleep does only block the calling thread), but you should be careful of some issues. Some of them have been already stated in the comments, like the possibility of time overlaps between threads. The main issue is that your code is slowly leaking resources. After creating a thread, the OS keeps some data structures even after the thread has finished running. This is necessary, for example to keep the thread's exit status until the thread's creator requires it. The function to clear these structures (conceptually equivalent to closing a file) is called join. A thread that has finished running and is not joined is termed a 'zombie thread'. The amount of memory required by these structures is very small, and your program should run for centuries for any reasonable amount of available RAM. Nevertheless, it is a good practice to join all the threads you create. A simple approach (if you know that 3600 s is more than enough time for the thread to finish) would be:
if __name__ == "__main__":
while True:
thr = threading.Thread(target=long_hourly_job)
thr.start()
thr.join(3600) # wait at most 3600 s for the thread to finish
if thr.isAlive(): # join does not return useful information
print("Ooops: the last job did not finish on time")
A better approach if you think that it is possible that sometimes 3600 s is not enough time for the thread to finish:
if __name__ == "__main__":
previous = []
while True:
thr = threading.Thread(target=long_hourly_job)
thr.start()
previous.append(thr)
time.sleep(3600)
for i in reversed(range(len(previous))):
t = previous[i]
t.join(0)
if t.isAlive():
print("Ooops: thread still running")
else:
print("Thread finished")
previous.remove(t)
I know that the print statement makes no sense: use logging instead.

Perhaps a little late. I tested the code from other answers but my main process got stuck (perhaps I'm doing something wrong?). I then tried a different approach. It's based on threading Timer class, but trying to emulate the QtCore.QTimer() behavior and features:
import threading
import time
import traceback
class Timer:
SNOOZE = 0
ONEOFF = 1
def __init__(self, timerType=SNOOZE):
self._timerType = timerType
self._keep = threading.Event()
self._timerSnooze = None
self._timerOneoff = None
class _SnoozeTimer(threading.Timer):
# This uses threading.Timer class, but consumes more CPU?!?!?!
def __init__(self, event, msec, callback, *args):
threading.Thread.__init__(self)
self.stopped = event
self.msec = msec
self.callback = callback
self.args = args
def run(self):
while not self.stopped.wait(self.msec):
self.callback(*self.args)
def start(self, msec: int, callback, *args, start_now=False) -> bool:
started = False
if msec > 0:
if self._timerType == self.SNOOZE:
if self._timerSnooze is None:
self._timerSnooze = self._SnoozeTimer(self._keep, msec / 1000, callback, *args)
self._timerSnooze.start()
if start_now:
callback(*args)
started = True
else:
if self._timerOneoff is None:
self._timerOneoff = threading.Timer(msec / 1000, callback, *args)
self._timerOneoff.start()
started = True
return started
def stop(self):
if self._timerType == self.SNOOZE:
self._keep.set()
self._timerSnooze.join()
else:
self._timerOneoff.cancel()
self._timerOneoff.join()
def is_alive(self):
if self._timerType == self.SNOOZE:
isAlive = self._timerSnooze is not None and self._timerSnooze.is_alive() and not self._keep.is_set()
else:
isAlive = self._timerOneoff is not None and self._timerOneoff.is_alive()
return isAlive
isAlive = is_alive
KEEP = True
def callback():
global KEEP
KEEP = False
print("ENDED", time.strftime("%M:%S"))
if __name__ == "__main__":
count = 0
t = Timer(timerType=Timer.ONEOFF)
t.start(5000, callback)
print("START", time.strftime("%M:%S"))
while KEEP:
if count % 10000000 == 0:
print("STILL RUNNING")
count += 1
Notice the while loop runs in a separate thread, and uses a callback function to invoke when the time is over (in your case, this callback function would be used to check if the long running process has finished).

Updating Popup.Animated to play gif until external task is completed (PYSimpleGUI)

I am looking to create a UI that displays an animated popup while another task is being carried out. That will exit upon completion. I am using PYSimpleGUI and am using the example listed here to base my work off. I can get a single frame of the animation to display once I start the code and exit upon completion of the task but can't get it to play the entire gif. Code:
import queue
import threading
import time
import PySimpleGUI as sg
# ############################# User callable CPU intensive code #############################
# Put your long running code inside this "wrapper"
# NEVER make calls to PySimpleGUI from this thread (or any thread)!
# Create one of these functions for EVERY long-running call you want to make
def long_function_wrapper(work_id, gui_queue):
# LOCATION 1
# this is our "long running function call"
#time.sleep(10) # sleep for a while as a simulation of a long-running computation
x = 0
while True:
print(x)
time.sleep(0.5)
x = x + 1
if x == 5:
break
# at the end of the work, before exiting, send a message back to the GUI indicating end
gui_queue.put('{} ::: done'.format(work_id))
# at this point, the thread exits
return
def the_gui():
gui_queue = queue.Queue() # queue used to communicate between the gui and long-running code
layout = [[sg.Text('Multithreaded Work Example')],
[sg.Text('This is a Test.', size=(25, 1), key='_OUTPUT_')],
[sg.Button('Go'), sg.Button('Exit')], ]
window = sg.Window('Multithreaded Window').Layout(layout)
# --------------------- EVENT LOOP ---------------------
work_id = 0
while True:
event, values = window.Read(timeout=100) # wait for up to 100 ms for a GUI event
if event is None or event == 'Exit':
#sg.PopupAnimated(None)
break
if event == 'Go': # clicking "Go" starts a long running work item by starting thread
window.Element('_OUTPUT_').Update('Starting long work %s'%work_id)
# LOCATION 2
# STARTING long run by starting a thread
thread_id = threading.Thread(target=long_function_wrapper, args=(work_id, gui_queue,), daemon=True)
thread_id.start()
#for i in range(200000):
work_id = work_id+1 if work_id < 19 else 0
#while True:
sg.PopupAnimated(sg.DEFAULT_BASE64_LOADING_GIF, background_color='white', time_between_frames=100)
#if message == None:
#break
# --------------- Read next message coming in from threads ---------------
try:
message = gui_queue.get_nowait() # see if something has been posted to Queue
except queue.Empty: # get_nowait() will get exception when Queue is empty
message = None # nothing in queue so do nothing
# if message received from queue, then some work was completed
if message is not None:
# LOCATION 3
# this is the place you would execute code at ENDING of long running task
# You can check the completed_work_id variable to see exactly which long-running function completed
completed_work_id = int(message[:message.index(' :::')])
sg.PopupAnimated(None)
#window['_GIF_'].update_animation(sg.DEFAULT_BASE64_LOADING_GIF, time_between_frames=100)
#window.read(timeout = 1000)
# if user exits the window, then close the window and exit the GUI func
window.Close()
############################# Main #############################
if __name__ == '__main__':
the_gui()
print('Exiting Program'
)

You've got your call to popup_animated inside of an "if" statement that is only executed once.
You must call popup_animated for every frame you wish to show. It's not spun off as a task that works in the background.
This change to your code will keep the animation going as long as there as background tasks running.
import queue
import threading
import time
import PySimpleGUI as sg
# ############################# User callable CPU intensive code #############################
# Put your long running code inside this "wrapper"
# NEVER make calls to PySimpleGUI from this thread (or any thread)!
# Create one of these functions for EVERY long-running call you want to make
def long_function_wrapper(work_id, gui_queue):
# LOCATION 1
# this is our "long running function call"
# time.sleep(10) # sleep for a while as a simulation of a long-running computation
x = 0
while True:
print(x)
time.sleep(0.5)
x = x + 1
if x == 5:
break
# at the end of the work, before exiting, send a message back to the GUI indicating end
gui_queue.put('{} ::: done'.format(work_id))
# at this point, the thread exits
return
def the_gui():
gui_queue = queue.Queue() # queue used to communicate between the gui and long-running code
layout = [[sg.Text('Multithreaded Work Example')],
[sg.Text('This is a Test.', size=(25, 1), key='_OUTPUT_')],
[sg.Text(size=(25, 1), key='_OUTPUT2_')],
[sg.Button('Go'), sg.Button('Exit')], ]
window = sg.Window('Multithreaded Window').Layout(layout)
# --------------------- EVENT LOOP ---------------------
work_id = 0
while True:
event, values = window.Read(timeout=100) # wait for up to 100 ms for a GUI event
if event is None or event == 'Exit':
# sg.PopupAnimated(None)
break
if event == 'Go': # clicking "Go" starts a long running work item by starting thread
window.Element('_OUTPUT_').Update('Starting long work %s' % work_id)
# LOCATION 2
# STARTING long run by starting a thread
thread_id = threading.Thread(target=long_function_wrapper, args=(work_id, gui_queue,), daemon=True)
thread_id.start()
# for i in range(200000):
work_id = work_id + 1 if work_id < 19 else 0
# while True:
# if message == None:
# break
# --------------- Read next message coming in from threads ---------------
try:
message = gui_queue.get_nowait() # see if something has been posted to Queue
except queue.Empty: # get_nowait() will get exception when Queue is empty
message = None # nothing in queue so do nothing
# if message received from queue, then some work was completed
if message is not None:
# LOCATION 3
# this is the place you would execute code at ENDING of long running task
# You can check the completed_work_id variable to see exactly which long-running function completed
completed_work_id = int(message[:message.index(' :::')])
window.Element('_OUTPUT2_').Update('Finished long work %s' % completed_work_id)
work_id -= 1
if not work_id:
sg.PopupAnimated(None)
if work_id:
sg.PopupAnimated(sg.DEFAULT_BASE64_LOADING_GIF, background_color='white', time_between_frames=100)
# window['_GIF_'].update_animation(sg.DEFAULT_BASE64_LOADING_GIF, time_between_frames=100)
# window.read(timeout = 1000)
# if user exits the window, then close the window and exit the GUI func
window.Close()
############################# Main #############################
if __name__ == '__main__':
the_gui()
print('Exiting Program')

Threading multiple functions in python

I've created a program that streams through twitter and based on the result of generated by the tweets it plays music using the pygame library. Below is a sample of my code.
class listener(StreamListener):
def on_status(self, status):
global mood_happy, mood_sad, mood_angry, mood_shocked, mood_romantic
try:
# print status
tweet_text = status.text
for mood_n_score in [[happy, 'mood_happy'], [sad, 'mood_sad'], [angry, 'mood_angry'],
[shocked, 'mood_shocked'], [romantic, 'mood_romantic']]:
lst_mood = mood_n_score[0]
type_mood = mood_n_score[1]
for mood in lst_mood:
if mood in tweet_text:
if type_mood == 'mood_happy':
mood_happy += 1
elif type_mood == 'mood_sad':
mood_sad += 1
elif type_mood == 'mood_angry':
mood_angry += 1
elif type_mood == 'mood_shocked':
mood_shocked += 1
else:
mood_romantic += 1
break
print('\n----------------')
print 'mood_happy:', mood_happy
print 'mood_sad:', mood_sad
print 'mood_angry:', mood_angry
print 'mood_shocked:', mood_shocked
print 'mood_romantic:', mood_romantic
top_mood=max(mood_happy,mood_sad,mood_angry,mood_shocked,mood_romantic)
if top_mood==mood_happy:
print "the mood is: happy"
pygame.mixer.music.load(file.mp3)
pygame.mixer.music.play()
As you can see, I have a streamer class which streams through twitter continously and prints the top mood. When I run my code to play the mp3 file, the streaming stops and only the music plays. How can I make my program stream through twitter and play music at the same time?
Thank you!

I have never used pygame, but based on what it does I think I can probably assume it's not thread-safe.
What I would do is have the streaming code in a thread using the threading module and have the music playing logic constantly wait on the main thread for a threading.Event to be set.
import threading
import pygame
new_mood_event = threading.Event()
class TwitterStreamer(StreamListener):
def run(self):
while True: # keep the streamer going forever
pass # define your code here
def on_status(self, status):
# ... Define your code here
if top_mood == mood_happy:
new_mood_event.mp3_file_path = 'happy_file.mp3'
new_mood_event.set() # alert the main thread we have a new mood to play
if __name__ == '__main__':
twitter_streamer = TwitterStreamer()
streaming_thread = threading.Thread(target=twitter_streamer.run) # creates a thread that will call `twitter_streamer.run()` when executed
streaming_thread.start() # starts the thread
# everything from here will be run in the main thread
while True: # creates an "event loop"
new_mood_event.wait() # blocks the main thread until `new_mood_event.set()` is called by `on_status`
new_mood_event.clear() # clears the event. if we don't clear the event, then `new_mood_event.wait()` will only block once
pygame.mixer.music.load(new_mood_event.mp3_file_path)
pygame.mixer.music.play()

How to exit from a generator at some specific time?

I'm reading tweets from Twitter Streaming API. After connecting to the API, I'm getting a generator.
I'm looping through each tweet received but I want to exit from the iterator, say, at 18PM. After receiving each tweet, I'm checking if it's later than the specified timestamp and stopping.
The issue is that I'm not receiving tweets frequently enough. So, I could receive one at 17:50 and the next one at 19PM. That's when I'll find out that the time has passed and I need to stop.
Is there a way to force the stop at 18PM exactly?
Here's a high-level view of my code:
def getStream(tweet_iter):
for tweet in tweet_iter:
#do stuff
if time_has_passed():
return
tweet_iter = ConnectAndGetStream()
getStream(tweet_iter)

Create a separate thread for the producer and use a Queue to communicate. I also had to use a threading.Event for stopping the producer.
import itertools, queue, threading, time
END_TIME = time.time() + 5 # run for ~5 seconds
def time_left():
return END_TIME - time.time()
def ConnectAndGetStream(): # stub for the real thing
for i in itertools.count():
time.sleep(1)
yield "tweet {}".format(i)
def producer(tweets_queue, the_end): # producer
it = ConnectAndGetStream()
while not the_end.is_set():
tweets_queue.put(next(it))
def getStream(tweets_queue, the_end): # consumer
try:
while True:
tweet = tweets_queue.get(timeout=time_left())
print('Got', tweet)
except queue.Empty:
print('THE END')
the_end.set()
tweets_queue = queue.Queue() # you might wanna use the maxsize parameter
the_end = threading.Event()
producer_thread = threading.Thread(target=producer,
args=(tweets_queue, the_end))
producer_thread.start()
getStream(tweets_queue, the_end)
producer_thread.join()

Your problem could be resolved by splitting the functionality of your design into two separated processes:
A twitter process that acts as wrapper to Twitter API and
A monitor process that is able to terminate the twitter process when the exit time is reached.
The following piece of code prototypes the functionality described above using Python's multiprocessing module:
import multiprocessing as mp
import time
EXIT_TIME = '12:21' #'18:00'
def twitter():
while True:
print 'Twittttttttttt.....'
time.sleep(5)
def get_time():
return time.ctime().split()[3][:5]
if __name__ == '__main__':
# Execute the function as a process
p = mp.Process( target=twitter, args=() )
p.start()
# Monitoring the process p
while True:
print 'Checking the hour...'
if get_time() == EXIT_TIME:
p.terminate()
print 'Current time:', time.ctime()
print 'twitter process has benn terminated...'
break
time.sleep(5)
Of course you can use p.join(TIMEOUT) instead of using the while True loop presented in my example as pointed here.

Here is an example with threading and python scheduler:
import threading
import time
import os
import schedule
def theKillingJob():
print("Kenny and Cartman die!")
os._exit(1)
schedule.every().day.at("18:00").do(theKillingJob,'It is 18:00')
def getStream(tweet_iter):
for tweet in tweet_iter:
#do stuff
def kenny():
while True:
print("Kenny alive..")
schedule.run_pending()
time.sleep(1)
def cartman():
while True:
print("Cartman alive..")
tweet_iter = ConnectAndGetStream()
getStream(tweet_iter)
# You can change whenever you want to check for tweets by changing sleep time here
time.sleep(1)
if __name__ == '__main__':
daemon_kenny = threading.Thread(name='kenny', target=kenny)
daemon_cartman = threading.Thread(name='cartman', target=cartman)
daemon_kenny.setDaemon(True)
daemon_cartman.setDaemon(True)
daemon_kenny.start()
daemon_cartman.start()
daemon_kenny.join()
daemon_cartman.join()

Concurrent Threads in Python

I am having issues getting 3 threads to run concurrently. I would like to have the "trade" loop, the "prices"loop and the "stop" loop run at the same time however it seems that the "stop" loop hijacks the program and runs while the others wait their turn. How should I set it up so that they all run at the same time?
import Queue
import threading
import time
import json
from execution import Execution
from settings import STREAM_DOMAIN, API_DOMAIN, ACCESS_TOKEN, ACCOUNT_ID
from strategy import TestRandomStrategy
from streaming import StreamingForexPrices
from event import TickEvent
from rates import stop
def trade(events, strategy, execution):
"""
Carries out an infinite while loop that polls the
events queue and directs each event to either the
strategy component of the execution handler. The
loop will then pause for "heartbeat" seconds and
continue.
"""
while True:
try:
event = events.get(False)
except Queue.Empty:
pass
else:
if event is not None:
if event.type == 'TICK':
strategy.calculate_signals(event)
elif event.type == 'ORDER':
print "Executing order!"
execution.execute_order(event)
time.sleep(heartbeat)
if __name__ == "__main__":
heartbeat = 0 # Half a second between polling
events = Queue.Queue()
# Trade 1000 unit of EUR/USD
instrument = "EUR_USD"
units = 1
stopLoss = stopper
# Create the OANDA market price streaming class
# making sure to provide authentication commands
prices = StreamingForexPrices(
STREAM_DOMAIN, ACCESS_TOKEN, ACCOUNT_ID,
instrument, events
)
#handle stopLoss price
stopper = stop()
# Create the execution handler making sure to
# provide authentication commands
execution = Execution(API_DOMAIN, ACCESS_TOKEN, ACCOUNT_ID)
# Create the strategy/signal generator, passing the
# instrument, quantity of units and the events queue
strategy = TestRandomStrategy(instrument, units, events, stopLoss)
# Create two separate threads: One for the trading loop
# and another for the market price streaming class
trade_thread = threading.Thread(target=trade, args=(events, strategy, execution))
price_thread = threading.Thread(target=prices.stream_to_queue, args=[])
rate_thread = threading.Thread(target=stop, args=[])
# Start both threads
trade_thread.start()
price_thread.start()
rate_thread.start()
Just fyi, everything worked great until I tried to add the "rate". The only things I have added are an additional thread, the stopLoss and the rate.py file.
rate.py:
import oandapy
import time
oanda = oandapy.API(environment="practice", access_token="xxxxxxxxx")
while True:
response = oanda.get_prices(instruments="EUR_USD")
prices = response.get("prices")
asking_price = prices[0].get("ask")
stop = asking_price - .001
print stop
time.sleep(1)
Thanks for the help in advance!

First of all, a remark:
don't use sleep if you can avoid it; for example, in the "trade" loop you
don't need sleep at all if you make a blocking .get() on your queue
Then, once the "rates.py" is imported it starts the while loop; you're
missing the stop() function (or your code is not complete ?)
EDIT: in case you want to add the stop function in rates.py, just put
the while loop code inside a def stop(): block like this
def stop():
while True:
response = oanda.get_prices(instruments="EUR_USD")
prices = response.get("prices")
asking_price = prices[0].get("ask")
stop = asking_price - .001
print stop
time.sleep(1)
(btw: do you really know what you're doing?)