I would like to create a job that rolls to all 10 munites.
I find a good example here. The problem is that the program is freezing during the waiting time and my other urls are blocked.
after me it's because of while True:
Is there a way to do it without going around this problem?
voici le code:
import schedule
import time
def job():
print("I'm working...")
schedule.every(10).minutes.do(job)
while True:
schedule.run_pending()
time.sleep(1)
*******************************************************************.
I found the right way to do it. Here is the link:
For that to work well, I removed this part:
# time.sleep(20)
# print('Checkpoint **************************')
# time.sleep(30)
# print('Bye -----------------------')
Here is the code that works:
import threading
class ThreadingExample(object):
""" Threading example class
The run() method will be started and it will run in the background
until the application exits.
"""
def __init__(self, interval=10):
""" Constructor
:type interval: int
:param interval: Check interval, in seconds
"""
self.interval = interval
thread = threading.Thread(target=self.run, args=())
thread.daemon = True # Daemonize thread
thread.start() # Start the execution
def run(self):
""" Method that runs forever """
while True:
# Do something
print('Doing something imporant in the background', self.interval)
pk_info_semaine = job_temp.objects.all()
for a in pk_info_semaine:
print('num_semaine:',a.num_semaine,'user_id:',a.user_id)
time.sleep(self.interval)
example = ThreadingExample()
Thank you all and thank you to the author: Paris Nakita Kejser Here
You can use celery + celerybeat together with Django to run scheduled tasks. You can write your method as a celery task, and add an entry in your settings.py file to make the task run every 10 minutes. The task will run in its on thread, hence not blocking your application.
voici le link to celery:
http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
Related
I want to execute a task after certain time, so I have tried a countdown timer with a condition of being finished (when countdown variable = 0, the task is performed). The thing is that I don't want to stop the execution of the main program while performing the countdown. I have tried this:
import time
def countdown(num_of_secs):
while(num_of_secs):
time.sleep(1)
num_of_secs -= 1
return num_of_secs
So, I run my code setting a number of seconds to the countdown, and when this countdown reaches the 0 value, a task must be executed. Using this code (it uses a while), when I call my function "countdown" it stops the execution of the main program, so it is the same as a big time.sleep. I want to carry out this countdown in the background, without stopping other actions until the countdown finishes and the task starts.
Thank you
Another alternative is by using threading.
I've got a simple example here with 2 Threads where the working thread is waiting for the countdown thread to finish and starting. The Main is still working fine.
import threading
import time
def do_something():
countdown_thread.join()
print("Starting Task")
time.sleep(3)
print("Finished Task")
def countdown(num_of_secs):
while(num_of_secs):
time.sleep(1)
num_of_secs -= 1
print(num_of_secs)
if __name__ == '__main__':
countdown_thread = threading.Thread(target=countdown, args=(3,))
work_thread = threading.Thread(target=do_something)
countdown_thread.start()
work_thread.start()
while True:
print("Main doing something")
time.sleep(1)
Example picture for multithreading: Sequential vs Threading
Usually python only has a single program flow, so every instruction needs to complete before the next one can get executed.
For your case you need asynchronicity, with e.g. asyncio.sleep(5) as a separate task in the same event loop.
import asyncio
async def sleeper():
print('Holding...')
await asyncio.sleep(5)
print('Doing Work!')
async def work():
print('Doing work')
print('while')
print('the other guy is sleeping')
async def main():
await asyncio.gather(sleeper(), work())
asyncio.run(main())
The most common and easiest way to implement this would be with a Timer object from the threading library. It would go as follows:
import threading
import time
i = 0
done = False
def show_results():
print("results from GPIO readings")
print("=)")
global done
done = True # signal end of while loop
def read_GPIO():
print("reading GPIO...")
t = threading.Timer(60, show_results) # task will trigger after 60 seconds
t.start()
# your while loop would go here
read_GPIO() # do work
while not done:
print("waiting", i) # doing work while waiting for timer
time.sleep(1)
i += 1
pass
Notice that the time library is used only for illustrative purposes. You could also start the timer recursively to check periodically GPIOs and print results or trigger an event. For more information on the threading library or the Timer object check the docs
I have a same thread running every 10 min. but when the new thread starts i want to quit the previous thread so it doesn't keep adding up the space. how can i achieve that. for scheduling of thread.I'm using python schedule library.
this is how I'm scheduling right now
schedule.every(10).minutes.do(sts,threadFunc)
There are two aspects to this question:
identify the currently running job, which is fairly easy.
Kill a running thread in python. There's no great solution for this, and the following code implements the 'stop flag' approach.
I'm solving the first challenge by using a global variable. This variable, named running_thread, holds the currently running thread so that a new job can kill it if needed.
The second challenge requires the running thread to constantly check the status of some flag ('the stop flag'). If the stop flag is set on that thread, it immediately exists.
Here's a code skeleton that demonstrates both these ideas. Jobs take a random amount of time, and I've scheduled them to start every 1 second.
import threading
import time
import schedule
import random
running_thread = None
class StoppableThread(threading.Thread):
"""Thread class with a stop() method. The thread itself has to check
regularly for the stopped() condition."""
def __init__(self, *args, **kwargs):
super(StoppableThread, self).__init__(*args, **kwargs)
self._stop_event = threading.Event()
def stop(self):
self._stop_event.set()
def stopped(self):
return self._stop_event.is_set()
def job():
current_thread = threading.currentThread()
sleep_time = random.random() * 5
print(f"Starting job, about to sleep {sleep_time} seconds, thread id is {current_thread.ident}")
counter = 0
while counter < sleep_time:
time.sleep(0.1)
counter += 0.1
if current_thread.stopped():
print ("Stopping job")
break
print(f"job with thread id {current_thread.ident} done")
def threadFunc():
global running_thread
if running_thread:
print("Trying to stop thread")
running_thread.stop()
print("Strting thread")
running_thread = StoppableThread(target = job)
running_thread.start()
schedule.every(1).seconds.do(threadFunc)
while True:
schedule.run_pending()
time.sleep(.5)
I know that the termination notice is made available via the meta-data url and that I can do something similar to
if requests.get("http://169.254.169.254/latest/meta-data/spot/termination-time").status_code == 200
in order to determine if the notice has been posted. I run a Python service on my Spot Instances that:
Loops over long polling SQS Queues
If it gets a message, it pauses polling and works on the payload.
Working on the payload can take 5-50 minutes.
Working on the payload will involve spawning a threadpool of up to 50 threads to handle parallel uploading of files to S3, this is the majority of the time spent working on the payload.
Finally, remove the message from the queue, rinse, repeat.
The work is idempotent, so if the same payload runs multiple times, I'm out the processing time/costs, but will not negatively impact the application workflow.
I'm searching for an elegant way to now also poll for the termination notice every five seconds in the background. As soon as the termination notice appears, I'd like to immediately release the message back to the SQS queue in order for another instance to pick it up as quickly as possible.
As a bonus, I'd like to shutdown the work, kill off the threadpool, and have the service enter a stasis state. If I terminate the service, supervisord will simply start it back up again.
Even bigger bonus! Is there not a python module available that simplifies this and just works?
I wrote this code to demonstrate how a thread can be used to poll for the Spot instance termination. It first starts up a polling thread, which would be responsible for checking the http endpoint.
Then we create pool of fake workers (mimicking real work to be done) and starts running the pool. Eventually the polling thread will kick in (about 10 seconds into execution as implemented) and kill the whole thing.
To prevent the script from continuing to work after Supervisor restarts it, we would simply put a check at the beginning of the __main__ and if the termination notice is there we sleep for 2.5 minutes, which is longer than that notice lasts before the instance is shutdown.
#!/usr/bin/env python
import threading
import Queue
import random
import time
import sys
import os
class Instance_Termination_Poll(threading.Thread):
"""
Sleep for 5 seconds and eventually pretend that we then recieve the
termination event
if requests.get("http://169.254.169.254/latest/meta-data/spot/termination-time").status_code == 200
"""
def run(self):
print("Polling for termination")
while True:
for i in range(30):
time.sleep(5)
if i==2:
print("Recieve Termination Poll!")
print("Pretend we returned the message to the queue.")
print("Now Kill the entire program.")
os._exit(1)
print("Well now, this is embarassing!")
class ThreadPool:
"""
Pool of threads consuming tasks from a queue
"""
def __init__(self, num_threads):
self.num_threads = num_threads
self.errors = Queue.Queue()
self.tasks = Queue.Queue(self.num_threads)
for _ in range(num_threads):
Worker(self.tasks, self.errors)
def add_task(self, func, *args, **kargs):
"""
Add a task to the queue
"""
self.tasks.put((func, args, kargs))
def wait_completion(self):
"""
Wait for completion of all the tasks in the queue
"""
try:
while True:
if self.tasks.empty() == False:
time.sleep(10)
else:
break
except KeyboardInterrupt:
print "Ctrl-c received! Kill it all with Prejudice..."
os._exit(1)
self.tasks.join()
class Worker(threading.Thread):
"""
Thread executing tasks from a given tasks queue
"""
def __init__(self, tasks, error_queue):
threading.Thread.__init__(self)
self.tasks = tasks
self.daemon = True
self.errors = error_queue
self.start()
def run(self):
while True:
func, args, kargs = self.tasks.get()
try:
func(*args, **kargs)
except Exception, e:
print("Exception " + str(e))
error = {'exception': e}
self.errors.put(error)
self.tasks.task_done()
def do_work(n):
"""
Sleeps a random ammount of time, then creates a little CPU usage to
mimic some work taking place.
"""
for z in range(100):
time.sleep(random.randint(3,10))
print "Thread ID: {} working.".format(threading.current_thread())
for x in range(30000):
x*n
print "Thread ID: {} done, sleeping.".format(threading.current_thread())
if __name__ == '__main__':
num_threads = 30
# Start up the termination polling thread
term_poll = Instance_Termination_Poll()
term_poll.start()
# Create our threadpool
pool = ThreadPool(num_threads)
for y in range(num_threads*2):
pool.add_task(do_work, n=y)
# Wait for the threadpool to complete
pool.wait_completion()
I am trying to implement a heartbeat call that works in the background. How do I create a threaded on interval call of say every 30 seconds, which calls the following function:
self.mqConn.heartbeat_tick()
Also how would I stop this thread?
Many thanks.
Use a thread containing a loop
from threading import Thread
import time
def background_task():
while not background_task.cancelled:
self.mqConn.heartbeat_tick()
time.sleep(30)
background_task.cancelled = False
t = Thread(target=background_task)
t.start()
background_task.cancelled = True
Alternatively, you could subclass timer, to make cancelling easy:
from threading import Timer
class RepeatingTimer(Timer):
def run(self):
while not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.wait(self.interval)
t = RepeatingTimer(30.0, self.mqConn.heartbeat_tick)
t.start() # every 30 seconds, call heartbeat_tick
# later
t.cancel() # cancels execution
Or you could use the Timer class in the threading module:
from threading import Timer
def hello():
print "hello, world"
t = Timer(30.0, hello)
t.start() # after 30 seconds, "hello, world" will be printed
t.cancel() # cancels execution, this only works before the 30 seconds is elapsed
This will not start every x seconds, rather it delays the thread for execution in x seconds. But you can still put that in a loop and use t.is_alive() to see its status.
A quick followup to Eric's answer: you can't subclass Timer in python 2, since it's actually a light function wrapper around the true class: _Timer. If you do you'll get the issue that pops up in this post.
Using _Timer instead fixes it:
from threading import _Timer
class RepeatingTimer(_Timer):
def run(self):
while not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.wait(self.interval)
t = RepeatingTimer(30.0, self.mqConn.heartbeat_tick)
t.start() # every 30 seconds, call heartbeat_tick
# later
t.cancel() # cancels execution
One way to do this would be to use the circuits application framework like this:
from circuits import Component, Event, Timer
class App(Component):
def init(self, mqConn):
self.mqConn = mqConn
Timer(30, Event.create("heartbeat"), persist=True).register(self)
def heartbeat(self):
self.mqConn.heartbeat_tick()
App().run()
Note: I'm the author of circuits :)
This is just a basic idea and structure -- You would need to adapt this to suit your exact application and requirements!
I have simple PyGTK app. Since I have to run multiple periodic tasks to fetch some data and refresh GUI, I extended Thread like this:
class MyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.setDaemon(True)
self.event = threading.Event()
self.event.set()
def run(self):
while self.event.is_set():
timer = threading.Timer(60, self._run)
timer.start()
timer.join()
def cancel(self):
self.event.clear()
def _run(self):
gtk.threads_enter()
# do what need to be done, fetch data, update GUI
gtk.threads_leave()
I start threads on app bootstrap, save them in some list and cancel them before exit. This works just perfect.
But now I want to add refresh button which will force one of the threads to run immediately and not wait period of time to be run, if not currently running.
I tried to do that by adding bool var to MyThread to indicate whether a thread is running or not (set before _run, reset on complete), and then just call MyThread._run() if not running, but that causes my app to become unresponsive and _run task to never finish execution.
I'm not sure why this happens. What is the best way to solve this problem? It would be also fine if I can make refresh running in background so it does not block GUI.
Maybe to call run and pass in number of seconds to 1 so timer can trigger it sooner?
Instead of using a Timer, use another Event object in combination with a timeout. You can then set that event from within your button callback. The following code illustrates this (I've stripped your cancelling code to keep it short):
import threading
class MyThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.sleep_event = threading.Event()
self.damon = True
def run(self):
while True:
self.sleep_event.clear()
self.sleep_event.wait(60)
threading.Thread(target=self._run).start()
def _run(self):
print "run"
my_thread = MyThread()
my_thread.start()
while True:
raw_input("Hit ENTER to force execution\n")
my_thread.sleep_event.set()
By default "run" will be printed every 60 seconds. If you hit ENTER it will be printed immediately, and then again after 60 seconds, etc.