I have a set of instructions, say {I} and I would like to perform this set {I}
at predefined time for instance each minute.
I'm not asking how to insert a delay of 1 minutes between to successive executions of
the set {I}, I want to start the instructions {I} each minute independently of the time of execution of {I}.
If I inderstand the following code
import time
while True:
{I}
time.sleep(60)
would simply insert a delay of 60 secs between the end of the execution of {I} and the following one. Is it true? Instead I would like that the set of instructions {I} starts each minute (for instance at 9.00 am, 9.01 am, 9.02 am, etc).
Is it possible to perform such a task inside python, or is it preferable to write a script with {I} that I execute each minutes, for instance, with Crontab?
Thank you in advance
Looks like signal.alarm and signal.signal(signal.SIGALRM, handler) should help you.
If you don't need finer resolution than a minute, cron would be the easiest option. Otherwise you'd end up re-writing something like it.
If you need intervals shorter than a minute, you might consider "timeouts" from the glib library. It has Python bindings. The timeout should then probably start the task in a separate process.
Something like APScheduler might meet your needs.
I'm sure there are other similar packages out there as well.
Chances are, you'd have to instantiate separate threads for every instruction to be run concurrently, and simply dispatch them in your delayed while loop.
You could spawn a thread every second using threading.Timer:
import threading
import time
def do_stuff(count):
print(count)
if c < 10: # Let's build in some way to quit
t = threading.Timer(1.0, do_stuff, args=[count+1])
t.start()
t = threading.Timer(0.0, do_stuff, args=[0])
t.start()
t.join()
Using the sched module is another possibility, but note that the sched.scheduler.run method blocks the main process until the event queue is empty. (So if the do_stuff function takes longer than a second, the next event won't run on time.) If you want nonblocking events, use threading.Timer.
Related
Maybe it's a very simple question, but I'm new in concurrency. I want to do a python script to run foo.py 10 times simultaneously with a time limit of 60 sec before automatically abort. The script is a non deterministic algorithm, hence all executions takes different times and one will be finished before the others. Once the first ends, I would like to save the execution time, the output of the algorithm and after that kill the rest of the processes.
I have seen this question run multiple instances of python script simultaneously and it looks very similar, but how can I add time limit and the possibility of when the first one finishes the execution, kills the rest of processes?
Thank you in advance.
I'd suggest using the threading lib, because with it you can set threads to daemon threads so that if the main thread exits for whatever reason the other threads are killed. Here's a small example:
#Import the libs...
import threading, time
#Global variables... (List of results.)
results=[]
#The subprocess you want to run several times simultaneously...
def run():
#We declare results as a global variable.
global results
#Do stuff...
results.append("Hello World! These are my results!")
n=int(input("Welcome user, how much times should I execute run()? "))
#We run the thread n times.
for _ in range(n):
#Define the thread.
t=threading.Thread(target=run)
#Set the thread to daemon, this means that if the main process exits the threads will be killed.
t.setDaemon(True)
#Start the thread.
t.start()
#Once the threads have started we can execute tha main code.
#We set a timer...
startTime=time.time()
while True:
#If the timer reaches 60 s we exit from the program.
if time.time()-startTime>=60:
print("[ERROR] The script took too long to run!")
exit()
#Do stuff on your main thread, if the stuff is complete you can break from the while loop as well.
results.append("Main result.")
break
#When we break from the while loop we print the output.
print("Here are the results: ")
for i in results:
print(f"-{i}")
This example should solve your problem, but if you wanted to use blocking commands on the main thread the timer would fail, so you'd need to tweak this code a bit. If you wanted to do that move the code from the main thread's loop to a new function (for example def main(): and execute the rest of the threads from a primary thread on main. This example may help you:
def run():
pass
#Secondary "main" thread.
def main():
#Start the rest of the threads ( in this case I just start 1).
localT=threading.Thread(target=run)
localT.setDaemon(True)
localT.start()
#Do stuff.
pass
#Actual main thread...
t=threading.Thread(target=main)
t.setDaemon(True)
t.start()
#Set up a timer and fetch the results you need with a global list or any other method...
pass
Now, you should avoid global variables at all costs as sometimes they may be a bit buggy, but for some reason the threading lib doesn't allow you to return values from threads, at least i don't know any methods. I think there are other multi-processing libs out there that do let you return values, but I don't know anything about them so I can't explain you anything. Anyways, I hope that this works for you.
-Update: Ok, I was busy writing the code and I didn't read the comments in the post, sorry. You can still use this method but instead of writing code inside the threads, execute another script. You could either import it as a module or actually run it as a script, here's a question that may help you with that:
How to run one python file in another file?
I want to run a function at the start of every minute, without it lagging over time. Using time.sleep(60) eventually lags.
while True:
now = datetime.datetime.now().second
if now == 0:
print(datetime.datetime.now())
The function doesn't take a minute to run so as long as it runs a the beginning it should be fine, I'm not sure if this code is resource-efficient, as its checking every millisecond or so and even if it drifts the if function should correct it.
Repeat scheduling shouldn't really be done in python, especially by using time.sleep. The best way would be to get your OS to schedule running the script, using something like cron if you're on Linux or Task Scheduler if you're on Windows
Assuming that you've examined and discarded operating-based solutions such as cron or Windows Scheduled Tasks, what you suggest will work but you're right in that it's CPU intensive. You would be better off sleeping for one second after each check so that:
It's less resource intensive; and, more importantly
It doesn't execute multiple times per at the start of each minute if the job takes less than a second.
In fact, you could sleep for even longer immediately after the payload by checking how long to the next minute, and use the minute to decide in case the sleep takes you into a second that isn't zero. Something like this may be a good start:
# Ensure we do it quickly first time.
lastMinute = datetime.datetime.now().minute - 1
# Loop forever.
while True:
# Get current time, do payload if new minute.
thisTime = datetime.datetime.now()
if thisTime.minute != lastMinute:
doPayload()
lastMinute = thisTime.minute
# Try to get close to hh:mm:55 (slow mode).
# If payload took more than 55s, just go
# straight to fast mode.
afterTime = datetime.datetime.now()
if afterTime.minute == thisTime.minute:
if afterTime.second < 55:
time.sleep (55 - afterTime.second)
# After hh:mm:55, check every second (fast mode).
time.sleep(1)
I write a lot of code that relies on precise periodic method calls. I've been using Python's futures library to submit calls onto the runtime's thread pool and sleeping between calls in a loop:
executor = ThreadPoolExecutor(max_workers=cpu_count())
def remote_call():
# make a synchronous bunch of HTTP requests
def loop():
while True:
# do work here
executor.submit(remote_call)
time.sleep(60*5)
However, I've noticed that this implementation introduces some drift after a long duration of running (e.g. I've run this code for about 10 hours and noticed about 7 seconds of drift). For my work I need this to run on the exact second, and millisecond would be even better. Some folks have pointed me to asyncio ("Fire and forget" python async/await), but I have not been able to get this working in Python 2.7.
I'm not looking for a hack. What I really want is something akin to Go's time.Tick or Netty's HashedWheelTimer.
Nothing like that comes with Python. You'd need to manually adjust your sleep times to account for time spent working.
You could fold that into an iterator, much like the channel of Go's time.Tick:
import itertools
import time
import timeit
def tick(interval, initial_wait=False):
# time.perf_counter would probably be more appropriate on Python 3
start = timeit.default_timer()
if not initial_wait:
# yield immediately instead of sleeping
yield
for i in itertools.count(1):
time.sleep(start + i*interval - timeit.default_timer())
yield
for _ in tick(300):
# Will execute every 5 minutes, accounting for time spent in the loop body.
do_stuff()
Note that the above ticker starts ticking when you start iterating, rather than when you call tick, which matters if you try to start a ticker and save it for later. Also, it doesn't send the time, and it won't drop ticks if the receiver is slow. You can adjust all that on your own if you want.
Is it possible to have 2, 3 or more threads in Python to be able to execute something simultaneously - at the exact same moment? Is it possible if one of the threads is late, for the other to be waiting for it, so the last request can be executed at the same time?
Example: There are two threads that are calculating specific parameters, after they have done that they need to click one button at the same time (to send post request to the server).
"Exact the same time" is really difficult, at almost the same time is possible but you need to use multiprocessing instead of threads. Here one example.
from time import time
from multiprocessing import Pool
def f(*args):
while time() < start + 5: #syncronize the execution of each process
pass
print(time())
start = time()
with Pool(10) as p:
p.map(f, range(10))
It prints
1495552973.6672032
1495552973.6672032
1495552973.669514
1495552973.667697
1495552973.6672032
1495552973.668086
1495552973.6693969
1495552973.6672032
1495552973.6677089
1495552973.669164
Note that some of the processes are really simultaneous (in the 10e-7 second precision). It's impossible to guarantee that all the processes will be executed at the very same moment.
However, if you limitate the number of processes to the number of core you actually have, then most of the time they will run exactly at the same moment.
I have struggled with this question for about a week -- time to ask someone who can bang out an answer in a couple minutes.
I am trying to run a python program once every 10 seconds. There are a lot of questions of this sort : Use sched module to run at a given time, Python threading.timer - repeat function every 'n' seconds, How to execute a function asynchronously every 60 seconds in Python?
Normally the solutions using sched or time.sleep would work, but I am trying to start a scheduled process from within cmd2, which is already running in a while False loop. (When you exit cmd2, it exits this loop).
Because of this, when I start a function to repeat every 10 seconds, I enter another loop nested within cmd2 and I am unable to enter cmd2 commands. I can only get back to cmd2 by exiting the sub-loop that is repeating the function, and thus the function stops repeating.
Evidently threading will solve this problem. I have tried threading.Timer without success. Perhaps the real problem is that I do not understand threads or multiprocessing.
Here is an example of code that is roughly isomorphic to the code I'm using, using sched module, which I got to work:
import cmd2
import repeated
class prompt(cmd2.Cmd):
"""this lets you enter commands"""
def default(self, line):
return cmd2.Cmd.default(self, line)
def do_exit(self, line):
return True
def do_repeated(self, line):
repeated.func_1()
Where repeated.py looks like this:
import sched
import time
def func_2(sc):
print 'doing stuff'
sc.enter(10, 0, func_2, (sc,))
def func_1():
s = sched.scheduler(time.time, time.sleep)
s.enter(0, 0, func_2, (s,))
s.run()
http://docs.python.org/2/library/queue.html?highlight=queue#Queue
Can you instance a Queue object outside of cmd2? There can be one thread that watches the queue and takes jobs from it at periodic intervals; while cmd2 is free to run or not run. The thread that processes the queue, and the queue object itself need to be in the outer scope, of course.
To schedule something at a particular time, you can insert a tuple which has the target time in it. Or you can have the thread just check at regular intervals, if that's good enough.
[Edit, if you have a process that is intended to repeat, you can have it requeue itself at the end of it's operation.]
As soon as I asked the question I was able to figure it out. Don't know why that happens sometimes.
This code
def f():
# do something here ...
# call f() again in 60 seconds
threading.Timer(60, f).start()
# start calling f now and every 60 sec thereafter
f()
From here: How to execute a function asynchronously every 60 seconds in Python?
Actually works for what I was trying to do. There are evidently some subtleties in how the function is called as an argument in threading.Timer. Before when I was including the arguments and even the parentheses after the function I was getting recursive depth errors --i.e. the function was calling itself without delay constantly.
So anyone else who has a problem like this, pay attention to how you call the function in threading.Timer(60, f).start(). If you write threading.Timer(60, f()).start() or something similar it will probably not work.