How can I schedule a periodic task in python without blocking?
Here is a simple situation. Assume this ticket variable becomes invalid after 2 hours. So I need to fetch it from a serve.
ticket = 1 # It expires every 2 hours
def process_using_ticket(): # This function is called using a get request from flask server
print('hello', ticket)
How can I reset to 1 every two hours without blocking?
One way could be to start a thread and sleep for 2 hours and then reset the variable but I am wondering if there are better alternatives.
Note: Everything runs in a docker.
This looks like a use case for cron, a simple utility found on all linux machines to schedule periodic tasks. A simple cron task could be created like:
$ crontab -e
Within the cron, make an entry
0 */2 * * * /home/username/ticket_script.py
Make sure that your script has executable permissions. In case you are printing something in your script, make the cron entry to redirect its ouput like
0 */2 * * * /home/username/ticket_script.py >> /home/username/ticket_script_runs.log
For gotchas on running this in Docker, you can go through this thread -> https://forums.docker.com/t/cronjobs-in-docker-container/2618/5 which says:
Most Docker containers only run the binary you specify (in this case /bin/bash), so there is no cron daemon running to trigger these jobs.
There are a number of ways you can deal with this - for small adhoc things, you can add the cron entry to your host's crontab, and trigger the job in the container using docker exec containername ...
You can replace the container entrypoint with a shell script that starts the cron, or if you have a need for several services, use something like supervisord to manage them.
And if you're going all in, you can make a cron container which you can then use to kick off tasks using docker exec.
The other approach is what you already have - go to sleep for 2 hours, or maintain a time_last_read timestamp, keep evaluating if its been 2 hours since your last read (time_now - time_last_read >= 2_HOURS), and re-read the value into memory once the condition is True and reset time_last_read.
Put it in a function with time.sleep(7200000). Like so:
import time
ticket = 1
def main_loop():
while True:
time.sleep(7200000)
ticket = 1
def process_using_ticket():
print('hello', ticket)
main_loop()
Related
I have a bunch of .py scripts as part of a project. Some of them i want to start and have running in the background whilst the others run through what they need to do.
For example, I have a script which takes a Screenshot every 10 seconds until the script is closed and i wish to have this running in the background whilst the other scripts get called and run through till finish.
Another example is a script which calculates the hash of every file in a designated folder. This has the potential to run for a fair amount of time so it would be good if the rest of the scripts could be kicked off at the same time so they do not have to wait for the Hash script to finish what it is doing before they are invoked.
Is Multiprocessor the right method for this kind of processing, or is there another way to achieve these results which would be better such as this answer: Run multiple python scripts concurrently
You could also use something like Celery to run the tasks async and you'll be able to call tasks from within your python code instead of through the shell.
It depends. With multiprocessing you can create a process manager, so it can spawn the processes the way you want, but there are more flexible ways to do it without coding. Multiprocessing is usually hard.
Check out circus, it's a process manager written in Python that you can use as a library, standalone or via remote API. You can define hooks to model dependencies between processes, see docs.
A simple configuration could be:
[watcher:one-shot-script]
cmd = python script.py
numprocesses = 1
warmup_delay = 30
[watcher:snapshots]
cmd = python snapshots.py
numprocesses = 1
warmup_delay = 30
[watcher:hash]
cmd = python hashing.py
numprocesses = 1
I want to run a program at a certain time.
For example, When I run the program, it will be executed at 5 p.m
How can I implement the above function using Python?
As pointed out in the comments this has nothing to do with python and rather with your OS.
So if you are on Linux run the command crontab -e and enter the lines:
* 9 * * 0-5 python /path/to/your/python/script
* 17 * * 6 python /path/to/your/python/script
The first line is for 9am every day except on saturday and the second one is for 17pm saturday.
If you are on Windows press the windows button + R, then type: taskschd.msc which opens the task scheduler. Here press "Action" and "Create task..." which opens a new windows which lets you create a new task, the instructions there is self explanatory.
Within Python, thus within a long/permanent running (service) process of your own, you can use sched.scheduler or threading.Timer from the standard lib or use custom code (in a thread).
For starting a python script in a timed manner ordinarly as a user use OS means (crontab [*nix], Scheduled Tasks [Windows], ...) or use 3rd party scheduling apps - as you would do for scheduling other programs.
A python program could control those schedulers via API (e.g. win32com.taskscheduler) or command-line/popen...
If you want to do it with python you could use Celery beats, although not recommended if you're not using celery.
On *nix systems you just want to use cron jobs instead and on windows use the task scheduler.
I am looking to run a Python script every n days as part of a Django app.
I need this script to add new rows to the database for each attribute that a user has.
For example a User has many Sites each that have multiple Metric data points. I need to add a new Metric data point for each Site every n days by running a .py script.
I have already written the script itself, but it just works locally.
Is it the right approach to:
1) Get something like Celery or just a simple cron task running to run the python script every n days.
2) Have the python script run through all the Metrics for each Site and add a new data point by executing a SQL command from the python script itself?
You can use Django's manage commad, then use crontab to run this command circularly.
The first step is to write the script - which you have already done. This script should run the task which is to be repeated.
The second step is to schedule the execution.
The de-facto way of doing that is to write a cron entry for the script. This will let the system's cron daemon trigger the script at the interval you select.
To do so, create an entry in the crontab file for the user that owns the script; the command to do so is crontab -e, which will load a plain text file in your preferred editor.
Next, you need to add an entry to this file. The format is:
minutes hours day-of-month month day-of-week script-to-execute
So if you want to run your script every 5 days:
0 0 */5 * * /bin/bash /home/user/path/to/python.py
I have a small python script that basically connects to a SQL Server (Micrsoft) database and gets users from there, and then syncs them to another mysql database, basically im just running queries to check if the user exists, if not, then add that user to the mysql database.
The script usually would take around 1 min to sync. I require the script to do its work every 5 mins (for example) exactly once (one sync per 5 mins).
How would be the best way to go about building this?
I have some test data for the users but on the real site, theres a lot more users so I can't guarantee the script takes 1 min to execute, it might even take 20 mins. However having an interval of say 15 mins everytime the script executes would be ideal for the problem...
Update:
I have the connection params for the sql server windows db, so I'm using a small ubuntu server to sync between the two databases located on different servers. So lets say db1 (windows) and db2 (linux) are the database servers, I'm using s1 (python server) and pymssql and mysql python modules to sync.
Regards
I am not sure cron is right for the job. It seems to me that if you have it run every 15 minutes but sometimes a synch takes 20 minutes you could have multiple processes running at once and possibly collide.
If the driving force is a constant wait time between the variable execution times then you might need a continuously running process with a wait.
def main():
loopInt = 0
while(loopInt < 10000):
synchDatabase()
loopInt += 1
print("call #" + str(loopInt))
time.sleep(300) #sleep 5 minutes
main()
(obviously not continuous, but long running) You can set the result of while to true and it will be continuous. (comment out loopInt += 1)
Edited to add: Please see note in comments about monitoring the process as you don't want the script to hang or crash and you not be aware of it.
You might want to use a system that handles queues, for example RabbitMQ, and use Celery as the python interface to implement it. With Celery, you can add tasks (like execution of a script) to a queue or run a schedule that'll perform a task after a given interval (just like cron).
Get started http://celery.readthedocs.org/en/latest/
I have a shell script that I want to run automatically every day at 08 AM, and I am not authorised to use the crontab because I don't have root permission
My home directory is /home/user1/.
Any suggestions?
Ideally you should have your system administrator add your user account to /etc/cron.allow - without that you do not have permission to use the crontab command, as you probably discovered.
If that is not possible, then you could use a wrapper shell script along these lines (note: untested):
#!/bin/bash
TIME="tomorrow 8am"
while :; do
# Get the current time in seconds since the Epoch
CURRENT=$(date +%s)
# Get the next start time in seconds
NEXT=$(date +%s -d "$TIME")
# Sleep for the intervening time
sleep $((NEXT-CURRENT))
# Execute the command to repeat
/home/user1/mycommand.sh
done
You start the wrapper script in the background, or e.g. in a screen session, and as long as it's active it will regularly execute your script. Keep in mind that:
Much like cron there is no real accuracy w.r.t. the start time. If you care about timing to the second, then this is not the way to go.
If the script is interrupted for whatever reason, such as a server reboot, you will have to somehow restart. Normally I'd suggest an #reboot entry in crontab, but that seems to not be an option for you.
If there is some sort of process-cleaning mechanism that kills long-term user processed you are probably out of luck.
Your system administrator may have simply neglected to allow users access to cron - or it may have been an explicit decision. In the second case they might not take too well to you leaving a couple of processes overnight in order to bypass that restriction.
Even if you dont have root permission you can set cron job. Chcek these 2 commands as user1, if you can modify it or its throwing any error.
crontab -l
If you can see then try this as well:
crontab -e
If you can open and edit, then you can run that script with cron.
by adding this line:
* 08 * * * /path/to/your/script
I don't think root permission is required to create a cron job. Editing a cronjob that's not owned by you - there's where you'd need root.
In a pinch, you can use at(1). Make sure the program you run reschedules the at job. Warning: this goes to heck if the machine is down for any length of time.