Scheduling Python Script to run every hour accurately - python

Before I ask, Cron Jobs and Task Scheduler will be my last options, this script will be used across Windows and Linux and I'd prefer to have a coded out method of doing this than leaving this to the end user to complete.
Is there a library for Python that I can use to schedule tasks? I will need to run a function once every hour, however, over time if I run a script once every hour and use .sleep, "once every hour" will run at a different part of the hour from the previous day due to the delay inherent to executing/running the script and/or function.
What is the best way to schedule a function to run at a specific time of day (more than once) without using a Cron Job or scheduling it with Task Scheduler?
Or if this is not possible, I would like your input as well.
AP Scheduler fit my needs exactly.
Version < 3.0
import datetime
import time
from apscheduler.scheduler import Scheduler
# Start the scheduler
sched = Scheduler()
sched.daemonic = False
sched.start()
def job_function():
print("Hello World")
print(datetime.datetime.now())
time.sleep(20)
# Schedules job_function to be run once each minute
sched.add_cron_job(job_function, minute='0-59')
out:
>Hello World
>2014-03-28 09:44:00.016.492
>Hello World
>2014-03-28 09:45:00.0.14110
Version > 3.0
(From Animesh Pandey's answer below)
from apscheduler.schedulers.blocking import BlockingScheduler
sched = BlockingScheduler()
#sched.scheduled_job('interval', seconds=10)
def timed_job():
print('This job is run every 10 seconds.')
#sched.scheduled_job('cron', day_of_week='mon-fri', hour=10)
def scheduled_job():
print('This job is run every weekday at 10am.')
sched.configure(options_from_ini_file)
sched.start()

Maybe this can help: Advanced Python Scheduler
Here's a small piece of code from their documentation:
from apscheduler.schedulers.blocking import BlockingScheduler
def some_job():
print "Decorated job"
scheduler = BlockingScheduler()
scheduler.add_job(some_job, 'interval', hours=1)
scheduler.start()

To run something every 10 minutes past the hour.
from datetime import datetime, timedelta
while 1:
print 'Run something..'
dt = datetime.now() + timedelta(hours=1)
dt = dt.replace(minute=10)
while datetime.now() < dt:
time.sleep(1)

For apscheduler < 3.0, see Unknown's answer.
For apscheduler > 3.0
from apscheduler.schedulers.blocking import BlockingScheduler
sched = BlockingScheduler()
#sched.scheduled_job('interval', seconds=10)
def timed_job():
print('This job is run every 10 seconds.')
#sched.scheduled_job('cron', day_of_week='mon-fri', hour=10)
def scheduled_job():
print('This job is run every weekday at 10am.')
sched.configure(options_from_ini_file)
sched.start()
Update:
apscheduler documentation.
This for apscheduler-3.3.1 on Python 3.6.2.
"""
Following configurations are set for the scheduler:
- a MongoDBJobStore named “mongo”
- an SQLAlchemyJobStore named “default” (using SQLite)
- a ThreadPoolExecutor named “default”, with a worker count of 20
- a ProcessPoolExecutor named “processpool”, with a worker count of 5
- UTC as the scheduler’s timezone
- coalescing turned off for new jobs by default
- a default maximum instance limit of 3 for new jobs
"""
from pytz import utc
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
from apscheduler.executors.pool import ProcessPoolExecutor
"""
Method 1:
"""
jobstores = {
'mongo': {'type': 'mongodb'},
'default': SQLAlchemyJobStore(url='sqlite:///jobs.sqlite')
}
executors = {
'default': {'type': 'threadpool', 'max_workers': 20},
'processpool': ProcessPoolExecutor(max_workers=5)
}
job_defaults = {
'coalesce': False,
'max_instances': 3
}
"""
Method 2 (ini format):
"""
gconfig = {
'apscheduler.jobstores.mongo': {
'type': 'mongodb'
},
'apscheduler.jobstores.default': {
'type': 'sqlalchemy',
'url': 'sqlite:///jobs.sqlite'
},
'apscheduler.executors.default': {
'class': 'apscheduler.executors.pool:ThreadPoolExecutor',
'max_workers': '20'
},
'apscheduler.executors.processpool': {
'type': 'processpool',
'max_workers': '5'
},
'apscheduler.job_defaults.coalesce': 'false',
'apscheduler.job_defaults.max_instances': '3',
'apscheduler.timezone': 'UTC',
}
sched_method1 = BlockingScheduler() # uses overrides from Method1
sched_method2 = BlockingScheduler() # uses same overrides from Method2 but in an ini format
#sched_method1.scheduled_job('interval', seconds=10)
def timed_job():
print('This job is run every 10 seconds.')
#sched_method2.scheduled_job('cron', day_of_week='mon-fri', hour=10)
def scheduled_job():
print('This job is run every weekday at 10am.')
sched_method1.configure(jobstores=jobstores, executors=executors, job_defaults=job_defaults, timezone=utc)
sched_method1.start()
sched_method2.configure(gconfig=gconfig)
sched_method2.start()

the simplest option I can suggest is using the schedule library.
In your question, you said "I will need to run a function once every hour"
the code to do this is very simple:
import schedule
def thing_you_wanna_do():
...
...
return
schedule.every().hour.do(thing_you_wanna_do)
while True:
schedule.run_pending()
you also asked how to do something at a certain time of the day
some examples of how to do this are:
import schedule
def thing_you_wanna_do():
...
...
return
schedule.every().day.at("10:30").do(thing_you_wanna_do)
schedule.every().monday.do(thing_you_wanna_do)
schedule.every().wednesday.at("13:15").do(thing_you_wanna_do)
# If you would like some randomness / variation you could also do something like this
schedule.every(1).to(2).hours.do(thing_you_wanna_do)
while True:
schedule.run_pending()
90% of the code used is the example code of the schedule library. Happy scheduling!

Run the script every 15 minutes of the hour.
For example, you want to receive 15 minute stock price quotes, which are updated every 15 minutes.
while True:
print("Update data:", datetime.now())
sleep = 15 - datetime.now().minute % 15
if sleep == 15:
run_strategy()
time.sleep(sleep * 60)
else:
time.sleep(sleep * 60)

#For scheduling task execution
import schedule
import time
def job():
print("I'm working...")
schedule.every(1).minutes.do(job)
#schedule.every().hour.do(job)
#schedule.every().day.at("10:30").do(job)
#schedule.every(5).to(10).minutes.do(job)
#schedule.every().monday.do(job)
#schedule.every().wednesday.at("13:15").do(job)
#schedule.every().minute.at(":17").do(job)
while True:
schedule.run_pending()
time.sleep(1)

The Python standard library does provide sched and threading for this task. But this means your scheduler script will have be running all the time instead of leaving its execution to the OS, which may or may not be what you want.

On the version posted by sunshinekitty called "Version < 3.0" , you may need to specify apscheduler 2.1.2 . I accidentally had version 3 on my 2.7 install, so I went:
pip uninstall apscheduler
pip install apscheduler==2.1.2
It worked correctly after that. Hope that helps.

clock.py
from apscheduler.schedulers.blocking import BlockingScheduler
import pytz
sched = BlockingScheduler(timezone=pytz.timezone('Africa/Lagos'))
#sched.scheduled_job('cron', day_of_week='mon-sun', hour=22)
def scheduled_job():
print('This job is run every week at 10pm.')
#your job here
sched.start()
Procfile
clock: python clock.py
requirements.txt
APScheduler==3.0.0
After deployment, the final step is to scale up the clock process. This is a singleton process, meaning you’ll never need to scale up more than 1 of these processes. If you run two, the work will be duplicated.
$ heroku ps:scale clock=1
Source: https://devcenter.heroku.com/articles/clock-processes-python

Perhaps Rocketry suits your needs. It's a powerful scheduler that is very easy to use, has a lot of built-in scheduling options and it is easy to extend:
from rocketry import Rocketry
from rocketry.conds import daily, every, after_success
app = Rocketry()
#app.task(every("1 hour 30 minutes"))
def do_things():
...
#app.task(daily.between("12:00", "17:00"))
def do_daily_afternoon():
...
#app.task(daily & after_success(do_things))
def do_daily_after_task():
...
if __name__ == "__main__":
app.run()
It has much more though:
String based scheduling syntax
Logical statements (AND, OR, NOT)
A lot of built-in scheduling options
Easy to customize (custom conditions, parameters etc.)
Parallelization (run on separate thread or process)
Paramatrization (execution order and input-output)
Persistence: put the logs anywhere you like
Modify scheduler on runtime (ie. build API on top of it)
Links:
Documentation: https://rocketry.readthedocs.io/
Source code: https://github.com/Miksus/rocketry
Disclaimer: I'm the author

Probably you got the solution already #lukik, but if you wanna remove a scheduling, you should use:
job = scheduler.add_job(myfunc, 'interval', minutes=2)
job.remove()
or
scheduler.add_job(myfunc, 'interval', minutes=2, id='my_job_id')
scheduler.remove_job('my_job_id')
if you need to use a explicit job ID
For more information, you should check: https://apscheduler.readthedocs.io/en/stable/userguide.html#removing-jobs

I found that scheduler needs to run the program every second. If using a online server it would be costly.
So I have following:
It run at each minute at the 5th second, and you can change it to hours days by recalculating waiting period in seconds
import time
import datetime
Initiating = True
print(datetime.datetime.now())
while True:
if Initiating == True:
print("Initiate")
print( datetime.datetime.now())
time.sleep(60 - time.time() % 60+5)
Initiating = False
else:
time.sleep(60)
print("working")
print(datetime.datetime.now())

This method worked for me using relativedelta and datetime and a modulo boolean check for every hour.
It runs every hour from the time you start it.
import time
from datetime import datetime, timedelta
from dateutil.relativedelta import relativedelta
#Track next run outside loop and update the next run time within the loop
nxt_run=datetime.now()
#because while loops evaluate at microseconds we basically need to use a boolean evaluation to track when it should run next
while True:
cnow = datetime.now() #track the current time
time.sleep(1) #good to have so cpu doesn't spike
if (cnow.hour % 1 == 0 and cnow >= nxt_run):
print(f"start #{cnow}: next run #{nxt_run}")
nxt_run=cnow+relativedelta(hours=1) #add an hour to the next run
else:
print(f"next run #{nxt_run}")

One option is to write a C/C++ wrapper that executes the python script on a regular basis. Your end-user would run the C/C++ executable, which would remain running in the background, and periodically execute the python script. This may not be the best solution, and may not work if you don't know C/C++ or want to keep this 100% python. But it does seem like the most user-friendly approach, since people are used to clicking on executables. All of this assumes that python is installed on your end user's computer.
Another option is to use cron job/Task Scheduler but to put it in the installer as a script so your end user doesn't have to do it.

Related

Python script scheduling [duplicate]

Before I ask, Cron Jobs and Task Scheduler will be my last options, this script will be used across Windows and Linux and I'd prefer to have a coded out method of doing this than leaving this to the end user to complete.
Is there a library for Python that I can use to schedule tasks? I will need to run a function once every hour, however, over time if I run a script once every hour and use .sleep, "once every hour" will run at a different part of the hour from the previous day due to the delay inherent to executing/running the script and/or function.
What is the best way to schedule a function to run at a specific time of day (more than once) without using a Cron Job or scheduling it with Task Scheduler?
Or if this is not possible, I would like your input as well.
AP Scheduler fit my needs exactly.
Version < 3.0
import datetime
import time
from apscheduler.scheduler import Scheduler
# Start the scheduler
sched = Scheduler()
sched.daemonic = False
sched.start()
def job_function():
print("Hello World")
print(datetime.datetime.now())
time.sleep(20)
# Schedules job_function to be run once each minute
sched.add_cron_job(job_function, minute='0-59')
out:
>Hello World
>2014-03-28 09:44:00.016.492
>Hello World
>2014-03-28 09:45:00.0.14110
Version > 3.0
(From Animesh Pandey's answer below)
from apscheduler.schedulers.blocking import BlockingScheduler
sched = BlockingScheduler()
#sched.scheduled_job('interval', seconds=10)
def timed_job():
print('This job is run every 10 seconds.')
#sched.scheduled_job('cron', day_of_week='mon-fri', hour=10)
def scheduled_job():
print('This job is run every weekday at 10am.')
sched.configure(options_from_ini_file)
sched.start()
Maybe this can help: Advanced Python Scheduler
Here's a small piece of code from their documentation:
from apscheduler.schedulers.blocking import BlockingScheduler
def some_job():
print "Decorated job"
scheduler = BlockingScheduler()
scheduler.add_job(some_job, 'interval', hours=1)
scheduler.start()
To run something every 10 minutes past the hour.
from datetime import datetime, timedelta
while 1:
print 'Run something..'
dt = datetime.now() + timedelta(hours=1)
dt = dt.replace(minute=10)
while datetime.now() < dt:
time.sleep(1)
For apscheduler < 3.0, see Unknown's answer.
For apscheduler > 3.0
from apscheduler.schedulers.blocking import BlockingScheduler
sched = BlockingScheduler()
#sched.scheduled_job('interval', seconds=10)
def timed_job():
print('This job is run every 10 seconds.')
#sched.scheduled_job('cron', day_of_week='mon-fri', hour=10)
def scheduled_job():
print('This job is run every weekday at 10am.')
sched.configure(options_from_ini_file)
sched.start()
Update:
apscheduler documentation.
This for apscheduler-3.3.1 on Python 3.6.2.
"""
Following configurations are set for the scheduler:
- a MongoDBJobStore named “mongo”
- an SQLAlchemyJobStore named “default” (using SQLite)
- a ThreadPoolExecutor named “default”, with a worker count of 20
- a ProcessPoolExecutor named “processpool”, with a worker count of 5
- UTC as the scheduler’s timezone
- coalescing turned off for new jobs by default
- a default maximum instance limit of 3 for new jobs
"""
from pytz import utc
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
from apscheduler.executors.pool import ProcessPoolExecutor
"""
Method 1:
"""
jobstores = {
'mongo': {'type': 'mongodb'},
'default': SQLAlchemyJobStore(url='sqlite:///jobs.sqlite')
}
executors = {
'default': {'type': 'threadpool', 'max_workers': 20},
'processpool': ProcessPoolExecutor(max_workers=5)
}
job_defaults = {
'coalesce': False,
'max_instances': 3
}
"""
Method 2 (ini format):
"""
gconfig = {
'apscheduler.jobstores.mongo': {
'type': 'mongodb'
},
'apscheduler.jobstores.default': {
'type': 'sqlalchemy',
'url': 'sqlite:///jobs.sqlite'
},
'apscheduler.executors.default': {
'class': 'apscheduler.executors.pool:ThreadPoolExecutor',
'max_workers': '20'
},
'apscheduler.executors.processpool': {
'type': 'processpool',
'max_workers': '5'
},
'apscheduler.job_defaults.coalesce': 'false',
'apscheduler.job_defaults.max_instances': '3',
'apscheduler.timezone': 'UTC',
}
sched_method1 = BlockingScheduler() # uses overrides from Method1
sched_method2 = BlockingScheduler() # uses same overrides from Method2 but in an ini format
#sched_method1.scheduled_job('interval', seconds=10)
def timed_job():
print('This job is run every 10 seconds.')
#sched_method2.scheduled_job('cron', day_of_week='mon-fri', hour=10)
def scheduled_job():
print('This job is run every weekday at 10am.')
sched_method1.configure(jobstores=jobstores, executors=executors, job_defaults=job_defaults, timezone=utc)
sched_method1.start()
sched_method2.configure(gconfig=gconfig)
sched_method2.start()
the simplest option I can suggest is using the schedule library.
In your question, you said "I will need to run a function once every hour"
the code to do this is very simple:
import schedule
def thing_you_wanna_do():
...
...
return
schedule.every().hour.do(thing_you_wanna_do)
while True:
schedule.run_pending()
you also asked how to do something at a certain time of the day
some examples of how to do this are:
import schedule
def thing_you_wanna_do():
...
...
return
schedule.every().day.at("10:30").do(thing_you_wanna_do)
schedule.every().monday.do(thing_you_wanna_do)
schedule.every().wednesday.at("13:15").do(thing_you_wanna_do)
# If you would like some randomness / variation you could also do something like this
schedule.every(1).to(2).hours.do(thing_you_wanna_do)
while True:
schedule.run_pending()
90% of the code used is the example code of the schedule library. Happy scheduling!
Run the script every 15 minutes of the hour.
For example, you want to receive 15 minute stock price quotes, which are updated every 15 minutes.
while True:
print("Update data:", datetime.now())
sleep = 15 - datetime.now().minute % 15
if sleep == 15:
run_strategy()
time.sleep(sleep * 60)
else:
time.sleep(sleep * 60)
#For scheduling task execution
import schedule
import time
def job():
print("I'm working...")
schedule.every(1).minutes.do(job)
#schedule.every().hour.do(job)
#schedule.every().day.at("10:30").do(job)
#schedule.every(5).to(10).minutes.do(job)
#schedule.every().monday.do(job)
#schedule.every().wednesday.at("13:15").do(job)
#schedule.every().minute.at(":17").do(job)
while True:
schedule.run_pending()
time.sleep(1)
The Python standard library does provide sched and threading for this task. But this means your scheduler script will have be running all the time instead of leaving its execution to the OS, which may or may not be what you want.
On the version posted by sunshinekitty called "Version < 3.0" , you may need to specify apscheduler 2.1.2 . I accidentally had version 3 on my 2.7 install, so I went:
pip uninstall apscheduler
pip install apscheduler==2.1.2
It worked correctly after that. Hope that helps.
clock.py
from apscheduler.schedulers.blocking import BlockingScheduler
import pytz
sched = BlockingScheduler(timezone=pytz.timezone('Africa/Lagos'))
#sched.scheduled_job('cron', day_of_week='mon-sun', hour=22)
def scheduled_job():
print('This job is run every week at 10pm.')
#your job here
sched.start()
Procfile
clock: python clock.py
requirements.txt
APScheduler==3.0.0
After deployment, the final step is to scale up the clock process. This is a singleton process, meaning you’ll never need to scale up more than 1 of these processes. If you run two, the work will be duplicated.
$ heroku ps:scale clock=1
Source: https://devcenter.heroku.com/articles/clock-processes-python
Perhaps Rocketry suits your needs. It's a powerful scheduler that is very easy to use, has a lot of built-in scheduling options and it is easy to extend:
from rocketry import Rocketry
from rocketry.conds import daily, every, after_success
app = Rocketry()
#app.task(every("1 hour 30 minutes"))
def do_things():
...
#app.task(daily.between("12:00", "17:00"))
def do_daily_afternoon():
...
#app.task(daily & after_success(do_things))
def do_daily_after_task():
...
if __name__ == "__main__":
app.run()
It has much more though:
String based scheduling syntax
Logical statements (AND, OR, NOT)
A lot of built-in scheduling options
Easy to customize (custom conditions, parameters etc.)
Parallelization (run on separate thread or process)
Paramatrization (execution order and input-output)
Persistence: put the logs anywhere you like
Modify scheduler on runtime (ie. build API on top of it)
Links:
Documentation: https://rocketry.readthedocs.io/
Source code: https://github.com/Miksus/rocketry
Disclaimer: I'm the author
Probably you got the solution already #lukik, but if you wanna remove a scheduling, you should use:
job = scheduler.add_job(myfunc, 'interval', minutes=2)
job.remove()
or
scheduler.add_job(myfunc, 'interval', minutes=2, id='my_job_id')
scheduler.remove_job('my_job_id')
if you need to use a explicit job ID
For more information, you should check: https://apscheduler.readthedocs.io/en/stable/userguide.html#removing-jobs
I found that scheduler needs to run the program every second. If using a online server it would be costly.
So I have following:
It run at each minute at the 5th second, and you can change it to hours days by recalculating waiting period in seconds
import time
import datetime
Initiating = True
print(datetime.datetime.now())
while True:
if Initiating == True:
print("Initiate")
print( datetime.datetime.now())
time.sleep(60 - time.time() % 60+5)
Initiating = False
else:
time.sleep(60)
print("working")
print(datetime.datetime.now())
This method worked for me using relativedelta and datetime and a modulo boolean check for every hour.
It runs every hour from the time you start it.
import time
from datetime import datetime, timedelta
from dateutil.relativedelta import relativedelta
#Track next run outside loop and update the next run time within the loop
nxt_run=datetime.now()
#because while loops evaluate at microseconds we basically need to use a boolean evaluation to track when it should run next
while True:
cnow = datetime.now() #track the current time
time.sleep(1) #good to have so cpu doesn't spike
if (cnow.hour % 1 == 0 and cnow >= nxt_run):
print(f"start #{cnow}: next run #{nxt_run}")
nxt_run=cnow+relativedelta(hours=1) #add an hour to the next run
else:
print(f"next run #{nxt_run}")
One option is to write a C/C++ wrapper that executes the python script on a regular basis. Your end-user would run the C/C++ executable, which would remain running in the background, and periodically execute the python script. This may not be the best solution, and may not work if you don't know C/C++ or want to keep this 100% python. But it does seem like the most user-friendly approach, since people are used to clicking on executables. All of this assumes that python is installed on your end user's computer.
Another option is to use cron job/Task Scheduler but to put it in the installer as a script so your end user doesn't have to do it.

Is there a way to schedule a python script to import files with modified times before 12am and then between 12am and 12pm?

I trying to import newer files twice a day at 12am and 12pm. I am using the following format;
now = datetime.now()
day_start = now.replace(hour=0, minute=0, second=0, microsecond=0)
day_end = day_start + timedelta(hours=12)
This will only capture one time. Is there a way to do this differently?
You could keep the script running the whole day by just putting the program to sleep for 12 hours
Put your task in an infinite loop with an input-based exit as follows
import datetime as dt
# Initialize to Start of Day
LAST_TASK_TIME = dt.datetime.combine(dt.date.today(), dt.time(0))
while True:
if (dt.datetime.now() - LAST_TASK_TIME).total_seconds() / 3600 >= 12:
do_task()
LAST_TASK_TIME += dt.timedelta(hours=12)
if input("Do you want to exit [Y / N] : ").strip().upper()[0] == 'Y':
break
else:
sleep(43195) # Put the program to sleep for 12 hours (-5 seconds)
post_exit_code_if_required()
This code will now run whatever code you want, once every 12 hours along with asking for a continuation confirmation before waiting for the next task time
If you do not want this waiting loop to interrupt your program flow, run this very code in a thread for parallel execution with some other main task (for example, a GUI Application on __main__)
For production ready tool that is pure python, I would use airflow. Airflow is pure Python pipelines orchestral tool that will allows building of pipelines/dags/jobs with beautiful UI
to see how the tasks are doing, plus aREST-APIs to interact with tasks from other tools.
For simple things, there multiple libraries: examples:
You could use schedule. It’s available via python -m pip install --user schedule
import schedule
import time
def file_loader():
print("I'm loading files...")
schedule.every().day.at("12:00").do(file_loader)
schedule.every().day.at("0:00").do(file_loader)
while True:
schedule.run_pending()
time.sleep(1)
Anothe tool is apscheduler also in PyPI pip install apscheduler
from pytz import utc
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore
from apscheduler.executors.pool import ProcessPoolExecutor
jobstores = {
'mongo': {'type': 'mongodb'},
'default': SQLAlchemyJobStore(url='sqlite:///jobs.sqlite')
}
executors = {
'default': {'type': 'threadpool', 'max_workers': 20},
'processpool': ProcessPoolExecutor(max_workers=5)
}
job_defaults = {
'coalesce': False,
'max_instances': 3
}
scheduler = BackgroundScheduler()
scheduler.add_job(file_loader, 'cron', hour=12, id='file_loader_id_12')
scheduler.add_job(file_loader, 'cron', hour=0, id='file_loader_id_24')
scheduler.configure(jobstores=jobstores, executors=executors, job_defaults=job_defaults, timezone=utc)
scheduler.start()

How to access return value from apscheduler?

I can't quite piece together how to access the return values from scheduled jobs in apscheduler. The job needs to run at a different time each day, and I need the return value from today's job to schedule tomorrow's job.
This link (how to get return value from apscheduler jobs) appears to be the best previous answer to this question. It suggests adding a listener to the scheduler. I've added a listener, but I'm not sure how to access it's return value. I can access the listeners attached to the scheduler, but I can't access their outputs. A listener, job_runs() in the code below, will print when a scheduled job runs.
Further, I know I need to access a JobExecutionEvent (https://apscheduler.readthedocs.io/en/latest/modules/events.html#module-apscheduler.events) which holds the return value from the function.
First, the function I want to access is run_all() where a bunch of operations are performed, but I just return True for the test case.
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.events import EVENT_JOB_EXECUTED, EVENT_JOB_ERROR, JobExecutionEvent
from datetime import datetime, timedelta
import logging
def run_all():
return True
def job_runs(event): # listener function
if event.exception:
print('The job did not run')
else:
print('The job completed # {}'.format(datetime.now()))
def job_return_val(event): # listener function
return event.retval
Then, I setup the scheduler, add the listeners, and add the job. The trigger is set to run the function 1 minute after the job is added to scheduler.
scheduler = BackgroundScheduler()
scheduler.add_listener(job_runs, EVENT_JOB_EXECUTED | EVENT_JOB_ERROR)
scheduler.add_listener(job_return_val, EVENT_JOB_EXECUTED)
cron_args = datetime_to_dict(datetime.now() + timedelta(minutes=1))
job = scheduler.add_job(run_all, "cron", **cron_args)
Next, I start the scheduler and print the scheduled job. Additionally, I setup logging so I know where the scheduler is.
test = scheduler.start()
scheduler.print_jobs()
logging.basicConfig()
logging.getLogger('apscheduler').setLevel(logging.DEBUG)
With the logging enabled, the scheduler reports that the job is run and removed from the scheduler, as I expect it to. job_runs() prints the correct output to the console. And with breakpoints, I know job_return_val() is called. However, I have no clue where the value it returns is sent to. The function appears to be called in a different thread called APScheduler. I don't know much about threads, but that makes sense. However, I do not understand when the output from that thread is returned to the main thread.
Finally, I've tried instantiating a JobExceptionEvent with the code, job_id, jobstore, and scheduled_run_time accessible from the attributes of scheduler and job, but the JobExceptionEvent does not seem to have any knowledge that the event was run in scheduler. That also seems to make sense due to the threading described in the preceding paragraph.
Any help in sorting through this would be great!
The return value of listener is not used anywhere (see the code), so there's no use to return any value anyway. If you need to schedule another job based on the value of previous job (acquired in the listener via the event object), you have to do it right in that listener.
EDIT: To illustrate how to do it (and prove it's possible), see this sample code:
from datetime import datetime
import time
from apscheduler.events import EVENT_JOB_ERROR, EVENT_JOB_EXECUTED
from apscheduler.schedulers.background import BackgroundScheduler
def tick():
print('Tick! The time is: %s' % datetime.now())
def tack():
print('Tack! The time is: %s' % datetime.now())
def listener(event):
if not event.exception:
job = scheduler.get_job(event.job_id)
if job.name == 'tick':
scheduler.add_job(tack)
if __name__ == '__main__':
scheduler = BackgroundScheduler()
scheduler.add_listener(listener, EVENT_JOB_EXECUTED | EVENT_JOB_ERROR)
scheduler.add_job(tick, 'interval', seconds=5)
scheduler.start()
try:
while True:
time.sleep(1)
except (KeyboardInterrupt, SystemExit):
scheduler.shutdown()
The output:
(venv) pasmen#nyx:~/tmp/x$ python test.py
Tick! The time is: 2019-04-03 19:51:29.192420
Tack! The time is: 2019-04-03 19:51:29.195878
Tick! The time is: 2019-04-03 19:51:34.193145
Tack! The time is: 2019-04-03 19:51:34.194898
Tick! The time is: 2019-04-03 19:51:39.193207
Tack! The time is: 2019-04-03 19:51:39.194868
Tick! The time is: 2019-04-03 19:51:44.193223
Tack! The time is: 2019-04-03 19:51:44.195066
...
You can use global variables for now. Here's an example:
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.triggers.cron import CronTrigger
def fn():
'''Increase `times` by one and print it.'''
global times
times += 1
print(times)
sched = BlockingScheduler()
times = 0
# Execute fn() each second.
sched.add_job(fn, trigger=CronTrigger(second='*/1'))
sched.start()
What you need would require the stateful jobs feature to be implemented.
I encountered the same problem a few days ago. My first solution was using keyword global, but not long before I realized it was problematic, because the variables defined outside the job could unexpectedly be changed, especially when they are local variables in a loop.
Then I thought about using listeners, too. But the callback passed into a listener takes only the event as a single argument, which means the only information you can get from the callback is the event itself, profoundly limiting what you could possibly do.
Finally I choose to pass a func to the task to be scheduled, which works fine to me. What you really have to do is just to use the return value of the scheduled task as an argument for the func, as the code below shows.
from apscheduler.schedulers.blocking import BlockingScheduler
from datetime import datetime
scheduler = BlockingScheduler()
def job_with_return(a:int, b:int, callback_return):
# You can do something heavier here with the inputs.
result = a + b
print(f'[in job] result is {result}')
if callback_return:
callback_return(result)
scheduler.add_job(func=job_with_return,
trigger='date',
args=(1, 2, lambda r: print(f'[out of job]: result is {r}')),
run_date=datetime.now(),
)
scheduler.start()
The output:
[in job] result is 3
[out of job]: result is 3
As to the request of OP,
The job needs to run at a different time each day, and I need the
return value from today's job to schedule tomorrow's job.
you can additionally feed the func with the scheduler as well as the time you want the task to run in tomorrow, so that you can arrange a schedule in the func.
from apscheduler.schedulers.blocking import BlockingScheduler
from datetime import datetime, timedelta
scheduler = BlockingScheduler()
today = datetime.now()
tomorrow = today + timedelta(seconds=1)
def task_today(a:int, b:int, scheduler:BlockingScheduler, callback_return, tomorrow:datetime):
# What we have to do today is to get the result and use it to schedule tomorrow's task.
result_today = a + b
print(f"[{datetime.now().strftime('%H:%M:%S')}] (Today) The result is {result_today}.")
scheduler.add_job(callback_return, 'date',
args=(result_today,),
run_date=tomorrow,
id='job_tomorrow')
def task_tomorrow(result_from_today:int):
result_tomrrow = result_from_today * 2
print(f"[{datetime.now().strftime('%H:%M:%S')}] (Tommorow) The result is {result_tomrrow}.")
scheduler.add_job(func=task_today,
trigger='date',
args=(1, 2, scheduler, task_tomorrow, tomorrow),
run_date=today,
id='job_today')
scheduler.start()
The output:
[22:22:40] (Today) The result is 3.
[22:22:41] (Tommorow) The result is 6.
You can even make a daily task with a recursion.
from apscheduler.schedulers.blocking import BlockingScheduler
from datetime import datetime, timedelta
scheduler = BlockingScheduler()
today = datetime.now()
def task_daily(result_yesterday:int, day_counter:int, scheduler:BlockingScheduler, callback_return):
# You can do something heavier here with more inputs.
result_today = result_yesterday + 2
day_counter += 1
tomorrow = datetime.now() + timedelta(seconds=1)
print(f"[{datetime.now().strftime('%H:%M:%S')}] (day {day_counter}) The result for today is {result_today}.")
scheduler.add_job(task_daily, 'date',
args=(result_today, day_counter, scheduler, callback_return),
run_date=tomorrow)
scheduler.add_job(func=task_daily,
trigger='date',
args=(0, 0, scheduler, task_daily),
run_date=today)
scheduler.start()
The output:
[22:43:17] (day 1) The result for today is 2.
[22:43:18] (day 2) The result for today is 4.
[22:43:19] (day 3) The result for today is 6.
[22:43:20] (day 4) The result for today is 8.
[22:43:21] (day 5) The result for today is 10.
[22:43:22] (day 6) The result for today is 12.
[22:43:23] (day 7) The result for today is 14.

Python - Apscheduler not stopping a job even after using 'remove_job'

This is my code
I'm using the remove_job and the shutdown functions of the scheduler to stop a job, but it keeps on executing.
What is the correct way to stop a job from executing any further?
from apscheduler.schedulers.background import BlockingScheduler
def job_function():
print "job executing"
scheduler = BlockingScheduler(standalone=True)
scheduler.add_job(job_function, 'interval', seconds=1, id='my_job_id')
scheduler.start()
scheduler.remove_job('my_job_id')
scheduler.shutdown()
Simply ask the scheduler to remove the job inside the job_function using the remove_function as #Akshay Pratap Singh Pointed out correctly, that the control never returns back to start()
from apscheduler.schedulers.background import BlockingScheduler
count = 0
def job_function():
print "job executing"
global count, scheduler
# Execute the job till the count of 5
count = count + 1
if count == 5:
scheduler.remove_job('my_job_id')
scheduler = BlockingScheduler()
scheduler.add_job(job_function, 'interval', seconds=1, id='my_job_id')
scheduler.start()
As you are using BlockingScheduler , so first you know it's nature.
So, basically BlockingScheduler is a scheduler which runs in foreground(i.e start() will block the program).In laymen terms, It runs in the foreground, so when you call start(), the call never returns. That's why all lines which are followed by start() are never called, due to which your scheduler never stopped.
BlockingScheduler can be useful if you want to use APScheduler as a standalone scheduler (e.g. to build a daemon).
Solution
If you want to stop your scheduler after running some code, then you should opt for other types of scheduler listed in ApScheduler docs.
I recommend BackgroundScheduler, if you want the scheduler to run in the background inside your application/program which you can pause, resume and remove at anytime, when you need it.
The scheduler needs to be stopped from another thread. The thread in which scheduler.start() is called gets blocked by the scheduler. The lines that you've written after scheduler.start() is unreachable code.
This is how I solved the problem. Pay attention to the position where the code schedule.shutdown() is located!
def do_something():
global schedule
print("schedule execute")
# schedule.remove_job(id='rebate')
schedule.shutdown(wait=False)
if __name__ == '__main__':
global schedule
schedule = BlockingScheduler()
schedule.add_job(do_something, 'cron', id='rebate', month=12, day=5, hour=17, minute=47, second=35)
schedule.start()
print('over')

How do I schedule an interval job with APScheduler?

I'm trying to schedule an interval job with APScheduler (v3.0.0).
I've tried:
from apscheduler.schedulers.blocking import BlockingScheduler
sched = BlockingScheduler()
def my_interval_job():
print 'Hello World!'
sched.add_job(my_interval_job, 'interval', seconds=5)
sched.start()
and
from apscheduler.schedulers.blocking import BlockingScheduler
sched = BlockingScheduler()
#sched.scheduled_job('interval', id='my_job_id', seconds=5)
def my_interval_job():
print 'Hello World!'
sched.start()
Either should work according to the docs, but the job never fires...
UPDATE:
It turns out there was something else, environment-related, preventing the task from running. This morning, the task is working fine without any modifications to the code from yesterday.
UPDATE 2:
After further testing, I've found that 'interval' jobs seem to be generally flaky... The above code now works in my dev environment, but not when I deploy to a staging env (I'm using a heroku app for staging). I have other apscheduler 'cron' jobs that work just fine in the staging/production envs.
When I turn on DEBUG logging for the "apscheduler.schedulers" logger, the log indicates that the interval job is added:
Added job "my_cron_job1" to job store "default"
Added job "my_cron_job2" to job store "default"
Added job "my_interval_job" to job store "default"
Scheduler started
Adding job tentatively -- it will be properly scheduled when the scheduler starts
Adding job tentatively -- it will be properly scheduled when the scheduler starts
Looking for jobs to run
Next wakeup is due at 2015-03-24 15:05:00-07:00 (in 254.210542 seconds)
How can the next wakeup be due 254 seconds from now when the interval job is set to 5 seconds??
You need to keep the thread alive. Here is a example of how I used it.
from subprocess import call
import time
import os
from pytz import utc
from apscheduler.schedulers.background import BackgroundScheduler
def job():
print("In job")
call(['python', 'scheduler/main.py'])
if __name__ == '__main__':
scheduler = BackgroundScheduler()
scheduler.configure(timezone=utc)
scheduler.add_job(job, 'interval', seconds=10)
scheduler.start()
print('Press Ctrl+{0} to exit'.format('Break' if os.name == 'nt' else 'C'))
try:
# This is here to simulate application activity (which keeps the main thread alive).
while True:
time.sleep(5)
except (KeyboardInterrupt, SystemExit):
# Not strictly necessary if daemonic mode is enabled but should be done if possible
scheduler.shutdown()
I haven't figured out what caused the original issue, but I got around it by swapping the order in which the jobs are scheduled, so that the 'interval' job is scheduled BEFORE the 'cron' jobs.
i.e. I switched from this:
def my_cron_job1():
print "cron job 1"
def my_cron_job2():
print "cron job 2"
def my_interval_job():
print "interval job"
if __name__ == '__main__':
from apscheduler.schedulers.blocking import BlockingScheduler
sched = BlockingScheduler(timezone='MST')
sched.add_job(my_cron_job1, 'cron', id='my_cron_job1', minute=10)
sched.add_job(my_cron_job2, 'cron', id='my_cron_job2', minute=20)
sched.add_job(my_interval_job, 'interval', id='my_job_id', seconds=5)
to this:
def my_cron_job1():
print "cron job 1"
def my_cron_job2():
print "cron job 2"
def my_interval_job():
print "interval job"
if __name__ == '__main__':
from apscheduler.schedulers.blocking import BlockingScheduler
sched = BlockingScheduler(timezone='MST')
sched.add_job(my_interval_job, 'interval', id='my_job_id', seconds=5)
sched.add_job(my_cron_job1, 'cron', id='my_cron_job1', minute=10)
sched.add_job(my_cron_job2, 'cron', id='my_cron_job2', minute=20)
and now both the cron jobs and the interval jobs run without a problem in both environments.
How can the next wakeup be due 254 seconds from now when the interval
job is set to 5 seconds??
It's simple:
you have many pending executions as your most of the jobs didn't completed in the interval-window of time.
You could use the following parameters in order to sort this out:
**misfire_grace_time**: Maximum time in seconds for the job execution to be allowed to delay before it is considered a misfire
**coalesce**: Roll several pending executions of jobs into one
To read more, check the documentation here.
The documentation had an error there. I've fixed it now.
That first line should be:
from apscheduler.schedulers.blocking import BlockingScheduler
It would've raised an ImportError though, but you didn't mention any.
Did you try any of the provided examples?
Ok, I've looked at the updated question.
The reason you're having problems may be that you could be using the wrong timezone. Your country is currently using daylight saving time in most locations, so the correct timezone would probably be MDT (Mountain Daylight Time). But that will break again when you move back to standard time. So I advise you to use a timezone like "America/Denver". That will take care of the DST switches.
Question: Are you using CentOS? So far it's the only known operating system where automatic detection of the local timezone is impossible.

Categories