I have used django_apscheduler to schedule jobs. And it's working fine. When I start server new job is added and periodically it's doing what I need to do. However if I exit django and start it again then django will fail with error.
apscheduler.jobstores.base.ConflictingIdError: u'Job identifier (myapp_db.jobs.test_job) conflicts with an existing job'
Basically old job exists in database and new job can not be created.
How can I remove all jobs during django startup.
I notice there is remove_all_job() function in apscheduler but I do not know from where to execute it?
I'm starting job.py from url.py with
import myapp.jobs
Thanks.
code:
import time
import sys
import requests
from bs4 import BeautifulSoup
import re
from kuce_db.models import NjuskaloData, UserVisitedData
import logging
from django.contrib.auth.models import User
from apscheduler.schedulers.background import BackgroundScheduler
from django_apscheduler.jobstores import DjangoJobStore, register_events, register_job
scheduler = BackgroundScheduler()
scheduler.add_jobstore(DjangoJobStore(), "default")
def get_trailing_number(s):
m = re.search(r'\d+$', s)
return int(m.group()) if m else None
#register_job(scheduler, "interval", seconds=300)
def test_job():
print("I'm a test job!")
register_events(scheduler)
#
scheduler.start()
print("Scheduler started!")
logging.basicConfig()
use replace_existing flag to tell it replace existed job
#register_job(scheduler, "interval", seconds=300, replace_existing=True)
def test_job():
print("I'm a test job!")
Related
I have this APScheduler code:
import atexit
from apscheduler.schedulers.background import BackgroundScheduler
from main.utils import run_employee_import
scheduler = BackgroundScheduler()
scheduler.add_job(run_employee_import, "interval", minutes=2)
scheduler.start()
# Shut down the scheduler when exiting the app
atexit.register(lambda: scheduler.shutdown())
When I add this code to settings.py to run it when the app starts to run, it gives me the following error:
raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
PS: I did not include the run_employee_import code because I tested it already (replaced its content with a simple pass) and nothing changed, so it is irrelevant to the error.
I completely changed the way I run the scheduler. And it worked, let me share the solution with you:
I create a .py file inside the app:
app/bulk_task.py
I create a start function and put the schedule code in it:
from apscheduler.schedulers.background import BackgroundScheduler
from main.utils import **<MY_TASK>**
def start():
scheduler = BackgroundScheduler()
scheduler.add_job(**<MY_TASK>**, "interval", minutes=5)
scheduler.start()
Then I open the apps.py file of my app, and I add the following code:
from django.apps import AppConfig
class MainConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "main"
def ready(self):
from . import bulk_task
bulk_task.start()
And that's it ;)
I am importing Schedule into Flask. My project contains WSGI however I know little about the relationship between Flask and WSGI. Now I have three main files:
wsgi.py: Automatically generated by other tool.
app.py : I put client request here.
test.py: Used to test Schedule.
I want to start a task which is a long task when server launch. Here is the part of wsgi.py:
# -*- coding: utf-8 -*-
from threading import Thread
import test
t = Thread(target=test.job)
t.start()
if __name__ == '__main__':
...
As you see I start a thread and let the job work in it.Here is my test.py.
import schedule
def job():
schedule.every(1).seconds.do(pr)
def pr():
print("I'm working...")
My problem is that the job never starts.
I find out my problem.I never let schedule execute jobs. Now wsgi.py looks like this.
# -*- coding: utf-8 -*-
from threading import Thread
import test
schedule.every(1).seconds.do(test.job)
t = Thread(target=test.run_schedule)
t.start()
if __name__ == '__main__':
...
And test.py:
import schedule
import time
start_time = time.time()
def job():
print("I'm working..." + str(time.time() - start_time))
def run_schedule():
while True:
schedule.run_pending()
time.sleep(1)
In order to work in separate thread, I create a thread and in this thread I loop every 1ms. In loop, schedule invoke run_pending to call the job if time out (in my case it's 1s).
I'm trying to use the APScheduler library to execute a job at two certain times: 1am PST and 2am PST.
Sort of lost, I am trying to figure out what am doing wrong. Here is my code so far:
#!/usr/bin/python
import sys
import re
import urllib
import csv
import urllib2
import logging
import datetime
from apscheduler.schedulers.blocking import BlockingScheduler
sched = BlockingScheduler()
logging.basicConfig()
def main():
#Make a rest call, store the output in a csv, close the file.
...
f.close()
sched.add_job(my_job, 'date', run_date = '2015-12-15 01:00:00', args=['text'])
try:
sched.start()
print(datetime.datetime.now())
except (KeyboardInterrupt):
logger.debug('Terminating...')
print (datetime.datetime.now())
The current time is 12:16 PST, I just ran the above script and it executed and created the file prior to the job time of 12:30:30pm; this is for testing purposes.
So, I'm trying to figure out what I am doing wrong with regards to using APScheduler.
I want to improve this code, what are some things I can change?
If you want to run your code at certain times every time, you should be using the cron trigger:
sched.add_job(my_job, 'cron', hour='1,2', args=['text'])
For better debugging:
logging.basicConfig(level=logging.DEBUG)
I have python code that I am developing for a website that, among other things, creates an excel sheet and then converts it into a json file. I need for this code to run continuously unless it is killed by the website administrator.
To this end, I am using APscheduler.
The code runs perfectly without APscheduler but when I attempt to add the rest of the code one of two things happens; 1) It runs forever and will not stop despite using "ctrl+C" and I need to stop it using task manager or 2) It only runs once, and then it stops
Code That doesn't Stop:
from apscheduler.scheduler import Scheduler
import logging
import time
logging.basicConfig()
sched = Scheduler()
sched.start()
(...)
code to make excel sheet and json file
(...)
#sched.interval_schedule(seconds = 15)
def job():
excelapi_final()
while True:
time.sleep(10)
sched.shutdown(wait=False)
Code that stops running after one time:
from apscheduler.scheduler import Scheduler
import logging
import time
logging.basicConfig()
sched = Scheduler()
(...)
#create excel sheet and json file
(...)
#sched.interval_schedule(seconds = 15)
def job():
excelapi_final()
sched.start()
while True:
time.sleep(10)
sched.shutdown(wait=False)
I understand from other questions, a few tutorials and the documentation that sched.shutdown should allow for the code to be killed by ctrl+C - however that is not working. Any ideas? Thanks in advance!
You could use the standalone mode:
sched = Scheduler(standalone=True)
and then start the scheduler like this:
try:
sched.start()
except (KeyboardInterrupt):
logger.debug('Got SIGTERM! Terminating...')
Your corrected code should look like this:
from apscheduler.scheduler import Scheduler
import logging
import time
logging.basicConfig()
sched = Scheduler(standalone=True)
(...)
code to make excel sheet and json file
(...)
#sched.interval_schedule(seconds = 15)
def job():
excelapi_final()
try:
sched.start()
except (KeyboardInterrupt):
logger.debug('Got SIGTERM! Terminating...')
This way the program will stop when Ctrl-C is pressed
You can gracefully shut it down:
import signal
from apscheduler.scheduler import Scheduler
import logging
import time
logging.basicConfig()
sched = Scheduler()
(...)
#create excel sheet and json file
(...)
#sched.interval_schedule(seconds = 15)
def job():
excelapi_final()
sched.start()
def gracefully_exit(signum, frame):
print('Stopping...')
sched.shutdown()
signal.signal(signal.SIGINT, gracefully_exit)
signal.signal(signal.SIGTERM, gracefully_exit)
Please check this python code:
#!/usr/bin/env python
import requests
import multiprocessing
from time import sleep, time
from requests import async
def do_req():
r = requests.get("http://w3c.org/")
def do_sth():
while True:
sleep(10)
if __name__ == '__main__':
do_req()
multiprocessing.Process( target=do_sth, args=() ).start()
When I press Ctrl-C (wait 2sec after run - let Process run), it doesn't stop. When I change the import order to:
from requests import async
from time import sleep, time
it stops after Ctrl-C. Why it doesn't stop/kill in first example?
It's a bug or a feature?
Notes:
Yes I know, that I didn't use async in this code, this is just stripped down code. In real code I use it. I did it to simplify my question.
After pressing Ctrl-C there is a new (child) process running. Why?
multiprocessing.__version__ == 0.70a1, requests.__version__ == 0.11.2, gevent.__version__ == 0.13.7
Requests async module uses gevent. If you look at the source code of gevent you will see that it monkey patches many of Python's standard library functions, including sleep:
request.async module during import executes:
from gevent import monkey as curious_george
# Monkey-patch.
curious_george.patch_all(thread=False, select=False)
Looking at the monkey.py module of gevent you can see:
https://bitbucket.org/denis/gevent/src/f838056c793d/gevent/monkey.py#cl-128
def patch_time():
"""Replace :func:`time.sleep` with :func:`gevent.sleep`."""
from gevent.hub import sleep
import time
patch_item(time, 'sleep', sleep)
Take a look at the code from the gevent's repository for details.