I have this method:
def getExchangeRates():
""" Here we have the function that will retrieve the latest rates from fixer.io """
rates = {}
response = urlopen('http://data.fixer.io/api/latest?access_key=c2f5070ad78b0748111281f6475c0bdd')
data = response.read()
rdata = json.loads(data.decode(), parse_float=float)
rates_from_rdata = rdata.get('rates', {})
for rate_symbol in ['USD', 'GBP', 'HKD', 'AUD', 'JPY', 'SEK', 'NOK']:
try:
rates[rate_symbol] = rates_from_rdata[rate_symbol]
except KeyError:
logging.warning('rate for {} not found in rdata'.format(rate_symbol))
pass
return rates
#require_http_methods(['GET', 'POST'])
def index(request):
rates = getExchangeRates()
fixerio_rates = [Fixerio_rates(currency=currency, rate=rate)
for currency, rate in rates.items()]
Fixerio_rates.objects.bulk_create(fixerio_rates)
return render(request, 'index.html')
I want to schedule this, let's say, for every day at 9am, except for weekends.
I haven't found some comprehensive tutorial on how to schedule this based on such a specific datetime, also, I don't know if I could schedule this method, or create another method in my tasks file that inherits this one, and runs at any specific date.
I do have the celery.py file in my project root, and the tasks.py file in my app folder.
Or, maybe celery isn't the way to go for this situation?
Any ideas?
There are some django packages that let you manage "cron-like" jobs using django admin interface. I used in the past both django-chronograph and django-chroniker (https://github.com/chrisspen/django-chroniker). There is also django-cron (https://django-cron.readthedocs.io/en/latest/installation.html), but I never used it.
All of them have similar approach: you create one single entry on your crontab runninng something like python manage.py runcrons every minute, and on your settings.py you add the package to show it on admin.
Take a look on the documentation of either Chroniker or Django-cron for more info on how to set it up.
Also, you can use Celery Beat to schedule tasks that you need.
Tasks can be scheduled with Celery Beat.
Celery Beat must be launched as another process. This beat process will kick scheduled tasks to the celery worker process that will launch the tasks like any other celery asynchronous task. To orquestrate these two process usually is a good idea use something like supervisord in production and honcho in development.
The scheduled tasks can be defined in the code, or stored in a database and handled through django-admin with the extension django-celery-beat
To add it by code the easiest way is create another method in the tasks.py file. For your requirement for every day at 9am, except for weekends "it could look like this"
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
# Executes every Monday morning at 9 a.m.
sender.add_periodic_task(
crontab(hour=9, minute=0, day_of_week=1),
test.s('Happy Mondays!'),
)
sender.add_periodic_task(
crontab(hour=9, minute=0, day_of_week=2),
test.s('Happy Tuesday!'),
)
sender.add_periodic_task(
crontab(hour=9, minute=0, day_of_week=3),
test.s('Happy Wednesday!'),
)
sender.add_periodic_task(
crontab(hour=9, minute=0, day_of_week=4),
test.s('Happy Thursday!'),
)
sender.add_periodic_task(
crontab(hour=9, minute=0, day_of_week=1),
test.s('Happy Friday!'),
)
#app.task
def test(arg):
print(arg)
Related
I have written an algorithm. And when I use python manage.py runserver, my website will run on the local server.
Now I want to run my algorithm after python manage.py runserver.
In other words, when I start the django website, I hope the algorithm will run in the background until it is completed. And I want to know if the algorithm is still running or the algorithm is complete.
What should I do?
Thanks.
something like this:
def function_that_downloads(my_args):
# do some he
re
def __init__(self, function_that_downloads):
threading.Thread.__init__(self)
self.runnable = function_that_downloads
self.daemon = True
def run(self):
self.runnable()
Hi you might want to checkout Django Celery Beat so later you can just define some task inside your Django Application and execute your function periodically
from celery import Celery
from celery.schedules import crontab
app = Celery()
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
# Calls test('hello') every 10 seconds.
sender.add_periodic_task(10.0, test.s('hello'), name='add every 10')
# Calls test('world') every 30 seconds
sender.add_periodic_task(30.0, test.s('world'), expires=10)
# Executes every Monday morning at 7:30 a.m.
sender.add_periodic_task(
crontab(hour=7, minute=30, day_of_week=1),
test.s('Happy Mondays!'),
)
#app.task
def test(arg):
print(arg)from celery import Celery
I am writing a program to schedule and cancel alarms in Flask. I am using the apscheduler library for the timings.
I need to be able to add events to the job queue at any point, so I need to be able to add events after the scheduler is run.
Currently, I have:
from apscheduler.schedulers.background import BackgroundScheduler
def cancel():
job = events[0]
job.remove()
def schedule():
sched = scheds[0]
try:
sched.shutdown()
except:
pass
job = sched.add_job(my_job, 'date', run_date=t, args=['text'])
events.append(job)
sched.start()
def schedule2():
sched = scheds[0]
try:
sched.shutdown()
except:
pass
job = sched.add_job(my_job, 'date', run_date=t2, args=['text'])
events.append(job)
sched.start()
Where scheds is an array to store a global scheduler, and events is an array which stores the events that are scheduled.
I need to run schedule, then schedule2, to schedule two different jobs. When I try this, I get an error which says that I cannot run schedule2 because the 'scheduler is already running'. How can I achieve this?
I have problem with daily scheduled tasks with crontab.
Here is my celery.py
app.conf.beat_schedule = {
'run-cache-updater': {
'task': 'tasks.run_cache_updater',
'schedule': crontab(
minute=0,
hour='1-4'
),
}
}
Below is my tasks.py
What I am doing there is, getting all records from DB. Triggering other jobs to update my caches on Redis.
#app.task
def run_cache_updater():
batch_size = 1000
cache_records = models.CacheRecords.objects.all()
def _chunk_list(all_records, size_of_batch):
for i in range(0, len(all_records), size_of_batch):
yield [item.id for item in all_records[i: i + batch_size]]
for items in _chunk_list(cache_records, batch_size):
update_cache.delay(items)
#app.task
def update_cache(ids_in_chunks):
for id in ids_in_chunks:
# Some calls are done here. Then sleep for 200 ms.
time.sleep(0.2)
My tasks are running good. However, they start to run between 1 and 4 and then they start again every 4 hours like 8-11, 15-18..
What I am doing wrong here and how can I fix it?
This sounds like a Celery bug, it's probably worth raising on their Github repo.
However, as a workaround, you could try the more explicit notation, hour='1,2,3,4', just in case the issue is in the parsing of that specific crontab interval style.
I need to implement a scheduled task in our Django app. DBader's schedule seems to be a good candidate for the job, however when run it as part of a Django project, it doesn't seem to produce the desired effect.
Specifically, this works fine as an independent program:
import schedule
import time
import logging
log = logging.getLogger(__name__)
def handleAnnotationsWithoutRequests(settings):
'''
From settings passed in, grab job-ids list
For each job-id in that list, perform annotation group/set logic [for details, refer to handleAnnotationsWithRequests(requests, username)
sans requests, those are obtained from db based on job-id ]
'''
print('Received settings: {}'.format(str(settings)))
def job():
print("I'm working...")
#schedule.every(3).seconds.do(job)
#schedule.every(2).seconds.do(handleAnnotationsWithoutRequests, settings={'a': 'b'})
invoc_time = "10:33"
schedule.every().day.at(invoc_time).do(handleAnnotationsWithoutRequests, settings={'a': 'b'})
while True:
schedule.run_pending()
time.sleep(1)
But this (equivalent) code run in Django context doesn't result in an invocation.
def handleAnnotationsWithoutRequests(settings):
'''
From settings passed in, grab job-ids list
For each job-id in that list, perform annotation group/set logic [for details, refer to handleAnnotationsWithRequests(requests, username)
sans requests, those are obtained from db based on job-id ]
'''
log.info('Received settings: {}'.format(str(settings)))
def doSchedule(settings):
'''
with scheduler library
Based on time specified in settings, invoke .handleAnnotationsWithoutRequests(settings)
'''
#settings will need to be reconstituted from the DB first
#settings = {}
invocationTime = settings['running_at']
import re
invocationTime = re.sub(r'([AaPp][Mm])', "", invocationTime)
log.info("Invocation time to be used: {}".format(invocationTime))
schedule.every().day.at(invocationTime).do(handleAnnotationsWithoutRequests, settings=settings)
while True:
schedule.run_pending()
time.sleep(1)
so the log from handleAnnotationsWithoutRequests() doesn't appear on the console.
Is this scheduling library compatible with Django? Are there any usage samples that one could refer me to?
I'm suspecting some thread issues are at work here. Perhaps there are better alternatives to be used? Suggestions are welcome.
Thank you in advance.
For web servers, you probably don't want something that runs in-process:
An in-process scheduler for periodic jobs [...]
https://github.com/Tivix/django-cron has proven a working solution.
There's also the heavyweight champion Celery and Celerybeat.
I do this a lot with Django Commands
The pattern I use is to setup a new Django command in my app and then make it a long-running process inside a never-ending while() loop.
I the loop iterates continuously with a custom defined sleep(1) timer.
The short version is here, with a bit of pseudo-code thrown in. You can see a working version of this pattern in my Django Reference Implementation.
class Command(BaseCommand):
help = 'My Long running job'
def handle(self, *args, **options):
self.stdout.write(self.style.SUCCESS(f'Starting long-running job.'))
while True:
if conditions met for job:
self.job()
sleep(5)
def job(self):
self.stdout.write(self.style.SUCCESS(f'Running the job...'))
How can I test if a task (task_id) is still processed in celery? I have the following scenario:
Start a task in a Django view
Store the BaseAsyncResult in the session
Shutdown the celery daemon (hard) so the task is not processed anymore
Check if the task is 'dead'
Any ideas? Can a lookup all task being processed by celery and check if mine is still there?
define a field (PickledObjectField) in your model to store the celery task:
class YourModel(models.Model):
.
.
celery_task = PickledObjectField()
.
.
def task():
self.celery_task = SubmitTask.apply_async(args = self.task_detail())
self.save()
In case your task is not specific on any model you should create one specifically for the celery tasks.
or else I suggest using django-celery. It has a nice monitoring feature:
http://ask.github.com/celery/userguide/monitoring.html#django-admin-monitor, saves the tasks details in a django model in a nice graphical way.
I think there is a better way than to store a task object in the model. For example, in case you wanted to check if a group of task (parallel) have completed:
# in the script you launch the task
from celery import group
job = group(
task1.s(param1, param2),
task2.s(param3, param4)
)
result = job.apply_async()
result.save()
# save the result ID in your model
your_obj_model = YourModel.objects.get(id='1234')
your_obj_model.task_id = result.id
your_obj_model.save()
Then in your view
from celery.result import GroupResult
# ...
task_result = GroupResult.restore(your_obj_model.task_id)
task_finished = task_result.ready()
# will be True or False