I am trying to schedule a few jobs inside my python. Supposely , the text from the logging should appear every 1 minute and every 5 minute from jobs.py file inside my docker container. However, the text is appearing every 2minutes inside the docker container. Is there a clash between the python schedule and cronjobs ?
Current Output inside the docker container
13:05:00 [I] werkzeug 172.20.0.2 - - [08/May/2022 13:05:00] "GET /reminder/send_reminders HTTP/1.1" 200 -
13:06:00 [I] werkzeug 172.20.0.2 - - [08/May/2022 13:06:00] "GET /feeds/update_feeds HTTP/1.1" 200 -
13:07:00 [D] schedule Running job Job(interval=1, unit=minutes, do=job_feeds_update, args=(), kwargs={})
13:07:00 [I] jobs job_feeds_update
13:07:00 [I] werkzeug 172.20.0.2 - - [08/May/2022 13:07:00] "GET /feeds/update_feeds HTTP/1.1" 200 -
13:08:00 [I] werkzeug 172.20.0.2 - - [08/May/2022 13:08:00] "GET /feeds/update_feeds HTTP/1.1" 200 -
13:09:00 [D] schedule Running job Job(interval=1, unit=minutes, do=job_feeds_update, args=(), kwargs={})
13:09:00 [I] jobs job_feeds_update
13:09:00 [I] werkzeug 172.20.0.2 - - [08/May/2022 13:09:00] "GET /feeds/update_feeds HTTP/1.1" 200 -
13:10:00 [I] werkzeug 172.20.0.2 - - [08/May/2022 13:10:00] "GET /feeds/update_feeds HTTP/1.1" 200 -
13:10:00 [I] werkzeug 172.20.0.2 - - [08/May/2022 13:10:00] "GET /reminder/send_reminders HTTP/1.1" 200 -
13:11:00 [D] schedule Running job Job(interval=1, unit=minutes, do=job_feeds_update, args=(), kwargs={})
13:11:00 [I] jobs job_feeds_update
13:11:00 [D] schedule Running job Job(interval=5, unit=minutes, do=job_send_reminders, args=(), kwargs={})
13:11:00 [I] jobs job_send_reminders
server.py
#Cron Job
#app.route('/feeds/update_feeds')
def update_feeds():
schedule.run_pending()
return 'OK UPDATED FEED!'
#app.route('/reminder/send_reminders')
def send_reminders():
schedule.run_pending()
return 'OK UPDATED STATUS!'
jobs.py
def job_feeds_update():
update_feed()
update_feed_eng()
logger.info("job_feeds_update")
schedule.every(1).minutes.do(job_feeds_update)
# send email reminders
def job_send_reminders():
send_reminders()
logger.info("job_send_reminders")
schedule.every(5).minutes.do(job_send_reminders)
Docker File
FROM alpine:latest
# Install curlt
RUN apk add --no-cache curl
# Copy Scripts to Docker Image
COPY reminders.sh /usr/local/bin/reminders.sh
COPY feeds.sh /usr/local/bin/feeds.sh
RUN echo ' */5 * * * * /usr/local/bin/reminders.sh' >> /etc/crontabs/root
RUN echo ' * * * * * /usr/local/bin/feeds.sh' >> /etc/crontabs/root
# Run crond -f for Foreground
CMD ["/usr/sbin/crond", "-f"]
I think you're running into a couple of issues:
As you suspected, your schedule is on a different schedule/interval than your cron job. They're out of sync (and you can't ever expect them to be in sync for the next reason). From the moment your jobs.py script was executed, that's the starting point from which the schedule counts the intervals.
i.e. if you're running something every minute but the jobs.py script starts at 30 seconds into the current minute (i.e. 01:00:30 - 1:00am 30 seconds past), then the scheduler will run the job at 1:01:30, then 1:02:30, then 1:03:30 and so on.
Schedule doesn't guarantee you precise frequency execution. When the scheduler runs a job, the job execution time is not taken into account. So if you schedule something like your feeds/reminders jobs, it could take a little bit to process. Once it's finished running, the scheduler decides that the next job will only run 1 minute after the end of the previous job. This means your execution time can throw off the schedule.
Try running this example in a python script to see what I'm talking about
# Schedule Library imported
import schedule
import time
from datetime import datetime
def geeks():
now = datetime.now() # current date and time
date_time = now.strftime("%m/%d/%Y, %H:%M:%S")
time.sleep(5)
print(date_time + "- Look at the timestamp")
geeks();
# Task scheduling
# After every 10mins geeks() is called.
schedule.every(1).seconds.do(geeks)
# Loop so that the scheduling task
# keeps on running all time.
while True:
# Checks whether a scheduled task
# is pending to run or not
schedule.run_pending()
time.sleep(0.1)
We've scheduled the geeks function to run every second. But if you look at the geeks function, I've added a time.sleep(5) to pretend that there may be some blocking API call here that can take 5 seconds. Then observe the timestamps logged - you'll notice they're not always consistent with the schedule we originally wanted!
Now onto how your cron job and scheduler are out of sync
Look at the following logs:
13:07:00 [D] schedule Running job Job(interval=1, unit=minutes, do=job_feeds_update, args=(), kwargs={})
13:07:00 [I] jobs job_feeds_update
13:07:00 [I] werkzeug 172.20.0.2 - - [08/May/2022 13:07:00] "GET /feeds/update_feeds HTTP/1.1" 200 -
# minute 8 doesn't trigger the schedule for feeds
13:09:00 [D] schedule Running job Job(interval=1, unit=minutes, do=job_feeds_update, args=(), kwargs={})
13:09:00 [I] jobs job_feeds_update
13:09:00 [I] werkzeug 172.20.0.2 - - [08/May/2022 13:09:00] "GET /feeds/update_feeds HTTP/1.1" 200 -
What's likely happening here is as follows:
at 13:07:00, your cron sends the request to feed items
at 13:07:00, the job schedule has a pending job for feed items
at 13:07:00:, the job finishes and schedule decides the next job can only run after 1 minute from now, which is roughly ~13:08:01 (note the 01, this is to account for milliseconds/timing of job executions, which lets assume it took 1 second to run the feed items update)
at 13:08:00, your cron job triggers the request asking schedule run_pending jobs.
at 13:08:00 however, there are no pending jobs to run because the next time feed items can run is 13:08:01 which is not right now.
at 13:09:00, your cron tab triggers the request again
at 13:09:00, there is a pending job available that should've run at 13:08:01 so that gets executed now.
I hope this illustrates the issue you're running into being out of sync between cron and schedule. This issue will become worse in a production environment. You can read more about Parallel execution for schedule as a means to keep things off the main thread but that will only go so far. Let's talk about...
Possible Solutions
Use run_all from schedule instead of run_pending to force jobs to trigger, regardless of when they're actually scheduled for.
But if you think about it, this is no different than simply calling job_feeds_update straight from your API route itself. This isn't a bad idea by itself but it's still not super clean as it will block the main thread of your API server until the job_feeds_update is complete, which might not be ideal if you have other routes that users need.
You could combine this with the next suggestion:
Use a jobqueue and threads
Check out the second example on the Parallel Execution page of schedule's docs. It shows you how to use a jobqueue and threads to offload jobs.
Because you run schedule.run_pending(), your main thread in your server is blocked until the jobs run. By using threads (+ the job queue), you can keep scheduling jobs in the queue + avoid blocking the main server with your jobs. This should optimize things a little bit further for you by letting jobs continue to be scheduled.
Use ischedule instead as it takes into account the job execution time and provides precise schedules: https://pypi.org/project/ischedule/. This might be the simplest solution for you in case 1+2 end up being a headache!
Don't use schedule and simply have your cron jobs hit a route that just runs the actual function (so basically counter to the advice of using 1+2 above). Problem with this is that if your functions take longer than a minute to run for feed updates, you may have multiple overlapping cron jobs running at the same time doing feed updates. So I'd recommend not doing this and relying on a mechanism to queue/schedule your requests with threads and jobs. Only mentioning this as a potential scenario of what else you could do.
Related
I have cron jobs that I schedule to run every minute, when I run the program for the first time it works fine. If I close and restart the program the jobs never run again, even though the scheduler.print_jobs() function shows the jobs are scheduled as this is printed:
Pending jobs:
daily_check (trigger: cron[second='0'], next run at: 2023-02-16 15:26:08 UTC)
daily_check (trigger: cron[second='30'], pending)
daily_check (trigger: cron[second='50'], pending)
This is the scheduling code that runs when the program runs
def schedule_daily_check():
try:
scheduler.add_job(daily_check,'cron',second=00,id='daily_kek',jitter=10,replace_existing=False, misfire_grace_time = 1,coalesce=True)
scheduler.add_job(daily_check,'cron',second=30,id='daily_kek2',jitter=10,replace_existing=False, misfire_grace_time = 1,coalesce=True)
scheduler.add_job(daily_check,'cron',second=50,id='daily_kek3',jitter=10,replace_existing=False, misfire_grace_time = 1,coalesce=True)
#scheduler.add_job(daily_check,'cron',second=43,second=random.randint(0,59),id='daily_kek4',jitter=60*60,replace_existing=False)
scheduler.start()
except ConflictingIdError:
scheduler.print_jobs()
I've been having issues in getting crontab tasks to run in my local time so what I did was create 24 tasks like the following for each hour of the day.
app.conf.beat_schedule = {
'crontab-test-8am': {
'task': 'celery.tasks.test_crontab',
'schedule': crontab(minute="0", hour="8"),
'args':('8am',)
},
}
#app.task()
def test_crontab(message):
print('CRONTAB HAS RUN - ' + message)
I eventually figured out how to have crontab correctly run in my local time with the following
app.conf.enable_utc = False
app.conf.update(timezone = "Australia/Perth") #8am AWST = 12am UTC
The problem is it's now running the tasks in both local time and UTC time.
[2022-07-22 08:00:00,000: INFO/MainProcess] Scheduler: Sending due task crontab-test-8am (celery.tasks.test_crontab)
[2022-07-22 08:00:00,010: INFO/MainProcess] Scheduler: Sending due task crontab-test-12am (celery.tasks.test_crontab)
[2022-07-22 08:00:00,015: WARNING/ForkPoolWorker-51] CRONTAB HAS RUN - 8am
[2022-07-22 08:00:00,223: WARNING/ForkPoolWorker-51] CRONTAB HAS RUN - 12am
From what I could find, celery beat stores the schedules in the celerybeat-schedule file. I deleted it and restarted celery beat but it did the same thing.
I'd like any suggestions as to how I could try fix this.
I have a Django site that uses the trio_cdp package to generate PDFs using a headless Google Chrome. This package is async, but my Django project is sync, so it has to run inside trio.run()
It's also using uwsgi locks so that only one client can generate a PDF at a time (headless Chrome loads the page in a single virtual tab, so it can only do one at a time)
Here is the code:
import trio
import base64
import requests
from django.conf import settings
from trio_cdp import open_cdp, page, target
try:
import uwsgi
have_uwsgi = True
except ModuleNotFoundError:
have_uwsgi = False
async def render_pdf_task(url, params):
r = requests.get(url=settings.CDP_URL)
if r.status_code == 200:
out = r.json()
ws_url = out[0]['webSocketDebuggerUrl']
else:
return None
async with open_cdp(ws_url) as conn:
targets = await target.get_targets()
target_id = targets[0].target_id
async with conn.open_session(target_id) as session:
async with session.page_enable():
async with session.wait_for(page.LoadEventFired):
await page.navigate(url)
await trio.sleep(0.5)
pdf = await page.print_to_pdf(**params)
pdfdata = base64.b64decode(pdf[0])
await conn.aclose()
return pdfdata
def render_pdf(url, params):
if have_uwsgi:
uwsgi.lock(1)
pdfdata = trio.run(render_pdf_task, url, params)
if have_uwsgi:
uwsgi.unlock(1)
return pdfdata
Annoyingly, any uwsgi worker that has run this particular task will later hang on exit until it is forcibly killed. If uwsgi runs and nobody visits the PDF-generating page, all the uwsgi workers exit fine. And it is consistently the uwsgi workers that ran the render_pdf function that need to be killed.
For example, pid 20887 had run render_pdf, and later when trying to shut down uwsgi, this happened:
SIGINT/SIGQUIT received...killing workers...
worker 10 buried after 1 seconds
worker 9 buried after 1 seconds
worker 7 buried after 1 seconds
worker 6 buried after 1 seconds
worker 5 buried after 1 seconds
worker 4 buried after 1 seconds
worker 3 buried after 1 seconds
worker 2 buried after 1 seconds
worker 1 buried after 1 seconds
Tue Jan 25 22:44:42 2022 - worker 8 (pid: 20887) is taking too much time to die...NO MERCY !!!
worker 8 buried after 1 seconds
goodbye to uWSGI.
How can I fix this? Any help is much appreciated :)
I was able to solve this myself. uWSGI's handler for SIGINT is overridden by trio.run(), but only if trio.run() is in the main thread. I solved this by running it in another thread.
I had in the back of my mind the impression that if a celery worker gets a task , and it is retried - it remains in the worker's memory (with the eta) - and doesn't return to the queue.
resulting in that if a celery task is retried and the worker is busy working on different tasks , and that task eta arrives- it has to wait until it finishes processing the other tasks.
I tried looking in the documentation for something that is aligned with what I remembered , but I can't find anything.
what I did to try and check it is create two tasks.
#app.task(bind=True, name='task_that_holds_worker', rate_limit='4/m',
default_retry_delay=5 * 60,
max_retries=int(60 * 60 * 24 * 1 / (60 * 5)))
def task_that_holds_worker(self, *args, **kwargs):
import time
time.sleep(50000)
#app.task(bind=True, name='retried_task', rate_limit='2/m',
default_retry_delay=10 * 60,
max_retries=int(60 * 60 * 24 * 1 / (60 * 10)))
def retried_task(self, *args, **kwargs):
self.retry()
the simplest tasks , just to check that if a task is busy with other task - the retried task is not processed by another worker.
I then launched one worker - and triggered those two tasks in the following way:
from some_app import tasks
from some_app.celery_app import app
current_app = app.tasks
async_result = tasks.retried_task.delay()
import time
time.sleep(20)
async_result = tasks.task_that_holds_worker.delay()
the worker processed the retried task , and retried it,
and then moved to the task that sleeps.
I then launched another worker and i can see that it is not getting the 'retried' task, only the first worker.
each worker launched was launced with --prefetch-multiplier=1 --concurrency=1
Is there something wrong with the way I reproduced this?
or is this the way a celery retried task behaves?
Thanks in advance!
celery: 4.1.2
Python: 3.6.2
Rabbitmq Image: rabbitmq:3.6.9-management
Seems like this is an issue with tasks with eta. the first available worker counts down until the task eta and doesn't release it back to the queue. (prefetch count is increased and ignored)
https://github.com/celery/celery/issues/2541
There is an error with how you reproduced it. Unless you have a special broker, celery will always requeue a task retry request back to the broker. Workers do not retain any memory of which task they attempted, and there is no data added to the retry request that allows celery to route the task request back to the same worker. There is no guarantee or assurance that the same worker will retry a task that it has seen before. You can confirm this in the code for celery in celery/app.task.py
# get the signature of the task as called
S = self.signature_from_request(
request, args, kwargs,
countdown=countdown, eta=eta, retries=retries,
**options
)
if max_retries is not None and retries > max_retries:
if exc:
# On Py3: will augment any current exception with
# the exc' argument provided (raise exc from orig)
raise_with_context(exc)
raise self.MaxRetriesExceededError(
"Can't retry {0}[{1}] args:{2} kwargs:{3}".format(
self.name, request.id, S.args, S.kwargs))
ret = Retry(exc=exc, when=eta or countdown)
if is_eager:
# if task was executed eagerly using apply(),
# then the retry must also be executed eagerly.
S.apply().get()
if throw:
raise ret
return ret
try:
S.apply_async()
except Exception as exc:
raise Reject(exc, requeue=False)
if throw:
raise ret
return ret
I've bolded the part where you can see how the retry works. Celery gets the tasks request signature (this include the task name, and the arguments to the task, and sets the eta, countdown, and retries). And then celery will simply call apply_async, which under the hood will just queue up a new task request to the broker.
Your sample did not work because celery workers will often pull more than one task request off of the broker, so what likely happened is that the first worker grabbed the task off of the broker before the second worker had come online.
I'm using python Apscheduler to schedule my jobs. All my jobs are stored as a cron job and use the BackgroundScheduler. I've the following codes:
def letschedule():
jobstores = {
'default': SQLAlchemyJobStore(url=app_jobs_store)
}
executors = {
'default': ThreadPoolExecutor(20),
'processpool': ProcessPoolExecutor(5)
}
job_defaults = {
'coalesce': False,
'max_instances': 1,
'misfire_grace_time':1200
}
scheduler = BackgroundScheduler(jobstores=jobstores, executors=executors, job_defaults=job_defaults, timezone=utc)
#jobstores=jobstores, executors=executors, job_defaults=job_defaults, timezone=utc
return scheduler
And I start the job scheduler as follow in the app:
sch = letschedule()
sch.start()
log.info('the scheduler started')
And I've the following add job function.
def addjobs():
jobs = []
try:
sch.add_job(forecast_jobs, 'cron', day_of_week=os.environ.get("FORECAST_WEEKOFDAY"),
id="forecast",
replace_existing=False,week='1-53',hour=os.environ.get("FORECAST_HOUR"),
minute=os.environ.get("FORECAST_MINUTE"), timezone='UTC')
jobs.append({'job_id':'forecast', 'type':'weekly'})
log.info('the forecast added to the scheduler')
except BaseException as e:
log.info(e)
pass
try:
sch.add_job(insertcwhstock, 'cron',
id="cwhstock_data", day_of_week='0-6', replace_existing=False,hour=os.environ.get("CWHSTOCK_HOUR"),
minute=os.environ.get("CWHSTOCK_MINUTE"),
week='1-53',timezone='UTC')
jobs.append({'job_id':'cwhstock_data', 'type':'daily'})
log.info('the cwhstock job added to the scheduler')
except BaseException as e:
log.info(e)
pass
return json.dumps({'data':jobs})
I use this in the flask application, I call the /activatejobs and the jobs are added to the scheduler and it works fine. However when I restart the wsgi server, the jobs aren't started again, I've to remove the .sqlite file and add the jobs again. What I want is the jobs are supposed to be restarted automatically once the scheduler is started (if there are already jobs in the database.)
I tried to get such result trying some ways, but couldn't. Any help would be greatly appreciated. Thanks in Advance.
I also had the same problem using FastApi framework. I could solve the problem after add this code to my app.py:
scheduler = BackgroundScheduler()
pg_job_store = SQLAlchemyJobStore(engine=my_engine)
scheduler.add_jobstore(jobstore=pg_job_store, alias='sqlalchemy')
scheduler.start()
Adding this code, after I restart the application server I could see apscheduler logs searching for jobs:
2021-10-20 14:37:53,433 - apscheduler.scheduler - INFO => Scheduler started
2021-10-20 14:37:53,433 - apscheduler.scheduler - DEBUG => Looking for jobs to run
Jobstore default:
No scheduled jobs
Jobstore sqlalchemy:
remove_from_db_job (trigger: date[2021-10-20 14:38:00 -03], next run at: 2021-10-20 14:38:00 -03)
2021-10-20 14:37:53,443 - apscheduler.scheduler - DEBUG => Next wakeup is due at 2021-10-20 14:38:00-03:00 (in 6.565892 seconds)
It works for me.