Im trying to get cronjobs running on my elasticbeanstalk ec2 instance using django-cron.
I created a cronjob.cron:
*/5 * * * * root /opt/python/run/venv/bin/python3 /opt/python/current/app/manage.py runcrons >> /var/log/cronjobs.log
and added the file using commands:
container_commands:
06_runcrons:
command: "cat .ebextensions/cronjobs.cron > /etc/cron.d/djangocrons && chmod 644 /etc/cron.d/djangocrons"
leader_only: true
The cron seems to run but nothing gets printed into my /var/log/cronjobs.log (Which it should, every 5mins).
I cannot test the script on the ec2 instance manually, because my environment variables I added using aws:elasticbeanstalk:application:environment: are neither present with the ec2-user or root. Where do those environment variables get written to? Are they only present during the deploy?
The server itself works fine, so my db environment variables work.
Do I need to use another user for my cronjobs? How can I solve this?
Edit:
Removing the user like:
*/5 * * * * /opt/python/run/venv/bin/python3 /opt/python/current/app/manage.py runcrons >> /var/log/cronjobs.log
results in
(CRON) bad username (/etc/cron.d/djangocrons)
in my /var/log/cron
Solved it by adding the file that sources the env variables in /opt/python/current/env
*/5 * * * * root source /opt/python/current/env && /opt/python/run/venv/bin/python3 /opt/python/current/app/manage.py runcrons >> /var/log/cronjobs.log
Related
I have a Python script which uses environment variables. This script works exactly as planned when run directly; however, I would like to run it as a cron job every minute for the time being.
Currently in my cron. directory I have a file called scrapers containing:
* * * * * root /usr/bin/python3.5 /code/scraper.py
This runs the Python script but the script fails, as in the script I use two environment variables.
I read I should add SHELL=/bin/bash to the cron file, so I did, but this didn't help.
SHELL=/bin/bash
* * * * * root /usr/bin/python3.5 /code/scraper.py
Then I read
In the crontab, before you command, add . $HOME/.profile.
SHELL=/bin/bash
* * * * * . $HOME/.profile; root /usr/bin/python3.5 /code/scraper.py
but this caused the cron to stop running altogether. What is the best way of 'sending' the env variables to the cron?
Instead of executing the whole ~/.profile what I'd do is move the variables that must be shared between your cron jobs and the account that has the profile, then I'd source these both in ~/.profile and in the cron job.
The last attempt you show in the question is not properly formatted. The user id should be coming right after the scheduling information, but you've added the sourcing of the profile before the user id, which surely cannot work.
Here's an example setup that I've tested here:
*/1 * * * * someuser . /tmp/t10/setenv && /usr/bin/python /tmp/t10/test.py
I've set it to execute every minute for testing purposes. Replace someuser with something that makes sense. The /tmp/t10/setenv script I used had this:
export FOO=foovalue
export BAR=barvalue
The /tmp/t10/test.py file had this:
import os
print os.environ["FOO"], os.environ["BAR"]
My cron emails me the output of the scripts it runs. I got an email with this output:
foovalue barvalue
You can set the env variable inline:
* * * * * root ENV_VAR=VALUE /usr/bin/python3.5 /code/scraper.py
Another way is use honcho that you can pass a file with env variables.
honcho -e /path/to/.env run /code/scraper.py
You can specify your two environment variables by this:
* * * * * root env A=1 B=2 /usr/bin/python3.5 /code/scraper.py
env is a system program that runs a specified program with additional variables:
$ env A=1 B=2 /bin/sh -c 'echo $A$B' # or just 'sh': would search in $PATH
12
You can add it to the top of your crontab and keep it out of version control. Let's say the environment variable causing you difficulty is export DJANGO_SECRET_KEY="FOOBAR_1241243124312341234":
crontab
DJANGO_SECRET_KEY="FOOBAR_1241243124312341234"
SCRIPT_NAME = my_cool_script
20 21 * * 1-5 bash ~/git_repo/cronjobs/$SCRIPT_NAME.sh 2&>1 | tee ~/git_repo/cronjobs/logs/$SCRIPT_NAME.log
my_cool_script.sh
#!/usr/bin/env bash
~/anaconda3/envs/django/bin/python ~/git_repo/django_project/manage.py run_command
This has worked well for me when the environment variables in question need to be kept secret and the loading of existing .bashrc does not play nice for whatever reason.
This is one of the approach I like, write a script to set environment and execute the script with its parameters as its parameters
set_env_to_process.sh
#!/usr/bin/env bash
echo "TEST_VAR before export is: <$TEST_VAR>"
export TEST_VAR=/opt/loca/netcdf
echo "TEST_VAR after export is: <$TEST_VAR>"
export PATH=$PATH:/usr/bin/python3.5
export PYTHTONPATH=$PYTHONPATH:/my/installed/pythonpath
# execute command and its parameters as input for this script
if [ $# -eq 0 ]; then
echo "No command to execute"
else
echo "Execute commands with its parameters: $#"
eval $#
fi
usage
/usr/bin/python3.5 /code/scraper.pyare taken as input for set_env_to_process.sh
set_env_to_process.sh set the correct env for script to run
It could be used as command line, cron, sudo, ssh to setup env
* * * * * root set_env_to_process.sh /usr/bin/python3.5 /code/scraper.py
When I run python manage.py runcrons, the cron job successfully executes (prints 'Executed'). But it doesn't do it automatically. It's supposed to execute every minute as I've stated in the Class. Here's the code:
settings.py
CRON_CLASSES = [
"app.views.MyCronJob"
]
app.views
class MyCronJob(CronJobBase):
RUN_EVERY_MINS = 1
schedule = Schedule(run_every_mins=RUN_EVERY_MINS)
code = 'my_app.my_cron_job' # not sure what this is supposed to be?
def do(self):
print('Executed')
Any idea?
you also need to set up the crontab entry for the cron:
> crontab -e
*/5 * * * * source /home/ubuntu/.bashrc && source /home/ubuntu/work/your-project/bin/activate && python /home/ubuntu/work/your-project/src/manage.py runcrons > /home/ubuntu/cronjob.log
taken from official documentation
+1 for Ayush's answer. This worked for me.
crontab -e
This will open the editor, copy paste the following line at the bottom and save & close.
*/5 * * * * source /home/ubuntu/.bashrc && source /home/ubuntu/work/your-project/bin/activate && python /home/ubuntu/work/your-project/src/manage.py runcrons > /home/ubuntu/cronjob.log
In addition to this try rerunning these in your Django project directory:
python manage.py runcrons --force
service cron start
I have a Python script which uses environment variables. This script works exactly as planned when run directly; however, I would like to run it as a cron job every minute for the time being.
Currently in my cron. directory I have a file called scrapers containing:
* * * * * root /usr/bin/python3.5 /code/scraper.py
This runs the Python script but the script fails, as in the script I use two environment variables.
I read I should add SHELL=/bin/bash to the cron file, so I did, but this didn't help.
SHELL=/bin/bash
* * * * * root /usr/bin/python3.5 /code/scraper.py
Then I read
In the crontab, before you command, add . $HOME/.profile.
SHELL=/bin/bash
* * * * * . $HOME/.profile; root /usr/bin/python3.5 /code/scraper.py
but this caused the cron to stop running altogether. What is the best way of 'sending' the env variables to the cron?
Instead of executing the whole ~/.profile what I'd do is move the variables that must be shared between your cron jobs and the account that has the profile, then I'd source these both in ~/.profile and in the cron job.
The last attempt you show in the question is not properly formatted. The user id should be coming right after the scheduling information, but you've added the sourcing of the profile before the user id, which surely cannot work.
Here's an example setup that I've tested here:
*/1 * * * * someuser . /tmp/t10/setenv && /usr/bin/python /tmp/t10/test.py
I've set it to execute every minute for testing purposes. Replace someuser with something that makes sense. The /tmp/t10/setenv script I used had this:
export FOO=foovalue
export BAR=barvalue
The /tmp/t10/test.py file had this:
import os
print os.environ["FOO"], os.environ["BAR"]
My cron emails me the output of the scripts it runs. I got an email with this output:
foovalue barvalue
You can set the env variable inline:
* * * * * root ENV_VAR=VALUE /usr/bin/python3.5 /code/scraper.py
Another way is use honcho that you can pass a file with env variables.
honcho -e /path/to/.env run /code/scraper.py
You can specify your two environment variables by this:
* * * * * root env A=1 B=2 /usr/bin/python3.5 /code/scraper.py
env is a system program that runs a specified program with additional variables:
$ env A=1 B=2 /bin/sh -c 'echo $A$B' # or just 'sh': would search in $PATH
12
You can add it to the top of your crontab and keep it out of version control. Let's say the environment variable causing you difficulty is export DJANGO_SECRET_KEY="FOOBAR_1241243124312341234":
crontab
DJANGO_SECRET_KEY="FOOBAR_1241243124312341234"
SCRIPT_NAME = my_cool_script
20 21 * * 1-5 bash ~/git_repo/cronjobs/$SCRIPT_NAME.sh 2&>1 | tee ~/git_repo/cronjobs/logs/$SCRIPT_NAME.log
my_cool_script.sh
#!/usr/bin/env bash
~/anaconda3/envs/django/bin/python ~/git_repo/django_project/manage.py run_command
This has worked well for me when the environment variables in question need to be kept secret and the loading of existing .bashrc does not play nice for whatever reason.
This is one of the approach I like, write a script to set environment and execute the script with its parameters as its parameters
set_env_to_process.sh
#!/usr/bin/env bash
echo "TEST_VAR before export is: <$TEST_VAR>"
export TEST_VAR=/opt/loca/netcdf
echo "TEST_VAR after export is: <$TEST_VAR>"
export PATH=$PATH:/usr/bin/python3.5
export PYTHTONPATH=$PYTHONPATH:/my/installed/pythonpath
# execute command and its parameters as input for this script
if [ $# -eq 0 ]; then
echo "No command to execute"
else
echo "Execute commands with its parameters: $#"
eval $#
fi
usage
/usr/bin/python3.5 /code/scraper.pyare taken as input for set_env_to_process.sh
set_env_to_process.sh set the correct env for script to run
It could be used as command line, cron, sudo, ssh to setup env
* * * * * root set_env_to_process.sh /usr/bin/python3.5 /code/scraper.py
I have written a custom management command post_message when I run python manage.py post_message the command executes well.
Now I want this command to run every 10 minutes.
I have a virtual envrironment.
I have a file - msg.cron the contents of the file are as follows -
#!SHELL=/bin/bash
*/10 * * * * source /home/username/Envs/project_name/bin/activate && /home/username/Code/project_name/manage.py post_message > /dev/null
I have done chmod +x on msg.cron
Having done this, I added
crontab msg.cron
Now when I do crontab -l contents of msg.cron are shown.
But the management command are not being run, what am I missing?
You don't need to activate the virtualenv in this case. You can just use the python in the bin directory of the virtualenv.
*/10 * * * * source /home/username/Envs/project_name/bin/python /home/username/Code/project_name/manage.py post_message > /dev/null
How to run in crontab
*/1 * * * * /home/user/Desktop/job/dp/ python manage.py statistics
with virtual env? I need to activate virtualenv first(Otherwise it does not work)
This is my virtual env:
source job/bin/activate
EDITED:
Try something like this:
*/1 * * * * . /path-to-env/bin/activate && /home/user/Desktop/job/dp/manage.py statistics
This should be read as: activate the env and if that was successful, excute the manage.py script. Since manage.py is supposed to have a python shebang and the virtual env sets the correct python interpreter, this should work.
Apparently cron usually runs with /bin/sh which does not know the source command. So one option is to use a dot as a source replacement. Another to set /bin/bash in the cron file:
SHELL=/bin/bash
*/1 * * * * source /path-to-env/bin/activate && /home/user/Desktop/job/dp/manage.py statistics
Read more about this issue at:
http://codeinthehole.com/writing/running-django-cronjobs-within-a-virtualenv/
The article doesn't mention that source can be replaced by a ., but i've just tried it and it worked for me. So you have several options to choose from now, the article even has others. ;)
Use something like ~/envs/someenv/lib/python /path/to/your/script
In your situation it will look like
*/1 * * * * ~/envs/someenv/lib/python /home/user/Desktop/job/dp/manage.py statistics