How to run in crontab
*/1 * * * * /home/user/Desktop/job/dp/ python manage.py statistics
with virtual env? I need to activate virtualenv first(Otherwise it does not work)
This is my virtual env:
source job/bin/activate
EDITED:
Try something like this:
*/1 * * * * . /path-to-env/bin/activate && /home/user/Desktop/job/dp/manage.py statistics
This should be read as: activate the env and if that was successful, excute the manage.py script. Since manage.py is supposed to have a python shebang and the virtual env sets the correct python interpreter, this should work.
Apparently cron usually runs with /bin/sh which does not know the source command. So one option is to use a dot as a source replacement. Another to set /bin/bash in the cron file:
SHELL=/bin/bash
*/1 * * * * source /path-to-env/bin/activate && /home/user/Desktop/job/dp/manage.py statistics
Read more about this issue at:
http://codeinthehole.com/writing/running-django-cronjobs-within-a-virtualenv/
The article doesn't mention that source can be replaced by a ., but i've just tried it and it worked for me. So you have several options to choose from now, the article even has others. ;)
Use something like ~/envs/someenv/lib/python /path/to/your/script
In your situation it will look like
*/1 * * * * ~/envs/someenv/lib/python /home/user/Desktop/job/dp/manage.py statistics
Related
I have a Python script which uses environment variables. This script works exactly as planned when run directly; however, I would like to run it as a cron job every minute for the time being.
Currently in my cron. directory I have a file called scrapers containing:
* * * * * root /usr/bin/python3.5 /code/scraper.py
This runs the Python script but the script fails, as in the script I use two environment variables.
I read I should add SHELL=/bin/bash to the cron file, so I did, but this didn't help.
SHELL=/bin/bash
* * * * * root /usr/bin/python3.5 /code/scraper.py
Then I read
In the crontab, before you command, add . $HOME/.profile.
SHELL=/bin/bash
* * * * * . $HOME/.profile; root /usr/bin/python3.5 /code/scraper.py
but this caused the cron to stop running altogether. What is the best way of 'sending' the env variables to the cron?
Instead of executing the whole ~/.profile what I'd do is move the variables that must be shared between your cron jobs and the account that has the profile, then I'd source these both in ~/.profile and in the cron job.
The last attempt you show in the question is not properly formatted. The user id should be coming right after the scheduling information, but you've added the sourcing of the profile before the user id, which surely cannot work.
Here's an example setup that I've tested here:
*/1 * * * * someuser . /tmp/t10/setenv && /usr/bin/python /tmp/t10/test.py
I've set it to execute every minute for testing purposes. Replace someuser with something that makes sense. The /tmp/t10/setenv script I used had this:
export FOO=foovalue
export BAR=barvalue
The /tmp/t10/test.py file had this:
import os
print os.environ["FOO"], os.environ["BAR"]
My cron emails me the output of the scripts it runs. I got an email with this output:
foovalue barvalue
You can set the env variable inline:
* * * * * root ENV_VAR=VALUE /usr/bin/python3.5 /code/scraper.py
Another way is use honcho that you can pass a file with env variables.
honcho -e /path/to/.env run /code/scraper.py
You can specify your two environment variables by this:
* * * * * root env A=1 B=2 /usr/bin/python3.5 /code/scraper.py
env is a system program that runs a specified program with additional variables:
$ env A=1 B=2 /bin/sh -c 'echo $A$B' # or just 'sh': would search in $PATH
12
You can add it to the top of your crontab and keep it out of version control. Let's say the environment variable causing you difficulty is export DJANGO_SECRET_KEY="FOOBAR_1241243124312341234":
crontab
DJANGO_SECRET_KEY="FOOBAR_1241243124312341234"
SCRIPT_NAME = my_cool_script
20 21 * * 1-5 bash ~/git_repo/cronjobs/$SCRIPT_NAME.sh 2&>1 | tee ~/git_repo/cronjobs/logs/$SCRIPT_NAME.log
my_cool_script.sh
#!/usr/bin/env bash
~/anaconda3/envs/django/bin/python ~/git_repo/django_project/manage.py run_command
This has worked well for me when the environment variables in question need to be kept secret and the loading of existing .bashrc does not play nice for whatever reason.
This is one of the approach I like, write a script to set environment and execute the script with its parameters as its parameters
set_env_to_process.sh
#!/usr/bin/env bash
echo "TEST_VAR before export is: <$TEST_VAR>"
export TEST_VAR=/opt/loca/netcdf
echo "TEST_VAR after export is: <$TEST_VAR>"
export PATH=$PATH:/usr/bin/python3.5
export PYTHTONPATH=$PYTHONPATH:/my/installed/pythonpath
# execute command and its parameters as input for this script
if [ $# -eq 0 ]; then
echo "No command to execute"
else
echo "Execute commands with its parameters: $#"
eval $#
fi
usage
/usr/bin/python3.5 /code/scraper.pyare taken as input for set_env_to_process.sh
set_env_to_process.sh set the correct env for script to run
It could be used as command line, cron, sudo, ssh to setup env
* * * * * root set_env_to_process.sh /usr/bin/python3.5 /code/scraper.py
Hello i have been following stackoverflow posts and im unable to get the crontab to run my python scripts in ubuntu.
After talking with the user below, i have modified my file but am not getting a "moduleNotFoundError: No module named aioredis"
when i deactivate my py3.9 venv environment, and call python feed_ingestor_to_postgres.py i get the same error. so clearly the pipenv isn't activating for me. any suggestions?
my launcher.sh file:
#!/bin/bash
WORKDIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
cd $WORKDIR
activate(){
../../env/bin/activate
}
python3 feed_ingestor_to_postgres.py
my crontab file:
* * * * * echo "Great cron, awesome job!" > /var/log/nonsense
* * * * * /home/ubuntu/phobos/phobos-trading-engine/exchange_feeds/launcher.sh
I have no idea what else to do here. any help would be appreciated
To manage python scripts from the crontab more easily, I would advise to use a simple shell wrapper script, for example "launcher.sh" which will be called by the crontab. The launcher will cd to the script location, manage the virtualenv, or possibly set env vars if required:
#!/bin/bash
WORKDIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
cd $WORKDIR
source ../../env/bin/activate
python feed_ingestor_to_postgres.py
Then in the crontab, call the launcher instead of your python script:
* * * * * ~/path/to/launcher.sh
It will be much more flexible than editing the crontab directly
Usually I SSH into my EC2 instance and run:
source MYVENV/bin/activate
How do I set my cronjob to activate the virtual environment? My Django script requires ENVIRONMENT variables that are stores in ~/.bash_profile
I tried following the steps here to no avail
Cron and virtualenv
SHELL=/bin/bash
*/1 * * * * root source /home/ec2-user/MYVENV/activate && python /home/script.py
This is my current setup above.
I get this following error in the log:
/bin/bash: root: command not found
create a shell script
eg scripts.sh
#!/bin/bash
source /home/user/MYVENV/bin/activate
python /path/to/file/script.py
Then in cron put
*/1 * * * * bash /path/to/shell/script/scripts.sh
The script will load all your environment variables and execute from the python in your environment
you can just run the python interpretor from your environment directly eg
MYENV/bin/python script.py
to find out what is the directory to your environment python interpretor, change into the virtual env then run
which python
in your case, this should become
*/1 * * * * /home/ec2-user/MYVENV/python /home/script.py
you can create a single wrapper bash script for executing your Django script. See the example below.
#!/bin/bash -l // this should pick up your ~/.bash_profile environment variables
cd /path to project dir/ // set it up if your project is not in python path
source /Users/<user>/.virtualenvs/dslab/bin/activate // this activates your environment
python /home/script.py // run your script
I have a Python script which uses environment variables. This script works exactly as planned when run directly; however, I would like to run it as a cron job every minute for the time being.
Currently in my cron. directory I have a file called scrapers containing:
* * * * * root /usr/bin/python3.5 /code/scraper.py
This runs the Python script but the script fails, as in the script I use two environment variables.
I read I should add SHELL=/bin/bash to the cron file, so I did, but this didn't help.
SHELL=/bin/bash
* * * * * root /usr/bin/python3.5 /code/scraper.py
Then I read
In the crontab, before you command, add . $HOME/.profile.
SHELL=/bin/bash
* * * * * . $HOME/.profile; root /usr/bin/python3.5 /code/scraper.py
but this caused the cron to stop running altogether. What is the best way of 'sending' the env variables to the cron?
Instead of executing the whole ~/.profile what I'd do is move the variables that must be shared between your cron jobs and the account that has the profile, then I'd source these both in ~/.profile and in the cron job.
The last attempt you show in the question is not properly formatted. The user id should be coming right after the scheduling information, but you've added the sourcing of the profile before the user id, which surely cannot work.
Here's an example setup that I've tested here:
*/1 * * * * someuser . /tmp/t10/setenv && /usr/bin/python /tmp/t10/test.py
I've set it to execute every minute for testing purposes. Replace someuser with something that makes sense. The /tmp/t10/setenv script I used had this:
export FOO=foovalue
export BAR=barvalue
The /tmp/t10/test.py file had this:
import os
print os.environ["FOO"], os.environ["BAR"]
My cron emails me the output of the scripts it runs. I got an email with this output:
foovalue barvalue
You can set the env variable inline:
* * * * * root ENV_VAR=VALUE /usr/bin/python3.5 /code/scraper.py
Another way is use honcho that you can pass a file with env variables.
honcho -e /path/to/.env run /code/scraper.py
You can specify your two environment variables by this:
* * * * * root env A=1 B=2 /usr/bin/python3.5 /code/scraper.py
env is a system program that runs a specified program with additional variables:
$ env A=1 B=2 /bin/sh -c 'echo $A$B' # or just 'sh': would search in $PATH
12
You can add it to the top of your crontab and keep it out of version control. Let's say the environment variable causing you difficulty is export DJANGO_SECRET_KEY="FOOBAR_1241243124312341234":
crontab
DJANGO_SECRET_KEY="FOOBAR_1241243124312341234"
SCRIPT_NAME = my_cool_script
20 21 * * 1-5 bash ~/git_repo/cronjobs/$SCRIPT_NAME.sh 2&>1 | tee ~/git_repo/cronjobs/logs/$SCRIPT_NAME.log
my_cool_script.sh
#!/usr/bin/env bash
~/anaconda3/envs/django/bin/python ~/git_repo/django_project/manage.py run_command
This has worked well for me when the environment variables in question need to be kept secret and the loading of existing .bashrc does not play nice for whatever reason.
This is one of the approach I like, write a script to set environment and execute the script with its parameters as its parameters
set_env_to_process.sh
#!/usr/bin/env bash
echo "TEST_VAR before export is: <$TEST_VAR>"
export TEST_VAR=/opt/loca/netcdf
echo "TEST_VAR after export is: <$TEST_VAR>"
export PATH=$PATH:/usr/bin/python3.5
export PYTHTONPATH=$PYTHONPATH:/my/installed/pythonpath
# execute command and its parameters as input for this script
if [ $# -eq 0 ]; then
echo "No command to execute"
else
echo "Execute commands with its parameters: $#"
eval $#
fi
usage
/usr/bin/python3.5 /code/scraper.pyare taken as input for set_env_to_process.sh
set_env_to_process.sh set the correct env for script to run
It could be used as command line, cron, sudo, ssh to setup env
* * * * * root set_env_to_process.sh /usr/bin/python3.5 /code/scraper.py
I have written a custom management command post_message when I run python manage.py post_message the command executes well.
Now I want this command to run every 10 minutes.
I have a virtual envrironment.
I have a file - msg.cron the contents of the file are as follows -
#!SHELL=/bin/bash
*/10 * * * * source /home/username/Envs/project_name/bin/activate && /home/username/Code/project_name/manage.py post_message > /dev/null
I have done chmod +x on msg.cron
Having done this, I added
crontab msg.cron
Now when I do crontab -l contents of msg.cron are shown.
But the management command are not being run, what am I missing?
You don't need to activate the virtualenv in this case. You can just use the python in the bin directory of the virtualenv.
*/10 * * * * source /home/username/Envs/project_name/bin/python /home/username/Code/project_name/manage.py post_message > /dev/null