I have a crontab running every hour. The user running it has environment variabless in the .bash_profile that work when the user runs the job from the terminal, however, obviously these don't get picked up by crontab when it runs.
I've tried setting them in .profile and .bashrc but they still don't seem to get picked up. Does anyone know where I can put environment vars that crontab can pick up?
You can define environment variables in the crontab itself when running crontab -e from the command line.
LANG=nb_NO.UTF-8
LC_ALL=nb_NO.UTF-8
# m h dom mon dow command
* * * * * sleep 5s && echo "yo"
This feature is only available to certain implementations of cron. Ubuntu and Debian currently use vixie-cron which allows these to be declared in the crontab file (also GNU mcron).
Archlinux and RedHat use cronie which does not allow environment variables to be declared and will throw syntax errors in the cron.log. Workaround can be done per-entry:
# m h dom mon dow command
* * * * * export LC_ALL=nb_NO.UTF-8; sleep 5s && echo "yo"
I got one more solution for this problem:
0 5 * * * . $HOME/.profile; /path/to/command/to/run
In this case it will pick all the environment variable defined in your $HOME/.profile file.
Of course $HOME is also not set, you have to replace it with the full path of your $HOME.
Setting vars in /etc/environment also worked for me in Ubuntu. As of 12.04, variables in /etc/environment are loaded for cron.
Have 'cron' run a shell script that sets the environment before running the command.
Always.
# #(#)$Id: crontab,v 4.2 2007/09/17 02:41:00 jleffler Exp $
# Crontab file for Home Directory for Jonathan Leffler (JL)
#-----------------------------------------------------------------------------
#Min Hour Day Month Weekday Command
#-----------------------------------------------------------------------------
0 * * * * /usr/bin/ksh /work1/jleffler/bin/Cron/hourly
1 1 * * * /usr/bin/ksh /work1/jleffler/bin/Cron/daily
23 1 * * 1-5 /usr/bin/ksh /work1/jleffler/bin/Cron/weekday
2 3 * * 0 /usr/bin/ksh /work1/jleffler/bin/Cron/weekly
21 3 1 * * /usr/bin/ksh /work1/jleffler/bin/Cron/monthly
The scripts in ~/bin/Cron are all links to a single script, 'runcron', which looks like:
: "$Id: runcron.sh,v 2.1 2001/02/27 00:53:22 jleffler Exp $"
#
# Commands to be performed by Cron (no debugging options)
# Set environment -- not done by cron (usually switches HOME)
. $HOME/.cronfile
base=`basename $0`
cmd=${REAL_HOME:-/real/home}/bin/$base
if [ ! -x $cmd ]
then cmd=${HOME}/bin/$base
fi
exec $cmd ${#:+"$#"}
(Written using an older coding standard - nowadays, I'd use a shebang '#!' at the start.)
The '~/.cronfile' is a variation on my profile for use by cron - rigorously non-interactive and no echoing for the sake of being noisy. You could arrange to execute the .profile and so on instead. (The REAL_HOME stuff is an artefact of my environment - you can pretend it is the same as $HOME.)
So, this code reads the appropriate environment and then executes the non-Cron version of the command from my home directory. So, for example, my 'weekday' command looks like:
: "#(#)$Id: weekday.sh,v 1.10 2007/09/17 02:42:03 jleffler Exp $"
#
# Commands to be done each weekday
# Update ICSCOPE
n.updics
The 'daily' command is simpler:
: "#(#)$Id: daily.sh,v 1.5 1997/06/02 22:04:21 johnl Exp $"
#
# Commands to be done daily
# Nothing -- most things are done on weekdays only
exit 0
If you start the scripts you are executing through cron with:
#!/bin/bash -l
They should pick up your ~/.bash_profile environment variables
Expanding on #carestad example, which I find easier, is to run the script with cron and have the environment in the script.
In crontab -e file:
SHELL=/bin/bash
*/1 * * * * $HOME/cron_job.sh
In cron_job.sh file:
#!/bin/bash
source $HOME/.bash_profile
some_other_cmd
Any command after the source of .bash_profile will have your environment as if you logged in.
Whatever you set in crontab will be available in the cronjobs, both directly and using the variables in the scripts.
Use them in the definition of the cronjob
You can configure crontab so that it sets variables that then the can cronjob use:
$ crontab -l
myvar="hi man"
* * * * * echo "$myvar. date is $(date)" >> /tmp/hello
Now the file /tmp/hello shows things like:
$ cat /tmp/hello
hi man. date is Thu May 12 12:10:01 CEST 2016
hi man. date is Thu May 12 12:11:01 CEST 2016
Use them in the script run by cronjob
You can configure crontab so that it sets variables that then the scripts can use:
$ crontab -l
myvar="hi man"
* * * * * /bin/bash /tmp/myscript.sh
And say script /tmp/myscript.sh is like this:
echo "Now is $(date). myvar=$myvar" >> /tmp/myoutput.res
It generates a file /tmp/myoutput.res showing:
$ cat /tmp/myoutput.res
Now is Thu May 12 12:07:01 CEST 2016. myvar=hi man
Now is Thu May 12 12:08:01 CEST 2016. myvar=hi man
...
For me I had to set the environment variable for a php application. I resolved it by adding the following code to my crontab.
$ sudo crontab -e
crontab:
ENVIRONMENT_VAR=production
* * * * * /home/deploy/my_app/cron/cron.doSomethingWonderful.php
and inside doSomethingWonderful.php I could get the environment value with:
<?php
echo $_SERVER['ENVIRONMENT_VAR']; # => "production"
I hope this helps!
Instead of
0 * * * * sh /my/script.sh
Use bash -l -c
0 * * * * bash -l -c 'sh /my/script.sh'
You can also prepend your command with env to inject Environment variables like so:
0 * * * * env VARIABLE=VALUE /usr/bin/mycommand
Expanding on #Robert Brisita has just expand , also if you don't want to set up all the variables of the profile in the script, you can select the variables to export on the top of the script
In crontab -e file:
SHELL=/bin/bash
*/1 * * * * /Path/to/script/script.sh
In script.sh
#!/bin/bash
export JAVA_HOME=/path/to/jdk
some-other-command
I'm using Oh-my-zsh in my macbook so I've tried many things to get the crontab task runs but finally, my solution was prepending the .zshrc before the command to run.
*/30 * * * * . $HOME/.zshrc; node /path/for/my_script.js
This task runs every 30 minutes and uses .zshrc profile to execute my node command.
Don't forget to use the dot before the $HOME var.
I tried most of the provided solutions, but nothing worked at first. It turns out, though, that it wasn't the solutions that failed to work. Apparently, my ~/.bashrc file starts with the following block of code:
case $- in
*i*) ;;
*) return;;
esac
This basically is a case statement that checks the current set of options in the current shell to determine that the shell is running interactively.
If the shell happens to be running interactively, then it moves on to sourcing the ~/.bashrc file.
However, in a shell invoked by cron, the $- variable doesn't contain the i value which indicates interactivity.
Therefore, the ~/.bashrc file never gets sourced fully. As a result, the environment variables never got set.
If this happens to be your issue, feel free to comment out the block of code as follows and try again:
# case $- in
# *i*) ;;
# *) return;;
# esac
I hope this turns out useful
Unfortunately, crontabs have a very limited environment variables scope, thus you need to export them every time the corntab runs.
An easy approach would be the following example, suppose you've your env vars in a file called env, then:
* * * * * . ./env && /path/to_your/command
this part . ./env will export them and then they're used within the same scope of your command
Another way - inspired by this this answer - to "inject" variables is the following (fcron example):
%daily 00 12 \
set -a; \
. /path/to/file/containing/vars; \
set +a; \
/path/to/script/using/vars
From help set:
-a Mark variables which are modified or created for export.
Using + rather than - causes these flags to be turned off.
So everything in between set - and set + gets exported to env and is then available for other scripts, etc. Without using set the variables get sourced but live in set only.
Aside from that it's also useful to pass variables when a program requires a non-root account to run but you'd need some variables inside that other user's environment. Below is an example passing in nullmailer vars to format the e-mail header:
su -s /bin/bash -c "set -a; \
. /path/to/nullmailer-vars; \
set +a; \
/usr/sbin/logcheck" logcheck
All the above solutions work fine.
It will create issues when there are any special characters in your environment variable.
I have found the solution:
eval $(printenv | awk -F= '{print "export " "\""$1"\"""=""\""$2"\"" }' >> /etc/profile)
For me I had to specify path in my NodeJS file.
// did not work!!!!!
require('dotenv').config()
instead
// DID WORK!!
require('dotenv').config({ path: '/full/custom/path/to/your/.env' })
I found this issue while looking at a similar problem that matched the title, but I am stuck with the environment file syntax that systemd or docker use:
FOO=bar
BAZ=qux
This won't work for Vishal's excellent answer because they aren't bash scripts (note the lack of export).
The solution I've used is to read each line into xargs and export them before running the command:
0 5 * * * export $(xargs < $HOME/.env); /path/to/command/to/run
Set Globally env
sudo sh -c "echo MY_GLOBAL_ENV_TO_MY_CURRENT_DIR=$(pwd)" >> /etc/environment"
Add scheduled job to start a script
crontab -e
*/5 * * * * sh -c "$MY_GLOBAL_ENV_TO_MY_CURRENT_DIR/start.sh"
=)
what worked for me (debian based):
create a file with all the needed env var :
#!/bin/bash
env | grep VAR1= > /etc/environment
env | grep VAR2= >> /etc/environment
env | grep VAR3= >> /etc/environment
then build the crontab content, by calling the env file before calling the script that needs it, therefore start the cron service
(crontab -l ; echo '* * * * * . /etc/environment; /usr/local/bin/python /mycode.py >> /var/log/cron-1.log 2>&1') | crontab
service cron start
nb : for python use case, be sure to call the whole python path, else wrong python could be invocated, generating non-sense syntax error
Related
I have a Python script which uses environment variables. This script works exactly as planned when run directly; however, I would like to run it as a cron job every minute for the time being.
Currently in my cron. directory I have a file called scrapers containing:
* * * * * root /usr/bin/python3.5 /code/scraper.py
This runs the Python script but the script fails, as in the script I use two environment variables.
I read I should add SHELL=/bin/bash to the cron file, so I did, but this didn't help.
SHELL=/bin/bash
* * * * * root /usr/bin/python3.5 /code/scraper.py
Then I read
In the crontab, before you command, add . $HOME/.profile.
SHELL=/bin/bash
* * * * * . $HOME/.profile; root /usr/bin/python3.5 /code/scraper.py
but this caused the cron to stop running altogether. What is the best way of 'sending' the env variables to the cron?
Instead of executing the whole ~/.profile what I'd do is move the variables that must be shared between your cron jobs and the account that has the profile, then I'd source these both in ~/.profile and in the cron job.
The last attempt you show in the question is not properly formatted. The user id should be coming right after the scheduling information, but you've added the sourcing of the profile before the user id, which surely cannot work.
Here's an example setup that I've tested here:
*/1 * * * * someuser . /tmp/t10/setenv && /usr/bin/python /tmp/t10/test.py
I've set it to execute every minute for testing purposes. Replace someuser with something that makes sense. The /tmp/t10/setenv script I used had this:
export FOO=foovalue
export BAR=barvalue
The /tmp/t10/test.py file had this:
import os
print os.environ["FOO"], os.environ["BAR"]
My cron emails me the output of the scripts it runs. I got an email with this output:
foovalue barvalue
You can set the env variable inline:
* * * * * root ENV_VAR=VALUE /usr/bin/python3.5 /code/scraper.py
Another way is use honcho that you can pass a file with env variables.
honcho -e /path/to/.env run /code/scraper.py
You can specify your two environment variables by this:
* * * * * root env A=1 B=2 /usr/bin/python3.5 /code/scraper.py
env is a system program that runs a specified program with additional variables:
$ env A=1 B=2 /bin/sh -c 'echo $A$B' # or just 'sh': would search in $PATH
12
You can add it to the top of your crontab and keep it out of version control. Let's say the environment variable causing you difficulty is export DJANGO_SECRET_KEY="FOOBAR_1241243124312341234":
crontab
DJANGO_SECRET_KEY="FOOBAR_1241243124312341234"
SCRIPT_NAME = my_cool_script
20 21 * * 1-5 bash ~/git_repo/cronjobs/$SCRIPT_NAME.sh 2&>1 | tee ~/git_repo/cronjobs/logs/$SCRIPT_NAME.log
my_cool_script.sh
#!/usr/bin/env bash
~/anaconda3/envs/django/bin/python ~/git_repo/django_project/manage.py run_command
This has worked well for me when the environment variables in question need to be kept secret and the loading of existing .bashrc does not play nice for whatever reason.
This is one of the approach I like, write a script to set environment and execute the script with its parameters as its parameters
set_env_to_process.sh
#!/usr/bin/env bash
echo "TEST_VAR before export is: <$TEST_VAR>"
export TEST_VAR=/opt/loca/netcdf
echo "TEST_VAR after export is: <$TEST_VAR>"
export PATH=$PATH:/usr/bin/python3.5
export PYTHTONPATH=$PYTHONPATH:/my/installed/pythonpath
# execute command and its parameters as input for this script
if [ $# -eq 0 ]; then
echo "No command to execute"
else
echo "Execute commands with its parameters: $#"
eval $#
fi
usage
/usr/bin/python3.5 /code/scraper.pyare taken as input for set_env_to_process.sh
set_env_to_process.sh set the correct env for script to run
It could be used as command line, cron, sudo, ssh to setup env
* * * * * root set_env_to_process.sh /usr/bin/python3.5 /code/scraper.py
This is how I configure crontab (by using crontab -e)
* * * * * /home/jeff/Desktop/scripts/job_pull_queue.sh >> /home/jeff/Desktop/scripts/log.txt
This is the content of /home/jeff/Desktop/scripts/job_pull_queue.sh
#!/bin/bash
echo "Running job_pull_queue.sh # $(date)"
cd /home/jeff/Documents/code/some_project
echo $(printenv)
/home/jeff/miniconda3/bin/python -m util.main
Now the problem is, when running ./job_pull_queue.sh in the terminal, it works, but I can tell from the log file that crontab never executes that last line /home/jeff/miniconda3/bin/python -m util.main (I can see the result from the previous echo in the log file, but not the python script itself), what happened? How do I fix it?
Update: here's the result from printenv when ran by crontab
SHELL=/bin/sh PWD=/home/jeff/Documents/code/some_project LOGNAME=jeff HOME=/home/jeff LANG=en_US.UTF-8 SHLVL=0 PATH=/usr/bin:/bin OLDPWD=/home/jeff _=/usr/bin/printenv
Ok...
My Python script reads several env variables from my user profile, and of course, these variables don't exist when crontab is running the script...
And I don't have detection/logging in place so I didn't know env variables are missing.
I'm new to crontab and having some trouble.
I know that the set up is
* * * * * command to execute
and that the * stand for min, hour, day of month, month, day of week respectively
Usually, to run my code in the terminal I do
source /Users/mmmm/PycharmProjects/kt/venv/bin/activate
cd tests
python test1.py
but I'm not sure how to set that up on crontab. What's the proper way to do so?
You can create a bash script. Let's call it script.sh
Inside script.sh you'll have
source /Users/mmmm/PycharmProjects/kt/venv/bin/activate
cd tests
python test1.py
Make sure you make the script executable with chmod +x script.sh!
Then you can add that script to crontab with
* * * * * /path/to/script.sh
This will activate the the venv, cd into tests and run test1.py while only putting the bash script into cron.
Here's a line I used for a django application management command
0 */2 * * * source /Users/jeff/.virtualenvs/myvirtualenv/bin/activate && /Users/jeff/ecommapp/manage.py amzn_scrape H >> /Users/jeff/ecommapp/log/scraper_results.log 2>&1
Activate the env and then call the command - last logging part is optional.
I have a Python script which uses environment variables. This script works exactly as planned when run directly; however, I would like to run it as a cron job every minute for the time being.
Currently in my cron. directory I have a file called scrapers containing:
* * * * * root /usr/bin/python3.5 /code/scraper.py
This runs the Python script but the script fails, as in the script I use two environment variables.
I read I should add SHELL=/bin/bash to the cron file, so I did, but this didn't help.
SHELL=/bin/bash
* * * * * root /usr/bin/python3.5 /code/scraper.py
Then I read
In the crontab, before you command, add . $HOME/.profile.
SHELL=/bin/bash
* * * * * . $HOME/.profile; root /usr/bin/python3.5 /code/scraper.py
but this caused the cron to stop running altogether. What is the best way of 'sending' the env variables to the cron?
Instead of executing the whole ~/.profile what I'd do is move the variables that must be shared between your cron jobs and the account that has the profile, then I'd source these both in ~/.profile and in the cron job.
The last attempt you show in the question is not properly formatted. The user id should be coming right after the scheduling information, but you've added the sourcing of the profile before the user id, which surely cannot work.
Here's an example setup that I've tested here:
*/1 * * * * someuser . /tmp/t10/setenv && /usr/bin/python /tmp/t10/test.py
I've set it to execute every minute for testing purposes. Replace someuser with something that makes sense. The /tmp/t10/setenv script I used had this:
export FOO=foovalue
export BAR=barvalue
The /tmp/t10/test.py file had this:
import os
print os.environ["FOO"], os.environ["BAR"]
My cron emails me the output of the scripts it runs. I got an email with this output:
foovalue barvalue
You can set the env variable inline:
* * * * * root ENV_VAR=VALUE /usr/bin/python3.5 /code/scraper.py
Another way is use honcho that you can pass a file with env variables.
honcho -e /path/to/.env run /code/scraper.py
You can specify your two environment variables by this:
* * * * * root env A=1 B=2 /usr/bin/python3.5 /code/scraper.py
env is a system program that runs a specified program with additional variables:
$ env A=1 B=2 /bin/sh -c 'echo $A$B' # or just 'sh': would search in $PATH
12
You can add it to the top of your crontab and keep it out of version control. Let's say the environment variable causing you difficulty is export DJANGO_SECRET_KEY="FOOBAR_1241243124312341234":
crontab
DJANGO_SECRET_KEY="FOOBAR_1241243124312341234"
SCRIPT_NAME = my_cool_script
20 21 * * 1-5 bash ~/git_repo/cronjobs/$SCRIPT_NAME.sh 2&>1 | tee ~/git_repo/cronjobs/logs/$SCRIPT_NAME.log
my_cool_script.sh
#!/usr/bin/env bash
~/anaconda3/envs/django/bin/python ~/git_repo/django_project/manage.py run_command
This has worked well for me when the environment variables in question need to be kept secret and the loading of existing .bashrc does not play nice for whatever reason.
This is one of the approach I like, write a script to set environment and execute the script with its parameters as its parameters
set_env_to_process.sh
#!/usr/bin/env bash
echo "TEST_VAR before export is: <$TEST_VAR>"
export TEST_VAR=/opt/loca/netcdf
echo "TEST_VAR after export is: <$TEST_VAR>"
export PATH=$PATH:/usr/bin/python3.5
export PYTHTONPATH=$PYTHONPATH:/my/installed/pythonpath
# execute command and its parameters as input for this script
if [ $# -eq 0 ]; then
echo "No command to execute"
else
echo "Execute commands with its parameters: $#"
eval $#
fi
usage
/usr/bin/python3.5 /code/scraper.pyare taken as input for set_env_to_process.sh
set_env_to_process.sh set the correct env for script to run
It could be used as command line, cron, sudo, ssh to setup env
* * * * * root set_env_to_process.sh /usr/bin/python3.5 /code/scraper.py
How to run in crontab
*/1 * * * * /home/user/Desktop/job/dp/ python manage.py statistics
with virtual env? I need to activate virtualenv first(Otherwise it does not work)
This is my virtual env:
source job/bin/activate
EDITED:
Try something like this:
*/1 * * * * . /path-to-env/bin/activate && /home/user/Desktop/job/dp/manage.py statistics
This should be read as: activate the env and if that was successful, excute the manage.py script. Since manage.py is supposed to have a python shebang and the virtual env sets the correct python interpreter, this should work.
Apparently cron usually runs with /bin/sh which does not know the source command. So one option is to use a dot as a source replacement. Another to set /bin/bash in the cron file:
SHELL=/bin/bash
*/1 * * * * source /path-to-env/bin/activate && /home/user/Desktop/job/dp/manage.py statistics
Read more about this issue at:
http://codeinthehole.com/writing/running-django-cronjobs-within-a-virtualenv/
The article doesn't mention that source can be replaced by a ., but i've just tried it and it worked for me. So you have several options to choose from now, the article even has others. ;)
Use something like ~/envs/someenv/lib/python /path/to/your/script
In your situation it will look like
*/1 * * * * ~/envs/someenv/lib/python /home/user/Desktop/job/dp/manage.py statistics