We Keep Coding

Activate conda environment in cron (including environment variables!) [duplicate]

I am trying to run a Django management command from cron. I am using virtualenv to keep my project sandboxed.
I have seen examples here and elsewhere that show running management commands from within virtualenv's like:
0 3 * * * source /home/user/project/env/bin/activate && /home/user/project/manage.py command arg
However, even though syslog shows an entry when the task should have started, this task never actually runs (the log file for the script is empty). If I run the line manually from the shell, it works as expected.
The only way I can currently get the command to run via cron, is to break the commands up and put them in a dumb bash wrapper script:
#!/bin/sh
source /home/user/project/env/bin/activate
cd /home/user/project/
./manage.py command arg
EDIT:
ars came up with a working combination of commands:
0 3 * * * cd /home/user/project && /home/user/project/env/bin/python /home/user/project/manage.py command arg
At least in my case, invoking the activate script for the virtualenv did nothing. This works, so on with the show.

You should be able to do this by using the python in your virtual environment:
/home/my/virtual/bin/python /home/my/project/manage.py command arg
EDIT: If your django project isn't in the PYTHONPATH, then you'll need to switch to the right directory:
cd /home/my/project && /home/my/virtual/bin/python ...
You can also try to log the failure from cron:
cd /home/my/project && /home/my/virtual/bin/python /home/my/project/manage.py > /tmp/cronlog.txt 2>&1
Another thing to try is to make the same change in your manage.py script at the very top:
#!/home/my/virtual/bin/python

Running source from a cronfile won't work as cron uses /bin/sh as its default shell, which doesn't support source. You need to set the SHELL environment variable to be /bin/bash:
SHELL=/bin/bash
*/10 * * * * root source /path/to/virtualenv/bin/activate && /path/to/build/manage.py some_command > /dev/null
It's tricky to spot why this fails as /var/log/syslog doesn't log the error details. Best to alias yourself to root so you get emailed with any cron errors. Simply add yourself to /etc/aliases and run sendmail -bi.
More info here:
http://codeinthehole.com/archives/43-Running-django-cronjobs-within-a-virtualenv.html
the link above is changed to:
https://codeinthehole.com/tips/running-django-cronjobs-within-a-virtualenv/

Don't look any further:
0 3 * * * /usr/bin/env bash -c 'cd /home/user/project && source /home/user/project/env/bin/activate && ./manage.py command arg' > /dev/null 2>&1
Generic approach:
* * * * * /usr/bin/env bash -c 'YOUR_COMMAND_HERE' > /dev/null 2>&1
The beauty about this is you DO NOT need to change the SHELL variable for crontab from sh to bash

I am sorry for that nth answer but I checked the answers and there is really simpler and neater.
Long story short
Use the python binary of your venv in your cron :
0 3 * * * /home/user/project/env/bin/python /home/user/project/manage.py
Long story
We activate the virtual environment when we want to set the current shell with the python config of that specific virtual environment(that is binaries and modules of that).
It is relevant to work with the current shell : execute multiple python commands on the current shell without the need to reference the full python path of the venv.
In the frame of a cron or even a bash, which value to activate the environment ?
Besides I read in some answers some references to bash rather than sh or still to define a wrapper to call the Python code. But why the hell should we bother with these ?
I repeat, just do it :
0 3 * * * /home/user/project/env/bin/python /home/user/project/manage.py
The documentation confirms that :
You don’t specifically need to activate an environment; activation
just prepends the virtual environment’s binary directory to your path,
so that “python” invokes the virtual environment’s Python interpreter
and you can run installed scripts without having to use their full
path. However, all scripts installed in a virtual environment should
be runnable without activating it, and run with the virtual
environment’s Python automatically.

The only correct way to run python cron jobs when using a virtualenv is to activate the environment and then execute the environment's python to run your code.
One way to do this is use virtualenv's activate_this in your python script, see: http://virtualenv.readthedocs.org/en/latest/userguide.html#using-virtualenv-without-bin-python
Another solution is echoing the complete command including activating the environment and piping it into /bin/bash. Consider this for your /etc/crontab:
***** root echo 'source /env/bin/activate; python /your/script' | /bin/bash

Rather than mucking around with virtualenv-specific shebangs, just prepend PATH onto the crontab.
From an activated virtualenv, run these three commands and python scripts should just work:
$ echo "PATH=$PATH" > myserver.cron
$ crontab -l >> myserver.cron
$ crontab myserver.cron
The crontab's first line should now look like this:
PATH=/home/me/virtualenv/bin:/usr/bin:/bin: # [etc...]

This is a simple way that keeps the crontab command very similar to regular command (tested in Ubuntu 18.04). Some key notes to keep in mind:
You can use the . command instead of source. (crontab uses sh by default, not bash, so it doesn't have source.)
~ and $variables are expanded in crontab commands. (It's only crontab environment statements that don't do variable expansion.)
Here are examples if you have a file ~/myproject/main.py:
* * * * * cd ~/myproject && . .venv/bin/activate && python main.py > /tmp/out1 2>&1
You could also directly call the specific path of the python in the venv directory, then you don't need to call activate.
* * * * * ~/myproject/.venv/bin/python ~/myproject/main.py > /tmp/out2 2>&1
The downside of that is you would need to specify the project path twice, which makes maintenance trickier. To avoid that, you could use a shell variable so you only specify the project path once:
* * * * * project_dir=~/myproject ; $project_dir/.venv/bin/python $project_dir/main.py > /tmp/out3 2>&1

The best solution for me was to both
use the python binary in the venv bin/ directory
set the python path
to include the venv modules directory.
man python mentions modifying the path in shell at $PYTHONPATH or in python with sys.path
Other answers mention ideas for doing this using the shell. From python, adding the following lines to my script allows me to successfully run it directly from cron.
import sys
sys.path.insert(0,'/path/to/venv/lib/python3.3/site-packages');
Here's how it looks in an interactive session --
Python 3.3.2+ (default, Feb 28 2014, 00:52:16)
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python3.3', '/usr/lib/python3.3/plat-x86_64-linux-gnu', '/usr/lib/python3.3/lib-dynload']
>>> import requests
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'requests'
>>> sys.path.insert(0,'/path/to/venv/modules/');
>>> import requests
>>>

I'd like to add this because I spent some time solving the issue and did not find an answer here for combination of variables usage in cron and virtualenv. So maybe it'll help someone.
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DIR_SMTH="cd /smth"
VENV=". venv/bin/activate"
CMD="some_python_bin do_something"
# m h dom mon dow command
0 * * * * $DIR_SMTH && $VENV && $CMD -k2 some_target >> /tmp/crontest.log 2>&1
It did not work well when it was configured like
DIR_SMTH="cd /smth && . venv/bin/activate"
Thanks #davidwinterbottom, #reed-sandberg and #mkb for giving the right direction. The accepted answer actually works fine until your python need to run a script which have to run another python binary from venv/bin directory.

If you're on python and using a Conda Virtual Environment where your python script contains the shebang #!/usr/bin/env python the following works:
* * * * * cd /home/user/project && /home/user/anaconda3/envs/envname/bin/python script.py 2>&1
Additionally, if you want to capture any outputs in your script (e.g. print, errors, etc) you can use the following:
* * * * * cd /home/user/project && /home/user/anaconda3/envs/envname/bin/python script.py >> /home/user/folder/script_name.log 2>&1

python script
from datetime import datetime
import boto # check wheather its taking the virtualenv or not
import sys
param1=sys.argv[1] #Param
myFile = open('appendtxt.txt', 'a')
myFile.write('\nAccessed on ' + param1+str(datetime.now()))
Cron command
*/1 * * * * cd /Workspace/testcron/ && /Workspace/testcron/venvcron/bin/python3 /Workspace/testcron/testcronwithparam.py param
In above command
*/1 * * * * - Execute every one minte
cd /Workspace/testcron/ - Path of the python script
/Workspace/testcron/venvcron/bin/python3 - Virtualenv path
Workspace/testcron/testcronwithparam.py - File path
param - parameter

I've added the following script as manage.sh inside my Django project, it sources the virtualenv and then runs the manage.py script with whatever arguments you pass to it. It makes it very easy in general to run commands inside the virtualenv (cron, systemd units, basically anywhere):
#! /bin/bash
# this is a convenience script that first sources the venv (assumed to be in
# ../venv) and then executes manage.py with whatever arguments you supply the
# script with. this is useful if you need to execute the manage.py from
# somewhere where the venv isn't sourced (e.g. system scripts)
# get the script's location
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
# source venv <- UPDATE THE PATH HERE WITH YOUR VENV's PATH
source $DIR/../venv/bin/activate
# run manage.py script
$DIR/manage.py "$#"
Then in your cron entry you can just run:
0 3 * * * /home/user/project/manage.sh command arg
Just remember that you need to make the manage.sh script executable

Add the path of the Python installation in the venv but don't activate the environment.
* * * * * /HDD1/shritam_kumar/VENOM/venv/bin/python /HDD1/shritam_kumar/Projects/Voelkner-DE/schedule_product_BA.py
It's that simple.

I had the same issue and spent a lot of time solving that.
None of the solutions here helped me, so I'm sharing what worked for me:
Open a new file "pick_name.sh" open it inside of your project directory.
Inside the "pick_name.sh" file, write and save the following lines:
#!/bin/bash
source /YOUR_VIRTUAL_ENV_PATH/bin/activate
export PYTHONPATH="${PYTHONPATH}:/PATH_TO_CUSTOM_MODULE_YOU_CREATED**OPTIONAL**"
export PYTHONPATH="${PYTHONPATH}:/PATH_TO_ANOTHER_CUSTOM_MODULE_YOU_CREATED**OPTIONAL**"
cd /PATH_TO_DIR_STORING_FILE_NAME.PY
python file_name.py
Go to /var/spool/cron/crontabs (or to where your cron management file sits) and open the 'root' file.
Add these lines to the root file which's inside the crontab folder:
# m h dom mon dow command
* * * * * /PATH_TO_DIR_WHERE_PICK_NAME.SH_SITS/pick_name.sh >> /YOUR_PROJECT_DIR/cron_output.txt 2>&1
Notes:
This command (section 4.) will run the "pick_name.sh" file.
In this example it runs every minute, so make sure you change it according to your needs.
It writes all logs to a log file called "cron_ouput".
No need to create the file before, it will be created automatically.
Make sure to replace all paths (I wrote them in capital letters) to your paths.
You can change file names, if so, make sure to change it in all appearances in my instructions to avoid errors.
If you want to add another py file to run by cron, you need to add it to the "pick_nam.sh" file* not to the cron. Simply duplicate section 2. lines in the "pick_nam.sh" but without the "#!/bin/bash" part.
Then, every time the cron will run "pick_name.sh" it will run all the files you specified inside of it.
Make sure to restart cron after changes, it could have saved me a lot of debugging time, use this command:
systemctl restart cron

Since a cron executes in its own minimal sh environment, here's what I do to run Python scripts in a virtual environment:
* * * * * . ~/.bash_profile; . ~/path/to/venv/bin/activate; python ~/path/to/script.py
(Note: if . ~/.bash_profile doesn't work for you, then try . ~/.bashrc or . ~/.profile depending on how your server is set up.)
This loads your bash shell environment, then activates your Python virtual environment, essentially leaving you with the same setup you tested your scripts in.
No need to define environment variables in crontab and no need to modify your existing scripts.

If you are a MacOS user like me, you can check the crontab error message at /var/mail/{username} file. like this
tail /var/mail/{username}
If there is an "operation not permitted" error, maybe you have to add cron to the Full Disk Access apps (Security & Privacy > Privacy > Full Disk Access apps/execs).
And click + button, go to /usr/sbin, double click the cron file.
then it will fix the "not permitted" error. detailed steps
And this is my code：
0 19 * * * cd /Users/user/Desktop/Project && source /Users/user/Desktop/Project/venv/bin/activate && python command arg

This is a solution that has worked well for me.
source /root/miniconda3/etc/profile.d/conda.sh && \
conda activate <your_env> && \
python <your_application> &
I am using miniconda with Conda version 4.7.12 on a Ubuntu 18.04.3 LTS.
I am able to place the above inside a script and run it via crontab as well without any trouble.

This will also work on crontab -e
* */5 * * * cd /home/project && sudo /home/project/venv/bin/python scripte.py

I had this same issue:
I had written a custom django command to check for geodjango position coordinates inside of geodjango polygons and had trouble automating the task to run, however using this command with crontab worked for me:
* * * * * ./home/project/locations/locations.sh >> /var/log/locations.log 2>&1

python don't extract zipfile when ran through crontab

#!/usr/bin/python
import requests, zipfile, StringIO, sys
extractDir = "myfolder"
zip_file_url = "download url"
response = requests.get(zip_file_url)
zipDocument = zipfile.ZipFile(StringIO.StringIO(response.content))
zipinfos = zipDocument.infolist()
for zipinfo in zipinfos:
extrat = zipDocument.extract(zipinfo,path=extractDir)
System configuration
Ubuntu OS 16.04
Python 2.7.12
$ python extract.py
when I run the code on Terminal with above command, it works properly and create the folder and extract the file into it.
Similarly, when I create a cron job using sodu rights the code executes but don't create any folder or extracts the files.
crontab command:-
40 10 * * * /usr/bin/sudo /usr/bin/python /home/ubuntu/demo/directory.py > /home/ubuntu/demo/logmyshit.log 2>&1
also tried
40 10 * * * /usr/bin/python /home/ubuntu/demo/directory.py > /home/ubuntu/demo/logmyshit.log 2>&1
Notes :
I check the syslog, it says the cron is running successfully
The above code gives no errors
also made the python program executable by chmod +x filename.py
Please help where am I going wrong.

Oups, there is nothing really wrong in running a Python script in crontab, but many bad things can happen because the environment is not the one you are used to.
When you type in an interactive shell python directory.py, the PATH and all required PYTHON environment variable have been set as part of login and interactive shell initialization, and the current directory is your home directory by default or anywhere you currently are.
When the same command is run from crontab, the current directory is not specified (but may not be what you expect), PATH is only /bin:/usr/bin and python environment variables are not set. That means that you will have to tweak environment variables in crontab file until you get a correct Python environment, and set the current directory.

I had a very similar problem and it turned out cron didn’t like importing matplotlib, I ended up having to specify Agg backend. I figured it out by putting log statements after each line to see how far the program got before it crapped out. Of course, my log was empty which tipped me off that it crashed on imports.
TLDR: log each line inside the script

Running cronjob (Python write file) on EC2

I have a simple Python script on EC2 to write a text to a file:
with open("/home/ec2-user/python/test.txt", "a") as f:
f.write("test\n\n")
That python script is here:
/home/ec2-user/python/write_file.py
When I run that script manually ('python /home/ec2-user/python/write_file.py') - new text is being written to a file ('/home/ec2-user/python/test.txt').
When schedule same script using Cronjob - no data is being added to a file. My Cronjob looks like this:
* * * * * python /home/ec2-user/python/write_file.py
I verify Cronjob is running and suspecting some ENV parameters are not the same during Cronjob execution (or something else is happening). What could be the case and how to fix it in a simple way?
Thanks.

I recently began using Amazon's linux distro on ec2 instances and after trying all kinds of things for cron all I needed was:
sudo service crond start
crontab -e
This allowed me to set a cron job as "ec2-user" without specifying the user or env variables or anything. For example, this single line worked:
0 12 * * * python3 /home/ec2-user/example.py
I also posted this here.

On crontab
HOME=/home/ec2-user/
* * * * * python /home/ec2-user/python/write_file.py
alternatively (testing purpose)
* * * * * root /home/ec2-user/python/write_file.py
On your write_file.py add shebang (first line)
#!/usr/bin/env python
BONUS: (feel free to ignore) my first response debug method for cron on minute tasks:
import os
while True:
print os.popen('pgrep -lf python').read()
executed from shell will pop out all python modules called by cron. After 60 secs you know whats going on.

it's bette to specify the absolute path of the python interpreter because the command may not be recognized by the system. Or you can add the shebang line to your script #!pathToPythonInterpreter at the first line of your script
make sure that the user who planning the execution of script have the right to read and write to the file
make sure that you are not using the system corntab /etc/crontab to planify because then you have to specify the user who will execute the command.
I hope that this will help you

Crontab not executing a Python script? [duplicate]

This question already has answers here:
CronJob not running
(19 answers)
Closed 2 years ago.
My python script is not running under my crontab.
I have placed this in the python script at the top:
#!/usr/bin/python
I have tried doing this:
chmod a+x myscript.py
Added to my crontab -e:
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=""
* * * * * /home/me/project/myscript.py
My /var/log/cron file says:
Sep 21 11:53:02 163-dhcp /USR/SBIN/CROND[2489]: (me) CMD (/home/me/project/myscript.py)
But my script is not running because when I check my sql database, nothing has changed. If I run it directly in the terminal like so:
python /home/me/project/myscript.py
I get the correct result.
This is the myscript.py:
#!/usr/bin/python
import sqlite3
def main():
con = sqlite3.connect("test.db")
with con:
cur = con.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS testtable(Id INTEGER PRIMARY KEY, Name TEXT)")
cur.execute("INSERT INTO testtable(Name) VALUES ('BoB')")
cur.execute("SELECT * FROM testtable")
print cur.fetchall()
if __name__ == "__main__":
main()
Per comments: Yes, /usr/bin/python exists. I can also run the python script directly using just /home/me/project/myscript.py. /usr/bin/python /home/me/project/myscript.py works. So I don't believe this is the cause?

There are a lot of half answers across the internet so I thought I would capture this to save someone else some time.
First, cronjob does a poor job of telling you where this is failing. I recommend sending stderr output to a log file like this:
Crontab Command:
# m h dom mon dow command
* * * * * /path/to/your_file.sh >> out.txt 2>&1
As this is likely running the command as user, check home directory for the log file. Note this script runs every minute which is good for debugging.
The next issue is you probably have a path problem... as script likely is trying to execute from your home directory. This script sets the current directory, echos it to file, and then runs your program.
Try this :
Script File
#!/bin/sh
cd "$(dirname "$0")";
CWD="$(pwd)"
echo $CWD
python your_python_file.py
Hope this saves someone else some debugging time!!!

What happens when you type
/home/me/project/myscript.py into the shell?
Can you explicitly use /usr/bin/python in your crontbb command?
Can you either use an absolute path to your test.db or cd to the correct directory then execute your python script?
This is helpful to have debug statements in your python and log some data. Crontab can be very tricky to debug.

It is possible that the script does not start because it cannot locate the python interpreter. Crontab environment may be very different from the shell environment which you are using. The search paths may be differ significantly.
Also, you test your script by starting the python interpreter explicitly while you expect the crontab to only start the script.
I put this line at the top of my python scripts:
\#!/bin/env python
This line will help locate the interpreter regardless of which directory it is installed in as long as it is in the search path.

It's usually because the python used by crontab is different from the one you use in the shell.
The easiest way to solve this is:
get the python you use in the shell:
$ which python # it may be "python3" or something else
/usr/bin/python
use that specific python in crontab file:
* * * * * /usr/bin/python test.py
Also want to mention that using env -i /bin/bash --noprofile --norc in the shell lets you have the same environment as the one used by crontab, and this is super helpful to debug.

Typically, crontab problems like this are caused by the PATH environment variable being more restrictive/different than what your normal user's PATH environment is. Since your shell uses the PATH environment to find the executable (e.g. /usr/bin/python is found in /usr/bin when you type "python" at a shell prompt), when the PATH is missing common locations, like /usr/bin or /usr/sbin, your cron job will fail. This has bit me many times. The simple fix is just to explicitly set the PATH yourself near the top of your crontab file, before any commands that need it. So, just edit the crontab as usual and add something like this near the top (if your binary is not in one of the below paths, you'll need to add it after a colon):
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
Alternately, just use absolute paths to your binaries and scripts in crontab.

Try this
* * * * * cd <directory_where_python_file_is> && bin/app etc/app_defaults.yaml
There is some path issue with cron. So when you move to directory with python file, cron works like charm!

I'd got the same problem. Despite the fact that the script executed manually was working, in crontab no options mentioned above were working at all. I've moved my script from /home/user/script_directory/ to /opt/scripts/ and it started to work. Possible cause of the problem should be the access (read) permissions to subfolder located in home directory.

Easiest way to handle this is to add your python installation's path to PATH in top of the shell script.
Something like:
#!/usr/bin/env bash
export PATH="{path to your python installation}:$PATH"
python {python_file_name}.py
As #Shargors said you can test it by
env -i /bin/bash --noprofile --norc

If you are using anaconda for python then the path to use will be :
/home/username/anaconda3/bin/python test.py

While the answers here clearly delineate the problem and solution, I wanted to add another answer which helped me.
If your python script is calling a database, then be sure you can connect to the db properly within the cron env (to identify the cron env--> https://askubuntu.com/questions/23009/reasons-why-crontab-does-not-work). I had a file that would run from the shell, but not as a crontab unless I connected to the database as root from within the python script.

Sometimes I am facing same problem. Whatever I try something as advised here, I may not get result.
So I begin to write "trigger" bash script as follow (let's name it trigger.sh):
#!/bin/bash
/full_path/python_script.py
And I am calling trigger.sh from crontab and everything is fine.
EDIT: Of course, don't forget to do following (give execution right):
$chmod +x python_script.py
$chmod +x trigger.sh

I was working on project that includes paramiko lib, when I run the Check_.py from cmdlin it works perfect but when I set the crontab it fails with error no module name paramiko.
So to make it short:
- there were two different python versions installed 3.7 and 2.4, so I used whreris python3 to locate the python path /usr/local/bin/python3.7m so replacing the python with the path will solve the issue.
Example
* * * * * cd /home/MKhair/hlthchk/BR/ && /usr/local/bin/python3.7m /home/MKhair/hlthchk/BR/Check_.py
* * * * * cd [ path-to-the-script-dir] && [path-to-python] [path-to-the-script]

Try to put in your crontab:
* * * * * python /path/to/your/script.py
rather than
* * * * * /path/to/your/script.py
Also the shebang line is #!/usr/bin/env python in some environments. env is an executable, and you have to know where it lives with "$ which env".

Is the cron user (where the script fails) and the terminal user (when the script succeeds) are same ?
Can you redirect the job output to some file as mentioned in Cron Job Log - How to Log?. We could see whether that helps.

This might be helpful for someone. I was having this same issue (or at least a similar issue) and what helped me was to get the path in which Python (Be aware of the version you want to use python, python3, etc...) by running this:
which python3
And then, I replaced python3 for the full path of python3 in my crontab file.

python script execution through crontab

what happens to my script in python that does not run through crontab every minute.
My script has execute permissions and then calls two other scripts in python.
This is the content of my crontab (#crontab -l):
*/1 * * * * /usr/bin/rsm/samplesMonitor.py
Thank you guys.

Check /var/log/syslog for errors.
DIAGNOSTICS
cron requires that each entry in a crontab end in a
newline character. If the last entry in a crontab is
missing a newline (ie, terminated by EOF), cron will
consider the crontab (at least partially) broken. A
warning will be written to syslog.
Update: According to your log message, the script is running but returning an error code. Cron will email you the output, if you have a mail agent installed.
Try either:
install a mail agent, such as: apt-get install exim4
change your cron line to log to file, like so:
* * * * * /usr/bin/rsm/samplesMonitor.py 2>&1 >> /tmp/script.log
Update 2: I re-read your question and it acurred to me that maybe you are running into python import issues. You say that your script calls two other scripts. My suggestion would be to test running your script from /. Python has a default behavior to find imports in the current working directory, so make sure your script can run from any path location.
In the crontab, you can set the starting working directory by calling your script from within another shell process. For example:
bash -c "cd THE_WORKING_DIR;/usr/bin/rsm/samplesMonitor.py"

I believe it should be */1, not *\1.

It should be */1 instead of *\1 (forward slash instead of backslash). Also, make sure the path is correct; there usually are no subdirectories under /usr/bin.

If you want it to run every minute, just do this
* * * * * /usr/bin/rsm/samplesMonitor.py