Importing packages that are subsequently installed (not present by default in python distribution for rhel 7.6) not working when ran as cron job
Hi Team,
I have a python(2.7) script which imports paramiko package. The script can import the paramiko package successfully when ran as a user(root or ftpuser) after logging in but it cannot import it when ran from cron job. I have tried out various options as provided in the brilliant stack overflow pages like the below but unfortunately couldn't resolve the issue.
1) Crontab not running my python script
I have provided the path to the paramiko package and verified it is successfully received at the script end by logging it when run as cron job and also I have given chmod -R 777 permission to the paramiko folder in the /opt/rh/python27/root/usr/lib/python2.7/site-packages location. Still the import is not functioning when ran as cron job
I have created a shell script and tried to invoke python script from with in the script and configured the shell script in cron job but it seems python script was not invoked
I have verified that there is only one python installation present in the server and so I' am using the correct path
I have disabled selinux option and tried after rebooting but issue still persists
Please note the issue exists not for just paramiko package but for other packages as well that was installed subsequently like mysql.connector e t c
Update1
It has to be something to do with the way I install the paramiko package because the script can even import other packages in the same path as that of paramiko and the permissions for both of them look identical only difference is former comes with the python distribution that is deployed from using the url https://access.redhat.com/solutions/1519803. Cannot figure out what is wrong with the installation steps as I install it as root after doing sudo su and then setting umask to 0022. I do pip install of parmiko and python-crontab as mentioned in their sites and both have the same issue
Another interesting thing is though I have the code to log exception around the failing import statement it never logs the exception but script seems to halt/hang at the import statment
Please help to resolve this issue...
PYTHON CODE
#!/usr/bin/env python
import sys
import logging
import os
def InitLog():
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(message)s',
filename=os.path.dirname(os.path.abspath(__file__)) + '/test_paramiko.log',
filemode='a'
)
logging.info('***************start logging****************')
InitLog()
logging.info('before import')
logging.info(sys.path)
try:
sys.path.append("/opt/rh/python27/root/usr/lib/python2.7/site-packages")
logging.info("sys path appended before import")
import paramiko
except ImportError:
logging.ERROR("Exception occured druing import")
logging.info('after import')
CRONTAB Entry
SHELL=/bin/bash
PATH=/opt/rh/python27/root/usr/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ftpuser/.local/bin:/home/ftpuser/bin
PYTHONPATH=/opt/rh/python27/root/usr/lib64/python27.zip:/opt/rh/python27/root/usr/lib64/python2.7:/opt/rh/python27/root/usr/lib64/python2.7/plat-linux2:/opt/rh/python27/root/usr/lib64/python2.7/lib-tk:/opt/rh/python27/root/usr/lib64/python2.7/lib-old:/opt/rh/python27/root/usr/lib64/python2.7/lib-dynload:/opt/rh/python27/root/usr/lib64/python2.7/site-packages:/opt/rh/python27/root/usr/lib/python2.7/site-packages
*/1 * * * * /opt/rh/python27/root/usr/bin/python /home/ftpuser/Ganesh/test_paramiko.py
#*/1 * * * * /home/ftpuser/Ganesh/test_cron.sh >> /home/ftpuser/Ganesh/tes_cron.txt 2>&1
#*/1 * * * * /home/ftpuser/Ganesh/test_cron.sh
Shell script
#!/opt/rh/python27/root/usr/bin/python
export PATH=$PATH:/opt/rh/python27/root/usr/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ftpuser/.local/bin:/home/ftpuser/bin
export PYTHONPATH=$PYTHONPATH:/opt/rh/python27/root/usr/lib64/python27.zip:/opt/rh/python27/root/usr/lib64/python2.7:/opt/rh/python27/root/usr/lib64/python2.7/plat-linux2:/opt/rh/python27/root/usr/lib64/python2.7/lib-tk:/opt/rh/python27/root/usr/lib64/python2.7/lib-old:/opt/rh/python27/root/usr/lib64/python2.7/lib-dynload:/opt/rh/python27/root/usr/lib64/python2.7/site-packages:/opt/rh/python27/root/usr/lib/python2.7/site-packages
python /home/ftpuser/Ganesh/test_paramiko.py
Expected result from my python script is to log the "after import" string
But currently it is printing only till "sys path appended before import" which also shows the normal python packages are getting imported successfully
This seems to be working now after adding one more environment variable to crontab as below
LD_LIBRARY_PATH=/opt/rh/python27/root/usr/lib64
#!/usr/bin/python
import requests, zipfile, StringIO, sys
extractDir = "myfolder"
zip_file_url = "download url"
response = requests.get(zip_file_url)
zipDocument = zipfile.ZipFile(StringIO.StringIO(response.content))
zipinfos = zipDocument.infolist()
for zipinfo in zipinfos:
extrat = zipDocument.extract(zipinfo,path=extractDir)
System configuration
Ubuntu OS 16.04
Python 2.7.12
$ python extract.py
when I run the code on Terminal with above command, it works properly and create the folder and extract the file into it.
Similarly, when I create a cron job using sodu rights the code executes but don't create any folder or extracts the files.
crontab command:-
40 10 * * * /usr/bin/sudo /usr/bin/python /home/ubuntu/demo/directory.py > /home/ubuntu/demo/logmyshit.log 2>&1
also tried
40 10 * * * /usr/bin/python /home/ubuntu/demo/directory.py > /home/ubuntu/demo/logmyshit.log 2>&1
Notes :
I check the syslog, it says the cron is running successfully
The above code gives no errors
also made the python program executable by chmod +x filename.py
Please help where am I going wrong.
Oups, there is nothing really wrong in running a Python script in crontab, but many bad things can happen because the environment is not the one you are used to.
When you type in an interactive shell python directory.py, the PATH and all required PYTHON environment variable have been set as part of login and interactive shell initialization, and the current directory is your home directory by default or anywhere you currently are.
When the same command is run from crontab, the current directory is not specified (but may not be what you expect), PATH is only /bin:/usr/bin and python environment variables are not set. That means that you will have to tweak environment variables in crontab file until you get a correct Python environment, and set the current directory.
I had a very similar problem and it turned out cron didn’t like importing matplotlib, I ended up having to specify Agg backend. I figured it out by putting log statements after each line to see how far the program got before it crapped out. Of course, my log was empty which tipped me off that it crashed on imports.
TLDR: log each line inside the script
I have a simple ansible playbook which will call a shell script on a remote server, the shell script will call another python script, which will do something, when I run the ansible playbook, the script is not working, but when I ssh to the server and run the same command manually, it worked. I've done some debugging, seems when calling the python script, if I delete all the import statements from the python script, it works from ansible, but I don't understand why it works when I ssh to the server and would like to have some suggestion on how to resolve this issue.
the python script:
#!/usr/bin/python
import socket
import argparse
import logging
import subprocess
import time
import imp
def main():
f = open('/afile', 'w')
f.write('a test line')
f.close()
if __name__ == '__main__':
main()
those imports are not using here, it will be used in my real script, here I just write a line into a file for debugging.
The ansible playbooks are just simply like:
---
- hosts: servers
tasks:
- name: trigger the script
shell: /start.sh
The start.sh then simply invoke the python script:
#!/bin/sh
/start.py
sorry, it's my bad, I didn't put all the scripts here, seems that there is another script which has things like
#!/bin/sh
/start & >> stdout.log
this caused the problem, I guess the first three modules imported have things related to standard io, so the solution is using nohup.
again, very sorry for the incomplete question.
I just can't figure out why this is happening...
in my fabric file I have this...
def func():
local("source ../venv/bin/activate")
It is returning 127 on the source command and I am not sure why, when i try to run source in my terminal manually it works. When I try a different command in fabric, like...
local("echo 'foo'")
it also works. Why would source be causing problems? I can't see any reason why this would be happening.
source is unknown by sh. If you want to run a command in the context of a virtual environment, use Fabric's context managers and run the activate binary without source. I've adapted from this answer.
from __future__ import with_statement
from fabric.api import *
from contextlib import contextmanager as _contextmanager
env.activate = '. ./.env/bin/activate'
#_contextmanager
def virtualenv():
with prefix(env.activate):
yield
def deploy():
with virtualenv():
local('echo hello world!')
This question already has answers here:
CronJob not running
(19 answers)
Closed 2 years ago.
My python script is not running under my crontab.
I have placed this in the python script at the top:
#!/usr/bin/python
I have tried doing this:
chmod a+x myscript.py
Added to my crontab -e:
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=""
* * * * * /home/me/project/myscript.py
My /var/log/cron file says:
Sep 21 11:53:02 163-dhcp /USR/SBIN/CROND[2489]: (me) CMD (/home/me/project/myscript.py)
But my script is not running because when I check my sql database, nothing has changed. If I run it directly in the terminal like so:
python /home/me/project/myscript.py
I get the correct result.
This is the myscript.py:
#!/usr/bin/python
import sqlite3
def main():
con = sqlite3.connect("test.db")
with con:
cur = con.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS testtable(Id INTEGER PRIMARY KEY, Name TEXT)")
cur.execute("INSERT INTO testtable(Name) VALUES ('BoB')")
cur.execute("SELECT * FROM testtable")
print cur.fetchall()
if __name__ == "__main__":
main()
Per comments: Yes, /usr/bin/python exists. I can also run the python script directly using just /home/me/project/myscript.py. /usr/bin/python /home/me/project/myscript.py works. So I don't believe this is the cause?
There are a lot of half answers across the internet so I thought I would capture this to save someone else some time.
First, cronjob does a poor job of telling you where this is failing. I recommend sending stderr output to a log file like this:
Crontab Command:
# m h dom mon dow command
* * * * * /path/to/your_file.sh >> out.txt 2>&1
As this is likely running the command as user, check home directory for the log file. Note this script runs every minute which is good for debugging.
The next issue is you probably have a path problem... as script likely is trying to execute from your home directory. This script sets the current directory, echos it to file, and then runs your program.
Try this :
Script File
#!/bin/sh
cd "$(dirname "$0")";
CWD="$(pwd)"
echo $CWD
python your_python_file.py
Hope this saves someone else some debugging time!!!
What happens when you type
/home/me/project/myscript.py into the shell?
Can you explicitly use /usr/bin/python in your crontbb command?
Can you either use an absolute path to your test.db or cd to the correct directory then execute your python script?
This is helpful to have debug statements in your python and log some data. Crontab can be very tricky to debug.
It is possible that the script does not start because it cannot locate the python interpreter. Crontab environment may be very different from the shell environment which you are using. The search paths may be differ significantly.
Also, you test your script by starting the python interpreter explicitly while you expect the crontab to only start the script.
I put this line at the top of my python scripts:
\#!/bin/env python
This line will help locate the interpreter regardless of which directory it is installed in as long as it is in the search path.
It's usually because the python used by crontab is different from the one you use in the shell.
The easiest way to solve this is:
get the python you use in the shell:
$ which python # it may be "python3" or something else
/usr/bin/python
use that specific python in crontab file:
* * * * * /usr/bin/python test.py
Also want to mention that using env -i /bin/bash --noprofile --norc in the shell lets you have the same environment as the one used by crontab, and this is super helpful to debug.
Typically, crontab problems like this are caused by the PATH environment variable being more restrictive/different than what your normal user's PATH environment is. Since your shell uses the PATH environment to find the executable (e.g. /usr/bin/python is found in /usr/bin when you type "python" at a shell prompt), when the PATH is missing common locations, like /usr/bin or /usr/sbin, your cron job will fail. This has bit me many times. The simple fix is just to explicitly set the PATH yourself near the top of your crontab file, before any commands that need it. So, just edit the crontab as usual and add something like this near the top (if your binary is not in one of the below paths, you'll need to add it after a colon):
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
Alternately, just use absolute paths to your binaries and scripts in crontab.
Try this
* * * * * cd <directory_where_python_file_is> && bin/app etc/app_defaults.yaml
There is some path issue with cron. So when you move to directory with python file, cron works like charm!
I'd got the same problem. Despite the fact that the script executed manually was working, in crontab no options mentioned above were working at all. I've moved my script from /home/user/script_directory/ to /opt/scripts/ and it started to work. Possible cause of the problem should be the access (read) permissions to subfolder located in home directory.
Easiest way to handle this is to add your python installation's path to PATH in top of the shell script.
Something like:
#!/usr/bin/env bash
export PATH="{path to your python installation}:$PATH"
python {python_file_name}.py
As #Shargors said you can test it by
env -i /bin/bash --noprofile --norc
If you are using anaconda for python then the path to use will be :
/home/username/anaconda3/bin/python test.py
While the answers here clearly delineate the problem and solution, I wanted to add another answer which helped me.
If your python script is calling a database, then be sure you can connect to the db properly within the cron env (to identify the cron env--> https://askubuntu.com/questions/23009/reasons-why-crontab-does-not-work). I had a file that would run from the shell, but not as a crontab unless I connected to the database as root from within the python script.
Sometimes I am facing same problem. Whatever I try something as advised here, I may not get result.
So I begin to write "trigger" bash script as follow (let's name it trigger.sh):
#!/bin/bash
/full_path/python_script.py
And I am calling trigger.sh from crontab and everything is fine.
EDIT: Of course, don't forget to do following (give execution right):
$chmod +x python_script.py
$chmod +x trigger.sh
I was working on project that includes paramiko lib, when I run the Check_.py from cmdlin it works perfect but when I set the crontab it fails with error no module name paramiko.
So to make it short:
- there were two different python versions installed 3.7 and 2.4, so I used whreris python3 to locate the python path /usr/local/bin/python3.7m so replacing the python with the path will solve the issue.
Example
* * * * * cd /home/MKhair/hlthchk/BR/ && /usr/local/bin/python3.7m /home/MKhair/hlthchk/BR/Check_.py
* * * * * cd [ path-to-the-script-dir] && [path-to-python] [path-to-the-script]
Try to put in your crontab:
* * * * * python /path/to/your/script.py
rather than
* * * * * /path/to/your/script.py
Also the shebang line is #!/usr/bin/env python in some environments. env is an executable, and you have to know where it lives with "$ which env".
Is the cron user (where the script fails) and the terminal user (when the script succeeds) are same ?
Can you redirect the job output to some file as mentioned in Cron Job Log - How to Log?. We could see whether that helps.
This might be helpful for someone. I was having this same issue (or at least a similar issue) and what helped me was to get the path in which Python (Be aware of the version you want to use python, python3, etc...) by running this:
which python3
And then, I replaced python3 for the full path of python3 in my crontab file.