We Keep Coding

Replicate ". ./bash.sh" shell command in python script [duplicate]

I want to implement a userland command that will take one of its arguments (path) and change the directory to that dir. After the program completion I would like the shell to be in that directory. So I want to implement cd command, but with external program.
Can it be done in a python script or I have to write bash wrapper?
Example:
tdi#bayes:/home/$>python cd.py tdi
tdi#bayes:/home/tdi$>

Others have pointed out that you can't change the working directory of a parent from a child.
But there is a way you can achieve your goal -- if you cd from a shell function, it can change the working dir. Add this to your ~/.bashrc:
go() {
cd "$(python /path/to/cd.py "$1")"
}
Your script should print the path to the directory that you want to change to. For example, this could be your cd.py:
#!/usr/bin/python
import sys, os.path
if sys.argv[1] == 'tdi': print(os.path.expanduser('~/long/tedious/path/to/tdi'))
elif sys.argv[1] == 'xyz': print(os.path.expanduser('~/long/tedious/path/to/xyz'))
Then you can do:
tdi#bayes:/home/$> go tdi
tdi#bayes:/home/tdi$> go tdi

That is not going to be possible.
Your script runs in a sub-shell spawned by the parent shell where the command was issued.
Any cding done in the sub-shell does not affect the parent shell.

cd is exclusively(?) implemented as a shell internal command, because any external program cannot change parent shell's CWD.

As codaddict writes, what happens in your sub-shell does not affect the parent shell. However, if your goal is to present the user with a shell in a different directory, you could always have Python use os.chdir to change the sub-shell's working directory and then launch a new shell from Python. This will not change the working directory of the original shell, but will leave the user with one in a different directory.

As explained by mrdiskodave
in Equivalent of shell 'cd' command to change the working directory?
there is a hack to achieve the desired behavior in pure Python.
I made some modifications to the answer from mrdiskodave to make it work in Python 3:
The pipes.quote() function has moved to shlex.quote().
To mitigate the issue of user input during execution, you can delete any previous user input with the backspace character "\x08".
So my adaption looks like the following:
import fcntl
import shlex
import termios
from pathlib import Path
def change_directory(path: Path):
quoted_path = shlex.quote(str(path))
# Remove up to 32 characters entered by the user.
backspace = "\x08" * 32
cmd = f"{backspace}cd {quoted_path}\n"
for c in cmd:
fcntl.ioctl(1, termios.TIOCSTI, c)

I shall try to show how to set a Bash terminal's working directory to whatever path a Python program wants in a fairly easy way.
Only Bash can set its working directory, so routines are needed for Python and Bash. The Python program has a routine defined as:
fob=open(somefile,"w")
fob.write(dd)
fob.close()
"Somefile" could for convenience be a RAM disk file. Bash "mount" would show tmpfs mounted somewhere like "/run/user/1000", so somefile might be "/run/user/1000/pythonwkdir". "dd" is the full directory path name desired.
The Bash file would look like:
#!/bin/bash
#pysync ---Command ". pysync" will set bash dir to what Python recorded
cd `cat /run/user/1000/pythonwkdr`

Setting environment variables accessible to program in root

I'm encountering an issue with my program not finding the environment variables when it is run as root. I currently have the program do:
#!/usr/bin/python3
from i2clibraries import i2c_adxl345
#various other imports
euid=os.geteuid()
if euid != 0:
args=['sudo',sys.executable]+sys.argv+[os.environ]
os.execlpe('sudo',*args)
#rest of program
Along with the environment variables, the necessary files are located in the directory this program is located in /home/pi/project-test
How do I set the environment variables to be accessible to this program when it restarts itself as root? They are:
export QUICK2WIRE_API_HOME=~/project-test/quick2wire-python-api
export PYTHONPATH=$PYTHONPATH:$QUICK2WIRE_API_HOME
Doing those exports within the directory of my program fixes it when run as user (pi) but not for root. Can I fix the QUICK2WIRE_API_HOME location above or do I need to load all of my libraries and programs into another location?
Note: substituting 'sudo -E' or 'sudo -E su' in does not carry the environment variables set as I expected. By the way, the actual error received is:
ImportError: No module named quick2wire.i2c
which is what the environment variables set the path to. Also, it has to be run as root as some of the program accesses the GPIO's and running as user (pi) returns:
RuntimeError: No access to /dev/mem
I've also toyed with the idea of breaking it up using multiprocessing or threading but both are just too far above my head at the moment to understand what I need to include and where to set functions, args, etc.
The complete program can be found here.
UPDATE: I redownloaded all the pertinent files into /root via Thinkbowl. Still doesn't want to accept the quick2wire library which is located in /root. Currently /root looks like:
. .config .gvfs quick2wire-python-api
.. .dbus i2clibraries .Xauthority
.bash_history .Desktop .idlerc .xsession-errors
.bashrc .gpio.sh .local
.cache .gstreamer-0.10 .profile
with i2c* and .quick* being the two libraries downloaded from thinkbowl. When I perform:
env | grep quick2wire
in /root I get:
OLDPWD=/root/quick2wire-python-api
QUICK2WIRE_API_HOME=/root/quick2wire-python-api
PYTHON=$PYTHONPATH:$QUICK2WIRE_API_HOME:/root/quick2wire-python-api

Crontab Python Script Execution (Can't find settings config file)

My Crontab -l
# m h dom mon dow command
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
00 8,20 * * * python /home/tomi/amaer/controller.py >>/tmp/out.txt 2>&1
My controller.py has config file settings.cfg also it uses other script in the folder it's located (I chmoded only controller.py)
The error
1;31mIOError^[[0m: [Errno 2] No such file or directory: 'settings.cfg'
I have no idea how to fix this? Please help me?
Edit: The part that read the config file
def main():
config=ConfigParser.ConfigParser()
config.readfp(open("settings.cfg"),"r")

As I initially wrote in my comment, this is because you are using relative path to the current working directory. However, that is not going to be the same when running all this via the cron executable rather than the python interpreter directly via the shebang.
Your current code would look for the "settings.cfg" in the current working directory which is where the cron executable resides, and not your script. Hence, you would need to change your code logic to using absolute paths by the help of the "os" built-in standard module.
Try to following line:
import os
...
def main():
config = ConfigParser.ConfigParser()
scriptDirectory = os.path.dirname(os.path.realpath(__file__))
settingsFilePath = os.path.join(scriptDirectory, "settings.cfg")
config.readfp(open(settingsFilePath,"r"))
This will get your the path of your script and then appends the "settings.cfg" with the appropriate dir separator for your operating system which is Linux in this particular case.
If the location of the config file changes any time in the future, you could use the argparse module for processing a command line argument to handle the config location properly, or even without it simply just using the first argument after the script name like sys.argv[1].

Your code is looking for settings.cfg in its current working directory.
This working directory will not be the same when cron executes the job, hence the error
You have two "easy" solutions:
Use an absolute path to the config file in your script (/home/tomi/amaer/config.cfg)
CD to the appropriate directory first in your crontab (cd /home/tomi/amaer/ && python /home/tomi/amaer/controller.py)
The "right" solution, though, would be to pass your script a parameter (or environment variable) that tells it where to look for the config file.
It's not exactly good practice to assume your config file will always be lying just next to your script.
You might want to have alook at this question: https://unix.stackexchange.com/questions/38951/what-is-the-working-directory-when-cron-executes-a-job

Crontab fails to execute Python script

crontab fails to execute a Python script. The command line I am using to run the Python script is ok.
These are solutions I had tried:
add #!/usr/bin/env python at the top of the main.py
add PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin at the top of crontab
chmod 777 to the main.py file
service cron restart
my crontab is:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
*/1 * * * * python /home/python_prj/main.py
and the log in /var/log/syslog is:
Nov 6 07:08:01 localhost CRON[28146]: (root) CMD (python /home/python_prj/main.py)
and nothing else.
The main.py script calls some methods from other modules under python_prj, does that matter?
Anyone can help me?

The main.py script calls some methods from other modules under python_prj, does that matter?
Yes, it does. All modules need to be findable at run time. You can accomplish this in several ways, but the most appropriate might be to set the PYTHONPATH variable in your crontab.
You might also want to set the MAILTO variable in crontab so you get emails with any tracebacks.
[update] here is the top of my crontab:
www:~# crontab -l
DJANGO_SETTINGS_MODULE=djangocron.settings
PATH=...
PYTHONPATH=/home/django
MAILTO="cron-notices#example.com"
...
# m h dom mon dow command
10-50/10 * * * * /home/django/cleanup_actions.py
...
(running cleanup actions every 10 minutes, except at the top of the hour).

Any file access in your scripts? And if so, have you used relative paths (or even: no explicit path) in your script?
When run from commandline, the actual folder is 'your path', where you start the script from. When run by cron, 'your path' may be different depending on environment variables.
So try using absolute paths to any files you access.

Check the permissions of the script. Make sure that it's executable by cron-- try chmod +x main.py.

Crontab not executing a Python script? [duplicate]

This question already has answers here:
CronJob not running
(19 answers)
Closed 2 years ago.
My python script is not running under my crontab.
I have placed this in the python script at the top:
#!/usr/bin/python
I have tried doing this:
chmod a+x myscript.py
Added to my crontab -e:
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=""
* * * * * /home/me/project/myscript.py
My /var/log/cron file says:
Sep 21 11:53:02 163-dhcp /USR/SBIN/CROND[2489]: (me) CMD (/home/me/project/myscript.py)
But my script is not running because when I check my sql database, nothing has changed. If I run it directly in the terminal like so:
python /home/me/project/myscript.py
I get the correct result.
This is the myscript.py:
#!/usr/bin/python
import sqlite3
def main():
con = sqlite3.connect("test.db")
with con:
cur = con.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS testtable(Id INTEGER PRIMARY KEY, Name TEXT)")
cur.execute("INSERT INTO testtable(Name) VALUES ('BoB')")
cur.execute("SELECT * FROM testtable")
print cur.fetchall()
if __name__ == "__main__":
main()
Per comments: Yes, /usr/bin/python exists. I can also run the python script directly using just /home/me/project/myscript.py. /usr/bin/python /home/me/project/myscript.py works. So I don't believe this is the cause?

There are a lot of half answers across the internet so I thought I would capture this to save someone else some time.
First, cronjob does a poor job of telling you where this is failing. I recommend sending stderr output to a log file like this:
Crontab Command:
# m h dom mon dow command
* * * * * /path/to/your_file.sh >> out.txt 2>&1
As this is likely running the command as user, check home directory for the log file. Note this script runs every minute which is good for debugging.
The next issue is you probably have a path problem... as script likely is trying to execute from your home directory. This script sets the current directory, echos it to file, and then runs your program.
Try this :
Script File
#!/bin/sh
cd "$(dirname "$0")";
CWD="$(pwd)"
echo $CWD
python your_python_file.py
Hope this saves someone else some debugging time!!!

What happens when you type
/home/me/project/myscript.py into the shell?
Can you explicitly use /usr/bin/python in your crontbb command?
Can you either use an absolute path to your test.db or cd to the correct directory then execute your python script?
This is helpful to have debug statements in your python and log some data. Crontab can be very tricky to debug.

It is possible that the script does not start because it cannot locate the python interpreter. Crontab environment may be very different from the shell environment which you are using. The search paths may be differ significantly.
Also, you test your script by starting the python interpreter explicitly while you expect the crontab to only start the script.
I put this line at the top of my python scripts:
\#!/bin/env python
This line will help locate the interpreter regardless of which directory it is installed in as long as it is in the search path.

It's usually because the python used by crontab is different from the one you use in the shell.
The easiest way to solve this is:
get the python you use in the shell:
$ which python # it may be "python3" or something else
/usr/bin/python
use that specific python in crontab file:
* * * * * /usr/bin/python test.py
Also want to mention that using env -i /bin/bash --noprofile --norc in the shell lets you have the same environment as the one used by crontab, and this is super helpful to debug.

Typically, crontab problems like this are caused by the PATH environment variable being more restrictive/different than what your normal user's PATH environment is. Since your shell uses the PATH environment to find the executable (e.g. /usr/bin/python is found in /usr/bin when you type "python" at a shell prompt), when the PATH is missing common locations, like /usr/bin or /usr/sbin, your cron job will fail. This has bit me many times. The simple fix is just to explicitly set the PATH yourself near the top of your crontab file, before any commands that need it. So, just edit the crontab as usual and add something like this near the top (if your binary is not in one of the below paths, you'll need to add it after a colon):
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
Alternately, just use absolute paths to your binaries and scripts in crontab.

Try this
* * * * * cd <directory_where_python_file_is> && bin/app etc/app_defaults.yaml
There is some path issue with cron. So when you move to directory with python file, cron works like charm!

I'd got the same problem. Despite the fact that the script executed manually was working, in crontab no options mentioned above were working at all. I've moved my script from /home/user/script_directory/ to /opt/scripts/ and it started to work. Possible cause of the problem should be the access (read) permissions to subfolder located in home directory.

Easiest way to handle this is to add your python installation's path to PATH in top of the shell script.
Something like:
#!/usr/bin/env bash
export PATH="{path to your python installation}:$PATH"
python {python_file_name}.py
As #Shargors said you can test it by
env -i /bin/bash --noprofile --norc

If you are using anaconda for python then the path to use will be :
/home/username/anaconda3/bin/python test.py

While the answers here clearly delineate the problem and solution, I wanted to add another answer which helped me.
If your python script is calling a database, then be sure you can connect to the db properly within the cron env (to identify the cron env--> https://askubuntu.com/questions/23009/reasons-why-crontab-does-not-work). I had a file that would run from the shell, but not as a crontab unless I connected to the database as root from within the python script.

Sometimes I am facing same problem. Whatever I try something as advised here, I may not get result.
So I begin to write "trigger" bash script as follow (let's name it trigger.sh):
#!/bin/bash
/full_path/python_script.py
And I am calling trigger.sh from crontab and everything is fine.
EDIT: Of course, don't forget to do following (give execution right):
$chmod +x python_script.py
$chmod +x trigger.sh

I was working on project that includes paramiko lib, when I run the Check_.py from cmdlin it works perfect but when I set the crontab it fails with error no module name paramiko.
So to make it short:
- there were two different python versions installed 3.7 and 2.4, so I used whreris python3 to locate the python path /usr/local/bin/python3.7m so replacing the python with the path will solve the issue.
Example
* * * * * cd /home/MKhair/hlthchk/BR/ && /usr/local/bin/python3.7m /home/MKhair/hlthchk/BR/Check_.py
* * * * * cd [ path-to-the-script-dir] && [path-to-python] [path-to-the-script]

Try to put in your crontab:
* * * * * python /path/to/your/script.py
rather than
* * * * * /path/to/your/script.py
Also the shebang line is #!/usr/bin/env python in some environments. env is an executable, and you have to know where it lives with "$ which env".

Is the cron user (where the script fails) and the terminal user (when the script succeeds) are same ?
Can you redirect the job output to some file as mentioned in Cron Job Log - How to Log?. We could see whether that helps.

This might be helpful for someone. I was having this same issue (or at least a similar issue) and what helped me was to get the path in which Python (Be aware of the version you want to use python, python3, etc...) by running this:
which python3
And then, I replaced python3 for the full path of python3 in my crontab file.