We Keep Coding

Windows "No such file or directory"

I am trying to run a bash script from my Python code. I am calling the script in a subprocess like so:
preprocessed = subprocess.check_output([
'bash',
'../Paraphrase_Demo/models/processing_utils/preprocess_data_givenBPE.sh',
phrase, src_lang, tgt_lang, this_dir
])
Here is the script preprocess_data_givenBPE.sh:
#!/usr/bin/env bash
REMOTE_PATH=$4/processing_utils
SCRIPTS=$REMOTE_PATH/tiny-moses
TOKENIZER=$SCRIPTS/tokenizer.perl
DETOKENIZER=$SCRIPTS/detokenizer_v2.perl
CLEAN=$SCRIPTS/clean-corpus-n.perl
NORM_PUNC=$SCRIPTS/normalize-punctuation.perl
REM_NON_PRINT_CHAR=$SCRIPTS/remove-non-printing-char.perl
BPEROOT=$4/zhen/tools/subword-nmt/subword_nmt
text=$1
flan=$2
BPE=$4/$2$3/data/bpecodes
echo $text | perl $NORM_PUNC -l $flan | perl $REM_NON_PRINT_CHAR |
perl $TOKENIZER -a -l $flan -q | python3 $BPEROOT/apply_bpe.py -c $BPE | cat
When I run my Python program I get the following output:
Can't open perl script "C:/Users/Administrator/source/repos/Paraphrasing/Paraphrase_Demo/models/processing_utils/tiny-moses/normalize-punctuation.perl": No such file or directory
Can't open perl script "C:/Users/Administrator/source/repos/Paraphrasing/Paraphrase_Demo/models/processing_utils/tiny-moses/remove-non-printing-char.perl": No such file or directory
Can't open perl script "C:/Users/Administrator/source/repos/Paraphrasing/Paraphrase_Demo/models/processing_utils/tiny-moses/tokenizer.perl": No such file or directory
python3: can't open file 'C:/Users/Administrator/source/repos/Paraphrasing/Paraphrase_Demo/models/zhen/tools/subword-nmt/subword_nmt/apply_bpe.py': [Errno 2] No such file or directory
I have copy and pasted each of those paths into powershell and ls'd to verify that they are correct, but it still says it can't find them. What is going on here? This also happens if I run the bash script directly.
Edit: I've tried using backslashes in the path and received the same error. I also found this post: Bash: cannot execute a perl script using an absolute path?, so I tried using a relative path from the directory of my Bash script but it still says "No such file or directory."

After lots of debugging, I found the issue. While the paths I listed exist if I ls them in powershell, typing bash in powershell doesn't just open a bash shell, it actually changes the directory structure. I think this may be related to the Windows Subsystem for Linux, but the result is that C: changes to /mnt/c once inside the bash shell. Replacing this in all my paths, I was able to run my scripts.

Run a script using full path

I have a folder called TEST with inside :
script.py
script.sh
The bash file is :
#!/bin/bash
# Run the python script
python script.py
If I run the bash file like this :
./TEST/script.sh
I have the following error :
python: can't open file 'script.py': [Errno 2] No such file or directory
How could I do, to tell my script.sh to look in the directory (which may change) and to allow me to run it for inside the TEST directory ?
Tricky, my python file run a sqlite database and I have the same problem when calling the script from outside the folder, it didn't look inside the folder to find the database!

Alternative
You are able to run the script directly by adding this line to the top of your python file:
#!/usr/bin/env python
and then making the file executable:
$ chmod +x script.py
With this, you can run the script directly with ./TEST/script.py
What you asked for specifically
This works to get the path of the script, and then pass that to python.
#!/bin/sh
SCRIPTPATH="$( cd "$(dirname "$0")" ; pwd -P )"
python "$SCRIPTPATH/script.py"
Also potentially useful:
You mentioned having this problem with accessing a sqlite DB in the same folder, if you are running this from a script to solve this problem, it will not work. I imagine this question may be of use to you for that problem: How do I get the path of a the Python script I am running in?

You could use $0 which is the name of the currently executing program, as invoked, combined with dirname which provides the directory component of a file path, to determine the path (absolute or relative) that the shell script was invoked under. Then, you can apply it to the python invocation.
This example worked for me:
$ t/t.sh
Hello, world!
$ cat t/t.sh
#!/bin/bash
python "$(dirname $0)/t.py"
Take it a step farther and change your current working directory which will also be inherited by python, thus helping it to find its database:
$ t/t.sh; cat t/t.sh ; cat t/t.py ; cat t/message.txt
hello, world!
#!/bin/bash
cd "$(dirname $0)"
python t.py
with(open('message.txt')) as msgf:
print(msgf.read())
hello, world!

From the shell script, you can always find your current directory: Getting the source directory of a Bash script from within. While the accepted answer to this question provide a very comprehensive and robust solution, your relatively simple case only really needs something like
#!/bin/bash
dir="$(dirname "${BASH_SOURCE[0]}")"
# Run the python script
python "$(dir)"/script.py
Another way to do it would be to change the directory from which you run the script:
#!/bin/bash
dir="$(dirname "${BASH_SOURCE[0]}")"
# Run the python script
(cd "$dir"; python script.py)
The parentheses ((...)) around cd and python create a subprocess, so that the directory does not change for the rest of your bash script. This may not be necessary if you don't do anything else in the bash portion, but is still useful to have if you ever decide to say source your script instead of running it as a subprocess.
If you do not change the directory in bash, you can do it in Python using a combination of sys.argv\[0\], os.path.dirname and os.chdir:
import sys
import os
...
os.chdir(os.path.dirname(sys.argv[0]))

Using bash to run python script

I have a python script that takes in a .txt file and outputs a .txt file. I want to create a bash file that I can click on from my desktop to execute the python script.
So far I have:
#!/bin/bash
cd /Desktop;
cd ./py-data;
python ./grab.py;
exit;
This just opens up the python script. Ideally I would like to click on the bash script and have the python script to run in the back ground and just produce the output without having to open the python script.
Solution:
change py.bat to py.command
at terminal:
$ cd ~/Desktop
$ chmod 755 py.command
open code on vim and place in (from: ./configure : /bin/sh^M : bad interpreter):
:set fileformat=unix
Changed code too:
#!bin/bash
cd ~/Desktop
python search.py
exit;

On MacOS, set the name of your script file to end in .command, and make the file executable. With that filename extension, you can double-click the file, and it will run Terminal application and any output (to stdout / stderr) will be displayed in the terminal window which pops up for execution.
=== /Users/john/Desktop/foo.command ===
#!/bin/bash
echo 'hello'
Then:
=== At command prompt===
$ cd ~/Desktop
$ chmod 755 foo.command
Double click on foo.command and you'll see window popup:
Last login: Thu Oct 5
/Users/john/Desktop/foo.command ; exit;
iMac:~/Desktop john$ /Users/john/Desktop/foo.command ; exit;
hello
logout
[Process completed]
In the popup window, you'll see lots of lines, plus your output ("hello").
In your particular example, I think you have two problems:
First, you mention /Desktop, which probably isn't what you want, as the user's Desktop is ~/Desktop. This would cause your script to fail.
Second, the output you would see in the popup window is the output your script writes directly to standard out. If you script is writing to another file, it may be working great, but you'll not see that information displayed in the popup (it will be in whatever file you wrote it to.) So it depends what your grab.py file actually does.
Finally, you say "run in the background". Technically, that's not what's happening, as it will run in a separate foreground process. I assume that's what you mean.

You could try chmod 777 on your bash script and then removing the extension. If you are on macOS then it should work. Not so sure about Linux, but it's worth a try.

chmod +x *filename*
Execute the above command in the terminal then make the bash file as follows
cd Desktop/*folder*/
python3 *filename.py*

run a python script in terminal without the python command

I have a python script let's name it script1.py. I can run it in the terminal this way:
python /path/script1.py
...
but I want to run like a command-line program:
arbitraryname
...
how can i do it ?

You use a shebang line at the start of your script:
#!/usr/bin/env python
make the file executable:
chmod +x arbitraryname
and put it in a directory on your PATH (can be a symlink):
cd ~/bin/
ln -s ~/some/path/to/myscript/arbitraryname

There are three parts:
Add a 'shebang' at the top of your script which tells how to execute your script
Give the script 'run' permissions.
Make the script in your PATH so you can run it from anywhere.
Adding a shebang
You need to add a shebang at the top of your script so the shell knows which interpreter to use when parsing your script. It is generally:
#!path/to/interpretter
To find the path to your python interpretter on your machine you can run the command:
which python
This will search your PATH to find the location of your python executable. It should come back with a absolute path which you can then use to form your shebang. Make sure your shebang is at the top of your python script:
#!/usr/bin/python
Run Permissions
You have to mark your script with run permissions so that your shell knows you want to actually execute it when you try to use it as a command. To do this you can run this command:
chmod +x myscript.py
Add the script to your path
The PATH environment variable is an ordered list of directories that your shell will search when looking for a command you are trying to run. So if you want your python script to be a command you can run from anywhere then it needs to be in your PATH. You can see the contents of your path running the command:
echo $PATH
This will print out a long line of text, where each directory is seperated by a semicolon. Whenever you are wondering where the actual location of an executable that you are running from your PATH, you can find it by running the command:
which <commandname>
Now you have two options: Add your script to a directory already in your PATH, or add a new directory to your PATH. I usually create a directory in my user home directory and then add it the PATH. To add things to your path you can run the command:
export PATH=/my/directory/with/pythonscript:$PATH
Now you should be able to run your python script as a command anywhere. BUT! if you close the shell window and open a new one, the new one won't remember the change you just made to your PATH. So if you want this change to be saved then you need to add that command at the bottom of your .bashrc or .bash_profile

Add the following line to the beginning script1.py
#!/usr/bin/env python
and then make the script executable:
$ chmod +x script1.py
If the script resides in a directory that appears in your PATH variable, you can simply type
$ script1.py
Otherwise, you'll need to provide the full path (either absolute or relative). This includes the current working directory, which should not be in your PATH.
$ ./script1.py

You need to use a hashbang. Add it to the first line of your python script.
#! <full path of python interpreter>
Then change the file permissions, and add the executing permission.
chmod +x <filename>
And finally execute it using
./<filename>
If its in the current directory,

Shell Script: Execute a python program from within a shell script

I've tried googling the answer but with no luck.
I need to use my works supercomputer server, but for my python script to run, it must be executed via a shell script.
For example I want job.sh to execute python_script.py
How can this be accomplished?

Just make sure the python executable is in your PATH environment variable then add in your script
python path/to/the/python_script.py
Details:
In the file job.sh, put this
#!/bin/sh
python python_script.py
Execute this command to make the script runnable for you : chmod u+x job.sh
Run it : ./job.sh

Method 1 - Create a shell script:
Suppose you have a python file hello.py
Create a file called job.sh that contains
#!/bin/bash
python hello.py
mark it executable using
$ chmod +x job.sh
then run it
$ ./job.sh
Method 2 (BETTER) - Make the python itself run from shell:
Modify your script hello.py and add this as the first line
#!/usr/bin/env python
mark it executable using
$ chmod +x hello.py
then run it
$ ./hello.py

Save the following program as print.py:
#!/usr/bin/python3
print('Hello World')
Then in the terminal type:
chmod +x print.py
./print.py

You should be able to invoke it as python scriptname.py e.g.
# !/bin/bash
python /home/user/scriptname.py
Also make sure the script has permissions to run.
You can make it executable by using chmod u+x scriptname.py.

Imho, writing
python /path/to/script.py
Is quite wrong, especially in these days. Which python? python2.6? 2.7? 3.0? 3.1? Most of times you need to specify the python version in shebang tag of python file. I encourage to use #!/usr/bin/env python2 #or python2.6 or python3 or even python3.1 for compatibility.
In such case, is much better to have the script executable and invoke it directly:
#!/bin/bash
/path/to/script.py
This way the version of python you need is only written in one file. Most of system these days are having python2 and python3 in the meantime, and it happens that the symlink python points to python3, while most people expect it pointing to python2.

This works for me:
Create a new shell file job. So let's say:
touch job.sh and add command to run python script (you can even add command line arguments to that python, I usually predefine my command line arguments).
chmod +x job.sh
Inside job.sh add the following py files, let's say:
python_file.py argument1 argument2 argument3 >> testpy-output.txt && echo "Done with python_file.py"
python_file1.py argument1 argument2 argument3 >> testpy-output.txt && echo "Done with python_file1.py"
Output of job.sh should look like this:
Done with python_file.py
Done with python_file1.py
I use this usually when I have to run multiple python files with different arguments, pre defined.
Note: Just a quick heads up on what's going on here:
python_file.py argument1 argument2 argument3 >> testpy-output.txt && echo "completed with python_file.py" .
Here shell script will run the file python_file.py and add multiple command-line arguments at run time to the python file.
This does not necessarily means, you have to pass command line arguments as well.
You can just use it like: python python_file.py, plain and simple.
Next up, the >> will print and store the output of this .py file in the testpy-output.txt file.
&& is a logical operator that will run only after the above is executed successfully and as an optional echo "completed with python_file.py" will be echoed on to your cli/terminal at run time.

This works best for me:
Add this at the top of the script:
#!c:/Python27/python.exe
(C:\Python27\python.exe is the path to the python.exe on my machine)
Then run the script via:
chmod +x script-name.py && script-name.py

I use this and it works fine
#/bin/bash
/usr/bin/python python python_script.py

Since the other posts say everything (and I stumbled upon this post while looking for the following).
Here is a way how to execute a python script from another python script:
Python 2:
execfile("somefile.py", global_vars, local_vars)
Python 3:
with open("somefile.py") as f:
code = compile(f.read(), "somefile.py", 'exec')
exec(code, global_vars, local_vars)
and you can supply args by providing some other sys.argv

Here I have demonstrated an example to run python script within a shell script. For different purposes you may need to read the output from a shell command, execute both python script and shell command within the same file.
To execute a shell command from python use os.system() method. To read output from a shell command use os.popen().
Following is an example which will grep all processes having the text sample_program.py inside of it. Then after collecting the process IDs (using python) it will kill them all.
#!/usr/bin/python3
import os
# listing all matched processes and taking the output into a variable s
s = os.popen("ps aux | grep 'sample_program.py'").read()
s = '\n'.join([l for l in s.split('\n') if "grep" not in l]) # avoiding killing the grep itself
print("To be killed:")
print(s)
# now manipulating this string s and finding the process IDs and killing them
os.system("kill -9 " + ' '.join([x.split()[1] for x in s.split('\n') if x]))
References:
Execute a python program from within a shell script
Assign output of os.system to a variable and prevent it from being displayed on the screen

If you have a bash script and you need to run inside of it a python3 script (with external modules), I recommend that you point in your bash script to your python path like this.
#!/usr/bin/env bash
-- bash code --
/usr/bin/python3 your_python.py
-- bash code --