Eventually I understand this and it works.
bash script:
#!/bin/bash
#$ -V
#$ -cwd
#$ -o $HOME/sge_jobs_output/$JOB_ID.out -j y
#$ -S /bin/bash
#$ -l mem_free=4G
c=$SGE_TASK_ID
cd /home/xxx/scratch/test/
FILENAME=`head -$c testlist|tail -1`
python testpython.py $FILENAME
python script:
#!/bin/python
import sys,os
path='/home/xxx/scratch/test/'
name1=sys.argv[1]
job_id=os.path.join(path+name1)
f=open(job_id,'r').readlines()
print f[1]
thx
Exported bash variables are actually environment variables. You get at them through the os.environ object with a dictionary-like interface. Note that there are two types of variables in Bash: those local to the current process, and those that are inherited by child processes. Your Python script is a child process, so you need to make sure that you export the variable you want the child process to access.
To answer your original question, you need to first export the variable and then access it from within the python script using os.environ.
##!/bin/bash
#$ -V
#$ -cwd
#$ -o $HOME/sge_jobs_output/$JOB_ID.out -j y
#$ -S /bin/bash
#$ -l mem_free=4G
c=$SGE_TASK_ID
cd /home/xxx/scratch/test/
export FILENAME=`head -$c testlist|tail -1`
chmod +X testpython.py
./testpython.py
#!/bin/python
import sys
import os
for arg in sys.argv:
print arg
f=open('/home/xxx/scratch/test/' + os.environ['FILENAME'],'r').readlines()
print f[1]
Alternatively, you may pass the variable as a command line argument, which is what your code is doing now. In that case, you must look in sys.argv, which is the list of arguments passed to your script. They appear in sys.argv in the same order you specified them when invoking the script. sys.argv[0] always contains the name of the program that's running. Subsequent entries contain other arguments. len(sys.argv) indicates the number of arguments the script received.
#!/bin/python
import sys
import os
if len(sys.argv) < 2:
print 'Usage: ' + sys.argv[0] + ' <filename>'
sys.exit(1)
print 'This is the name of the python script: ' + sys.argv[0]
print 'This is the 1st argument: ' + sys.argv[1]
f=open('/home/xxx/scratch/test/' + sys.argv[1],'r').readlines()
print f[1]
use this inside your script (EDITED per Aarons suggestion):
def main(args):
do_something(args[0])
if __name__ == "__main__":
import sys
main(sys.argv[1:])
Take a look at parsing Python arguments. Your bash code would be fine, just need to edit your Python script to take the argument.
Command line arguments to the script are available as sys.argv list.
Related
I have a script (test.py) that I run as a command from the terminal, which calls another script (main.py). I want to store the arguments entered by the user as variables, and then pass them onto the second script.
E.g. when I run 'test -t foo' I want to save 'foo' as 'test=foo', and then when I call 'os.system("python main.py")' at the end of test.py I want main.py to print 'foo'.
This is what I have so far:
test.py
import os, argparse
parser = argparse.ArgumentParser()
parser.add_argument("-t", "--test", action="store", help="Store argument as variable")
args = parser.parse_args()
#I'm not sure how to save the argument as a variable
os.system("python main.py") #I need to keep this line - please see the comments below
terminal commands
chmod +x test.py
mv test.py test
mkdir -p ~/bin
cp test ~/bin
echo 'export PATH=$PATH":$HOME/bin"' >> .profile
main.py
from __main__ import * #this does not work
if args.test:
print(#variable)
In case it is helpful for anyone, I have found a way around the problem:
test.py
import os, argparse
parser = argparse.ArgumentParser()
parser.add_argument("-t", "--test", action="store", type=str, default="None", help="Store argument as variable")
args = parser.parse_args()
with open ("variables.py", 'w') as variables:
variables.writelines(args.test)
os.system("python main.py")
terminal commands
chmod +x test.py
mv test.py test
mkdir -p ~/bin
cp test ~/bin
echo 'export PATH=$PATH":$HOME/bin"' >> .profile
main.py
with open ("variables.py", 'r') as variables:
test = variables.read()
print(test)
It's probably not very pythonic but it does the trick.
This question already has answers here:
Call Python script from bash with argument
(8 answers)
Closed 11 months ago.
I'm working with an HPC where I write a bash script which executes my python_script.py job like so:
#!/bin/bash -l
#$ -l h_rt=0:00:10
#$ -l mem=1G
#$ -cwd
module load python3/3.8
python3 python_script.py
In my python_script.py I define the variables repeat, and directory like so:
repeat = 10
directory = r'User/somedir/'
I would like to be able to set these variables in my bash script, so that they overwrite these values within python_script.py and are used instead.
The easier way is to use in your python script :
import sys
repeat = sys.argv[1]
directory = sys.argv[2]
and in your bash script
repeat="10"
directory="User/somedir/"
python3 python_script.py $repeat $directory
I am manually running file with command line
python script.py input_files/input.txt text_out/output.json
while inside script.py
there is
input_path = sys.argv[1]
out_path = sys.argv[2]
now I have shell script and I want to make it for all files in one go.
I am facing issue.
My shell script is like below
there are two folders 1) input_files and 2) text_out
for i in input_files/*.txt;
do name=`echo "$i" | cut -d'.' -f1`
echo "$name"
python script.py -i "$i" text_out/"${name}.json"
done
but when I execude .sh as stated above, it is throwing error as sys.argv is not picking properly.
out_path = sys.argv[2]
IndexError: list index out of range
If you can guide what to change in .py or in shell .sh script would be kind.
I don't know exactly why you're getting a ListIndexOutOfRange, but it doesn't really matter, since you're also passing -i after script.py, so out_path cannot be what you expect.
$ cat script.py
import sys; print(len(sys.argv)); print(sys.argv); print({i:v for i, v in enumerate(sys.argv)})
$ (set -x; i=input_files/foo.txt; name=`echo "$i" | cut -d'.' -f1`; python script.py -i "$i" text_out/"${name}.json")
+ i=input_files/foo.txt
++ echo input_files/foo.txt
++ cut -d. -f1
+ name=input_files/foo
+ python script.py -i input_files/foo.txt text_out/input_files/foo.json
4
['script.py', '-i', 'input_files/foo.txt', 'text_out/input_files/foo.json']
{0: 'script.py', 1: '-i', 2: 'input_files/foo.txt', 3: 'text_out/input_files/foo.json'}
I recommend using argparse whenever you need to deal with cli arguments in Python. It will give better feedback and reduce the ambiguity of looking directly at indices.
$ cat script2.py
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-i', '--input-file', type=argparse.FileType('r'))
parser.add_argument('output_file', type=argparse.FileType('w'))
print(parser.parse_args())
$ (set -x; i=input_files/foo.txt; name=`echo "$i" | cut -d'.' -f1`; python script2.py -i "$i" text_out/"${name}.json")
+ i=input_files/foo.txt
++ echo input_files/foo.txt
++ cut -d. -f1
+ name=input_files/foo
+ python script2.py -i input_files/foo.txt text_out/input_files/foo.json
Namespace(input_file=<_io.TextIOWrapper name='input_files/foo.txt' mode='r' encoding='UTF-8'>, output_file=<_io.TextIOWrapper name='text_out/input_files/foo.json' mode='w' encoding='UTF-8'>)
for i in input_files/*.txt;
do name=`echo "$i" | cut -d'.' -f1`
echo "$name"
python script.py "$i" text_out/"${name}.json"
done
You get the IndexError: list index out of range error because you try to access the list at index 2 for out_path (in your Python script).
But in your shell script you're just passing one argument in the Python script (at this line: python script.py text_out/"${name}.json").
Your first example (python script.py input_files/input.txt text_out/output.json) works ofc, because you're just passing two arguments into the Python script. That's why you can access sys.argv easily at index 1 and 2.
You should check the length of len(sys.argv) to know how many arguments are passed into the Python script.
Example Shell Script
Your shell script should look something like this to get rid if the IndexError:
for i in input_files/*.txt;
do name=`echo "$i" | cut -d'.' -f1`
echo "$name"1
# Pass two args into your Python script
python script.py input.txt output.txt
done
I have a file named "uscf" in /usr/local/bin:
#! /bin/sh
python3 ~/Desktop/path/to/uscf.py
I have already chmod +x this file so that I can run it from my terminal with the command "uscf". How can I run this with command line arguments so that the arguments are accessible through sys.argv in uscf.py?
EDIT: Added below example for clarification:
The uscf.py file:
import sys
if len(sys.argv) > 1:
print(sys.argv)
Running it from the command line:
Abraham$ uscf these are arguments
Expected output:
these are arguments
In sh the "$#" variable contains all the positional arguments passed to the script. You can use that to pass it to your python script:
#!/bin/sh
python3 $HOME/Desktop/path/to/uscf.py "$#"
I do have a setenv.sh script which is called at the begining of several others scripts, most of them being bash scripts. Obviously this script does set some enviroment variables, which are user later by these scripts.
Now, the problem is that I want to implement the same behaviour into some python scripts and I discovered that the enviroment of the python script is not updated if you run the setenv.
As I do not want to create another script which calls first the setenv.sh and other this myscript.py --- I am looking for a way to convince python to load these variables (parsing the seteve.h is not an option... as it is more complex).
Here's a little library that does it:
https://github.com/aneilbaboo/shellvars-py
The easiest solution is clearly the one you don't want, which is to create a new script file for each python script.
You could, however, do roughly the equivalent by having the python script call itself. Of course, you need to signal it to not do that on the second invocation, otherwise you'll end up with an infinite (tail) recursion.
The following little "module" (which you can just import, but you should do it right at startup, before anything else) will check to see if the environment variable SETENV has been set, and if so, it will re-issue the python command (to the best of its ability, so it might get things wrong if it wasn't just a simple script execution) after sourcing the file named by SETENV. It lacks lots of error-checking and shouldn't be considered production-ready; rather a proof-of-concept:
# file env_set.py
import os
import sys
if sys.argv[0] and "SETENV" in os.environ:
setenv = os.environ["SETENV"]
del os.environ["SETENV"]
os.execvp("bash", ["bash", "-c",
"source " + setenv + "; exec python " + sys.argv[0] + ' "${#}"',
"--"] + sys.argv[1:])
And a little test:
# file test_env_set.py
import env_set
import os
import sys
for name in sys.argv[1:]:
if name in os.environ:
print(name + "=" + os.environ[name])
else:
print("Undefined: " + name)
# file setenv.sh
export the_answer=42
$ python test_env_set.py SETENV the_answer
Undefined: SETENV
Undefined: the_answer
$ SETENV=setenv.sh python test_env_set.py SETENV the_answer
Undefined: SETENV
the_answer=42
Perhaps try to run env before export and after and than compare the results. Something like this
$ pwd
/tmp/test
$ cat setenv.sh
#!/bin/bash
export test=1234
$ cat test.sh
#!/bin/bash
source /tmp/test/setenv.sh
echo $test
$ ./test.sh
1234
$ python test.py
test=1234
$ cat test.py
#/usr/bin/env python
import os, subprocess
p=subprocess.Popen('env',stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True)
oldEnv=p.communicate()[0]
p=subprocess.Popen('source /tmp/test/setenv.sh ; env',stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell = True)
newEnv=p.communicate()[0]
for newStr in newEnv.split('\n'):
flag = True
for oldStr in oldEnv.split('\n'):
if newStr == oldStr:
#not exported by setenv.sh
flag = False
break
if flag:
#exported by setenv.sh
print newStr