We Keep Coding

How to embed Python code in a fish script?

I'm trying to convert over a Bash script that includes the following commands:
PYCODE=$(cat << EOF
#INSERT_PYTHON_CODE_HERE
EOF
)
RESPONSE=$(COLUMNS=999 /usr/bin/env python3 -c "$PYCODE" $#)
The idea being that a sed find/replace is then used to inject an arbitrary Python script where #INSERT_PYTHON_CODE_HERE is, creating the script that is then ran.
The corresponding Fish command would seem to be something like this
set PYCODE "
#INSERT_PYTHON_CODE_HERE
"
set RESPONSE (COLUMNS=999 /usr/bin/env python3 -c "$PYCODE" $argv)
but this falls apart when you have a Python script that can include both ' and " (and any other valid) characters.
What is the correct way to handle translate this use of EOF?
As a side note, I would prefer not to modify the sed command that is injecting the python code, but for reference here it is:
set FISH_SCRIPT (sed -e "/#INSERT_PYTHON_CODE_HERE/r $BASE_DIR/test_sh.py" $BASE_DIR/../src/mfa.fish)

Send parameters to python from bash

I have a bash script that calls a python script with parameters.
In the bash script, I'm reading a file that contains one row of parameters separated by ", and then calls the python script with the line I read.
My problem is that the python gets the parameters separated by the space.
The line looks like this: "param_a" "Param B" "Param C"
Code Example:
Bash Script:
LINE=`cat $tmp_file`
id=`python /full_path/script.py $LINE`
Python Script:
print sys.argv[1]
print sys.argv[2]
print sys.argv[3]
Received output:
"param_a"
"Param
B"
Wanted output:
param_a
Param B
Param C
How can I send the parameters to the Python script the way I need?
Thanks!

What about
id=`python /full_path/script.py $tmp_file`
and
import sys
for line in open(sys.argv[1]):
print(line)
?

The issue is in how bash passes the arguments. Python has nothing do to with it.
So, you have to solve all these stuff before sending it to Python, I decided to use awk and xargs for this. (but xargs is the actual MVP here.)
LINE=$(cat $tmp_file)
awk -v ORS="\0" -v FPAT='"[^"]+"' '{for (i=1;i<=NF;i++){print substr($i,2,length($i)-2)}}' <<<$LINE |
xargs -0 python ./script.py
First $(..) is preferred over backticks, because it is more readable. You are making a variable after all.
awk only reads from stdin or a file, but you can force it to read from a variable with the <<<, also called "here string".
With awk I loop over all fields (as defined by the regex in the FPAT variable), and print them without the "".
The output record separator I choose is the NULL character (-v ORF='\0'), xargs will split on this character.
xargs will now parse the piped input by separating the arguments on NULL characters (set with -0) and execute the command given with the parsed arguments.
Note, while awk is found on most UNIX systems, I make use of FPAT which is a GNU awk extension and you might not be having GNU awk as default (for example Ubuntu), but gnu awk is usually just a install gawk away.
Also, the next command would be a quick and easy solution, but generally considered as unsafe, since eval will execute everything it receives.
eval "python ./script "$LINE

This can be done using bash arrays:
tmp_file='gash.txt'
# Set IFS to " which splits on double quotes and removes them
# Using read is preferable to using the external program cat
# read -a reads into the array called "line"
# UPPERCASE variable names are discouraged because of collisions with bash variables
IFS=\" read -ra line < "$tmp_file"
# That leaves blank and space elements in "line",
# we create a new array called "params" without those elements
declare -a params
for((i=0; i < ${#line[#]}; i++))
do
p="${line[i]}"
if [[ -n "$p" && "$p" != " " ]]
then
params+=("$p")
fi
done
# `backticks` are frowned upon because of poor readability
# I've called the python script "gash.py"
id=$(python ./gash.py "${params[#]}")
echo "$id"
gash.py:
import sys
print "1",sys.argv[1]
print "2",sys.argv[2]
print "3",sys.argv[3]
Gives:
1 param_a
2 Param B
3 Param C

How to pass parameters to a python script that will be run using a bash script?

I have a python script which actually does some test and write the results to a file. This python file must be provided with list of arguments:
myPc#myPc:~/Desktop/tests$ ./testScript.py --final fromTo 201983:310029 -o res.txt
Every time I need new result for different ranges I have to do them manually so I decided to write a bash script as follows:
#!/bin/bash
a=`wc -l < inFile.txt`
declare -a periods
declare -i index=0
echo "Number of Periods is: $a"
while IFS='' read -r line || [[ -n "$line" ]]; do
periods[$index]=$line
((index += 1))
done < "inFile.txt"
for ((i=0;i<1300;i++)); do
f1=${period[$i]}
f2=${period[$i+1]}
python testScript.py "--final fromTo $f1:$f2 -o res$i.txt"
done
#END
The ranges are already in a text file so I am reading them and store them into an array.
I wish to use the bash to re-run every time the python script with different ranges until all ranges are covered and all results for all ranges are wrote in a separate file. But I am facing an issue that the parameters I try to pass to python are not even passed. I don't really understand what is the issue in here.
"--final fromTo $f1:$f2 -o res$i.txt"
this is the parameter I want to pass and once finished I will pass new params and run it again but it seems it is not even looking at them

The quotes around arguments:
python testScript.py "--final fromTo $f1:$f2 -o res$i.txt"
are causing the complete string to be passed as a single argument with spaces to Python script.
By removing quotes (or at least putting them around words):
python testScript.py --final fromTo "$f1:$f2" -o "res$i.txt"
the sys.argv will be populated correctly, and your argument parser will work (note that argparse uses sys.argv by default).
As a little proof, test.py script:
#!/usr/bin/env python
import sys
print(sys.argv)
Now try:
$ python test.py 1 2 3
['test.py', '1', '2', '3']
$ python test.py "1 2 3"
['test.py', '1 2 3']
Unrelated to this, but for completeness' sake, your bash script can be simplified by using the mapfile built-in (aka readarray, available since bash 4.0):
#!/bin/bash
mapfile -t periods <inFile.txt
cnt="${#periods[#]}"
echo "Number of Periods is: $cnt"
for ((i=0; i<cnt; i++)); do
python testScript.py --final fromTo "${period[$i]}:${period[$i+1]}" -o "res$i.txt"
done

What you type on the commandline is six strings: "./testScript.py" "--final" "fromTo" "201983:310029" "-o" "res.txt"
$ cat args.py
#!/usr/bin/env python
import sys
print sys.argv
$ ./args.py --final fromTo 201983:310029 -o res.txt
['./args.py', '--final', 'fromTo', '201983:310029', '-o', 'res.txt']
When you put double-quotes around the last four strings, you force your shell to treat them as a single string. Not Python's fault.
./args.py "--final fromTo 201983:310029 -o res.txt"
['./args.py', '--final fromTo 201983:310029 -o res.txt']
Just put the quotes around the items that must be handled as single strings:
$ f1=201983; f2=310029; i=1
$ ./args.py --final fromTo "$f1:$f2" -o "res$i.txt"
['./args.py', '--final', 'fromTo', '201983:310029', '-o', 'res1.txt']

write to file with tee command in python

I have a python script with variable names e.g V and It. I make a filename based on the parameter as following:
file_out=io.open("file_Iter" + str(It) + "_V_" +str(V)+".txt", 'w')
Then I'd like to redirect all my terminal output to a this file, so I use this command:
os.system("echo - START RUN $(LANG=en_US date +%b_%d_%Y_%k_%M)- | tee -a $file_out")
The file_out is created and also the echo command is shown correctly on the terminal but it is not written to file_out. If in tee command, I put e.g tee testfile.txt, then this file is created and also echo command writes in it.
Q: How should I change tee in order to write to the file that is created by variable names?

I'm not sure I understood it correctly, but I guess that what you want to do is the following:
fileName = "file_Iter" + str(It) + "_V_" +str(V)+".txt"
file_out=io.open(fileName, 'w')
os.system("echo - START RUN $(LANG=en_US date +%b_%d_%Y_%k_%M)- | tee -a " + fileName)
Pay attention to the end of the command, where fileName is concatenated.

$file_out refers to a shell variable -- a variable in the context of the shell that you're executing this command. However, file_out is a Python variable in the context of your Python script.
So, we need to pass this from Python into the shell. Luckily, Python provides an easy way to do this with string formatting commands. This will do what you want, I think:
os.system("echo - START RUN $(LANG=en_US date +%b_%d_%Y_%k_%M)- | " +
"tee -a {}".format(file_out))
However, there are a couple concerns with this. First, you should be using subprocess.Popen instead of os.system -- it's a newer module that's intended to replace os.system.
Also, you should be aware that, depending on the content of V and It, it may be possible for this shell to run unexpected commands. Imagine that V has the value '.txt ; sudo rm -rf / #'. Now your file name is 'file_IterIt_V_.txt ; sudo rm -rf / #.txt', and your shell command is 'echo - START RUN $(LANG=en_US date +%b_%d_%Y_%k_%M)- | tee -a file_IterIt_V_.txt ; sudo rm -rf / #.txt', which will happily remove many files from your computer. If you're going to do this, you should make sure to escape the filename in your shell command.

Zip function for shell scripts

I'm trying to write a shell script that will make several targets into several different paths. I'll pass in a space-separated list of paths and a space-separated list of targets, and the script will make DESTDIR=$path $target for each pair of paths and targets. In Python, my script would look something like this:
for path, target in zip(paths, targets):
exec_shell_command('make DESTDIR=' + path + ' ' + target)
However, this is my current shell script:
#! /bin/bash
packages=$1
targets=$2
target=
set_target_number () {
number=$1
counter=0
for temp_target in $targets; do
if [[ $counter -eq $number ]]; then
target=$temp_target
fi
counter=`expr $counter + 1`
done
}
package_num=0
for package in $packages; do
package_fs="debian/tmp/$package"
set_target_number $package_num
echo "mkdir -p $package_fs"
echo "make DESTDIR=$package_fs $target"
package_num=`expr $package_num + 1`
done
Is there a Unix tool equivalent to Python's zip function or an easier way to retrieve an element from a space-separated list by its index? Thanks.

Use an array:
#!/bin/bash
packages=($1)
targets=($2)
if (("${#packages[#]}" != "${#targets[#]}"))
then
echo 'Number of packages and number of targets differ' >&2
exit 1
fi
for index in "${!packages[#]}"
do
package="${packages[$index]}"
target="${targets[$index]}"
package_fs="debian/tmp/$package"
mkdir -p "$package_fs"
make "DESTDIR=$package_fs" "$target"
done

Here is the solution
paste -d ' ' paths targets | sed 's/^/make DESTDIR=/' | sh
paste is equivalent of zip in shell. sed is used to prepend the make command (using regex) and result is passed to sh to execute

There's no way to do that in bash. You'll need to create two arrays from the input and then iterate through a counter using the values from each.