We Keep Coding

subprocess.getstatus() return limited-length string

I use subprocess.getstatus() to get output of bash command running inside python script, in my case top linux command:
output = subprocess.getoutput("top -bc -n 1 | grep some_process_name")
Unfortunately, output string of the function is limited to 80 chars. If the string is longer, I just get the first 80 chars.
Any other alternate way to get long outputs in shell commands, in full?

You could pipe the output to a text file, and then display the text file.

execute one line of python with variable input from bash

I want to write one line of python and wrap in such a way than it will run from bash.
I looked into it and the idea seemd possible but, I'd specifically like to incorporate a variable input.
As an illustrative example; Here's what worked:
python -c "print('my favorite number to itself is '+str(5**5))"
However, I'd like to have a variable input so I tried to pipe one in, but the following doesn't work:
5 | python -c "print('my favorite number to itself is '+str($1**$1))"
What can I do to keep it simple, keep it one line, and add a variable input?

You could do:
i=5; python -c "print('my favorite number to itself is '+str($i**$i))"

With xargs:
echo 5 | xargs -I {} python -c "print('my favorite number to itself is '+str({}**{}))"

Send parameters to python from bash

I have a bash script that calls a python script with parameters.
In the bash script, I'm reading a file that contains one row of parameters separated by ", and then calls the python script with the line I read.
My problem is that the python gets the parameters separated by the space.
The line looks like this: "param_a" "Param B" "Param C"
Code Example:
Bash Script:
LINE=`cat $tmp_file`
id=`python /full_path/script.py $LINE`
Python Script:
print sys.argv[1]
print sys.argv[2]
print sys.argv[3]
Received output:
"param_a"
"Param
B"
Wanted output:
param_a
Param B
Param C
How can I send the parameters to the Python script the way I need?
Thanks!

What about
id=`python /full_path/script.py $tmp_file`
and
import sys
for line in open(sys.argv[1]):
print(line)
?

The issue is in how bash passes the arguments. Python has nothing do to with it.
So, you have to solve all these stuff before sending it to Python, I decided to use awk and xargs for this. (but xargs is the actual MVP here.)
LINE=$(cat $tmp_file)
awk -v ORS="\0" -v FPAT='"[^"]+"' '{for (i=1;i<=NF;i++){print substr($i,2,length($i)-2)}}' <<<$LINE |
xargs -0 python ./script.py
First $(..) is preferred over backticks, because it is more readable. You are making a variable after all.
awk only reads from stdin or a file, but you can force it to read from a variable with the <<<, also called "here string".
With awk I loop over all fields (as defined by the regex in the FPAT variable), and print them without the "".
The output record separator I choose is the NULL character (-v ORF='\0'), xargs will split on this character.
xargs will now parse the piped input by separating the arguments on NULL characters (set with -0) and execute the command given with the parsed arguments.
Note, while awk is found on most UNIX systems, I make use of FPAT which is a GNU awk extension and you might not be having GNU awk as default (for example Ubuntu), but gnu awk is usually just a install gawk away.
Also, the next command would be a quick and easy solution, but generally considered as unsafe, since eval will execute everything it receives.
eval "python ./script "$LINE

This can be done using bash arrays:
tmp_file='gash.txt'
# Set IFS to " which splits on double quotes and removes them
# Using read is preferable to using the external program cat
# read -a reads into the array called "line"
# UPPERCASE variable names are discouraged because of collisions with bash variables
IFS=\" read -ra line < "$tmp_file"
# That leaves blank and space elements in "line",
# we create a new array called "params" without those elements
declare -a params
for((i=0; i < ${#line[#]}; i++))
do
p="${line[i]}"
if [[ -n "$p" && "$p" != " " ]]
then
params+=("$p")
fi
done
# `backticks` are frowned upon because of poor readability
# I've called the python script "gash.py"
id=$(python ./gash.py "${params[#]}")
echo "$id"
gash.py:
import sys
print "1",sys.argv[1]
print "2",sys.argv[2]
print "3",sys.argv[3]
Gives:
1 param_a
2 Param B
3 Param C

Inserting python code in a bash script

I've got the following bash script:
#!/bin/bash
while read line
do
ORD=`echo $line | cut -c 7-21`
if [[ -r ../FASTA_SEC/${ORD}.fa ]]
then
WCR=`fgrep -o N ../FASTA_SEC/$ORD.fa | wc -l`
WCT=`wc -m < ../FASTA_SEC/$ORD.fa`
PER1=`echo print $WCR/$WCT.*100 | python`
WCTRIN=`fgrep -o N ../FASTA_SEC_EDITED/$ORD"_Trimmed.fa" | wc -l`
WCTRI=`wc -m < ../FASTA_SEC_EDITED/$ORD"_Trimmed.fa"`
PER2=`echo print $WCTRIN/$WCTRI.*100 | python`
PER3=`echo print $PER1-$PER2 | python`
echo $ORD $PER1 $PER2 $PER3 >> Log.txt
if [ $PER2 -ge 30 -a $PER3 -lt 10 ]
then
mv ../FASTA_SEC/$ORD.fa ./TRASH/$ORD.fa
mv ../FASTA_SEC_EDITED/$ORD"_Trimmed.fa" ./TRASH/$ORD"_Trimmed.fa"
fi
fi
done < ../READ/Data.txt
$PER variables are floating numbers as u might have noticed so I cannot use them normaly in the nested if conditional. I'd like to do this conditional iteration in python but I have no clue how do it whithin a bash script also I dont know how to import the value of the variables $PER2 and $PER3 into python. Could I write directly python code in the same bash script invvoking python somehow?
Thank you for your help, first time facing this.

You can use python -c CMD to execute a piece of python code from the command line. If you want bash to interpolate your environment variables, you should use double quotes around CMD.
You can return a value by calling sys.exit, but keep in mind that true and false in Python have the reverse meaning in bash.
So your code would be:
if python -c "import sys; sys.exit(not($PER2 > 30 and $PER3 < 10 ))"

It is possible to feed Python code to the standard input of python executable with the help of here document syntax:
variable=$(date)
python2.7 <<SCRIPT
print "The current date: %s" % "${variable}"
SCRIPT
In order to avoid parameter substitution (interpretation within the block), quote the first limit string: <<'SCRIPT'.
If you want to assign the output to a variable, use command substitution:
output=$(python2.7 <<SCRIPT
print "The current date: %s" % "${variable}"
SCRIPT
)
Note, it is not recommended to use back quotes for command substitution, as it is impossible to nest them, and the form $(...) is more readable.

maybe this helps?
$ X=4; Y=7; Z=$(python -c "print($X * $Y)")
$ echo $Z
28
python -c "str" takes "str" as input and runs it.
but then why not rewrite all in python? bash commands can nicely be executed with subprocess which is included in python or (need to install that) sh.

Eagerly return lines from stdin Python

I'm making a script which takes has some others script output piped into it. The other script takes a while to complete, and prints the progress onto the console along with the data I want to parse.
Since I'm piping the result to my script, I want to be able to do 2 things. As my input comes, I would like to echo it out onto the screen. After the command completes, I would like to have a list of lines that were passed via stdin.
My first though was to use a simple
for line in sys.stdin:
sys.stdout.write(line + '\n')
lines.append(line)
sys.stdout.flush()
but to my surprise, the command waits until stdin hits EOF, until it starts yielding lines.
My current workaround is this:
line = sys.stdin.readline()
lines = []
while line:
sys.stdout.write(line.strip() + '\n')
lines.append(line.strip())
sys.stdout.flush()
line = sys.stdin.readline()
But this does not always wait until the whole input is used.
Is there any other way to do this? It seems strange that the for solution behaves the way it does.

edited to answer your question regarding exiting on end of input
The workaround you describe, or something similar like this below appears to be necessary:
#!/usr/bin/env python
import sys
lines = []
while True:
line = sys.stdin.readline()
if not line:
break
line = line.rstrip()
sys.stdout.write(line + '\n')
lines.append(line)
sys.stdout.flush()
This is explained in the python man page, under the -u option:
-u Force stdin, stdout and stderr to be totally unbuffered. On
systems where it matters, also put stdin, stdout and stderr in
binary mode. Note that there is internal buffering in xread-
lines(), readlines() and file-object iterators ("for line in
sys.stdin") which is not influenced by this option. To work
around this, you will want to use "sys.stdin.readline()" inside
a "while 1:" loop.
I created a file dummy.py containing the code above, then ran this:
for i in 1 2 3 4 5; do sleep 5; echo $i; echo; done | ./dummy.py
This is the output:
harold_mac:~ harold$ for i in 1 2 3 4 5; do sleep 5; echo $i; done | ./dummy.py
1
2
3
4
5
harold_mac:~ harold$

Python uses buffered input. If you check with python --help you see:
-u : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
So try the unbuffered option with:
command | python -u your_script.py

Other people have already told you about the unbuffered output. I will just add a couple of thoughts:
often it is better to print debug info to stderr, and stderr output is usually unbuffered
it is simplier to delegate intermediate output to special tools. For example, there is a tee utility, that allows to split stdout of a previous command. Assuming you are in bash, you can print the intermediate output to stdout right away, and use process substitution instead of printing to a file (instead of awk you will call your python script):
$ python -c 'for i in range(5): print i+1' | tee >( awk '{print "from awk", $0**2 }')
1
2
3
4
5
from awk 1
from awk 4
from awk 9
from awk 16
from awk 25

You need to make 1) stdin in your python program and 2) stdout on the contrary side of the pipe both to be line buffered. To get this
1) use stdin = os.fdopen(sys.stdin.fileno(), 'r', 1) in your program;
2) use stdbuf -oL to change buffering mode of the output of the other program:
stdbuf -oL otherprogram | python yourscript.py