We Keep Coding

Python store output as a variable after running an executable command

"TMalign..." is an executable file that I used to get data. How could I store the output into a variable so that I could extract target values from the output. The executable file is compiled from a long .cpp, so I do not think I could call the variable names from there.
import sys,os
os.system("./TMalign 3w4u.pdb 6bb5.pdb -u 139") #some command I have
The output is like, and I need to extract the TM-score values:
*********************************************************************
* TM-align (Version 20190822): protein structure alignment *
* References: Y Zhang, J Skolnick. Nucl Acids Res 33, 2302-9 (2005) *
* Please email comments and suggestions to yangzhanglab#umich.edu *
*********************************************************************
Name of Chain_1: 3w4u.pdb (to be superimposed onto Chain_2)
Name of Chain_2: 6bb5.pdb
Length of Chain_1: 141 residues
Length of Chain_2: 139 residues
Aligned length= 139, RMSD= 1.07, Seq_ID=n_identical/n_aligned= 0.590
TM-score= 0.94726 (if normalized by length of Chain_1, i.e., LN=141, d0=4.42)
TM-score= 0.96044 (if normalized by length of Chain_2, i.e., LN=139, d0=4.38)
TM-score= 0.96044 (if normalized by user-specified LN=139.00 and d0=4.38)
(You should use TM-score normalized by length of the reference structure)
(":" denotes residue pairs of d < 5.0 Angstrom, "." denotes other aligned residues)
SLTKTERTIIVSMWAKISTQADTIGTETLERLFLSHPQTKTYFPHFDLHPGSAQLRAHGSKVVAAVGDAVKSIDDIGGALSKLSELHAYILRVDPVNFKLLSHCLLVTLAARFPADFTAEAHAAWDKFLSVVSSVLTEKYR
:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::. .
-LSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTYFPHFDLSHGSAQVKGHGKKVADALTNAVAHVDDMPNALSALSDLHAHKLRVDPVNFKLLSHCLLVTLAAHLPAEFTPAVHASLDKFLASVSTVLTSK-Y
Total CPU time is 0.03 seconds
Thanks for help!

You should look toward the following approach:
import re
from subprocess import check_output
ret = check_output(['./TMalign', '3w4u.pdb', '6bb5.pdb', '-u', '139'])
tm_scores = []
for line in str(ret).split('\\n'):
if re.match(r'^TM-score=', line):
score = line.split()[1:2] # Extract the value
tm_scores.extend(score) # Saving only values
# tm_scores now contains: ['0.94726', '0.96044', '0.96044']
While it being somewhat elaborate, it is a flexible and tunable solution. Note, if it will be used among other code, it would be better to wrap this into a function.

My function wasn't that smart,I will let ouput write in to a file to do the follow
import os
cmd = './TMalign 3w4u.pdb 6bb5.pdb -u 139'
os.system(cmd + ">> 1.txt")

passing a variable to a subprocess [duplicate]

This question already has answers here:
Why does passing variables to subprocess.Popen not work despite passing a list of arguments?
(5 answers)
Closed 1 year ago.
I was wondering how I can pass a python variable to subprocess.check_output command.
In this particular case I have lower and upper python variables to be passed to the subprocess.check_output command, but I'm certain the way I have done it below isn't correct because it's not giving me the expected result.
If I input the values for the lower and upper bound values manually it does work.
for qq in range (0, 5, 1):
lo = glob.glob(path2 + "IM" + path1 + "*_GM.nii.gz")
lo = ' '.join(lo)
lower = qq - 0.5
upper = qq + 0.5
subprocess.check_output(['fslstats {} -l lower -u upper -V | cut -d " " -f 1'.format(lo)], shell=True)
Any suggestions how I can pass the lower and upper variables?
Note:
lo= /Users/say/Documents/awIM/network5/awfc_GM.nii.gz
path2=/Users/say/Documents/aw
path1=/network5/awfc
Thanks

Posted Community Wiki because this is a question already asked and answered elsewhere in the knowledgebase.
Doing this correctly (but for the removal of the cut in favor of native-Python string manipulation) might look something like:
glob_str = path2 + "IM" + path1 + "*_GM.nii.gz"
glob_list = glob.glob(glob_str)
if len(glob_list) == 0:
raise Exception("No results found from glob expression %r" % glob_str)
for qq in range (0, 5, 1):
lower = qq - 0.5
upper = qq + 0.5
args = ['fslstats'] + glob_list + [ '-l', str(lower), '-u', str(upper), '-V' ]
### EVERYTHING BELOW HERE IS UNNECESSARILY COMPLICATED BY THE USE OF 'cut'
### CONSIDER REPLACING WITH THE ALTERNATE IMPLEMENTATION BELOW.
p1 = subprocess.Popen(args, stdout=subprocess.PIPE)
p1.stdout.close()
p2 = subprocess.Popen(['cut', '-d', ' ', '-f1'], stdin=p1.stdout)
(stdout, _) = p2.communicate()
if p1.wait() != 0:
raise Exception("fslstats run as %r returned exit status %r" % (args, p1.returncode))
print("Result is: %r" % (stdout.split("\n"),))
To remove cut, you might change everything below the line assigning args as follows:
stdout = subprocess.check_output(args)
first_column = [ line.split()[0] for line in stdout.split('\n') ]
print("Result is: %r" % first_column)
Note:
We're not using shell=True. Keeping this disabled makes for an implementation where you have more control -- a shell isn't doing things behind your back, and you don't need to know how that shell works and how it's implemented to avoid (potentially security-impacting) bugs.
To implement a pipeline without shell=True, we're following the practices documented at https://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline
We're actually passing the values of the lower and upper variables, instead of passing the command lower and upper strings.
We're not joining our glob results into a string (which would break our command if any filenames resulting from that glob contained spaces), but are instead passing the list directly on the argument list for fslstats.
Because you care about the exit status of fslstats, not cut, you need to check that yourself. (Even with shell=True, you get default shell behavior, which returns only the exit status of the last pipeline component).

Convert popen output to list, and perform action

def calc_execution():
import subprocess
get_pid_detectmotion = "pgrep -f detectmotion.py"
pidcmd = subprocess.Popen(get_pid_detectmotion.split(), stdout=subprocess.PIPE)
pidcmd, error = pidcmd.communicate()
#print pidcmd
#detectmotion_file_pid = int(out.rstrip())
get_length_pid_running="ps -o etime= -p" + pidcmd
length_pid_detectmotion_running = subprocess.Popen(get_length_pid_running.split())#, int(pidcmd))
print length_pid_detectmotion_running
print list(length_pid_detectmotion_running)
Outputs:
TypeError: 'Popen' object is not iterable
23:15:59
How can I convert the output of length_pid_detectmotion_running to a list, then get the closest value to the left, if there are (3). For example: 23:15:59 I want to print out 23 within a list like length_pid_detectmotion_running[0]

Popen is useful when you want to perform some parallel tasks, like controlled/modified printing line by line while program is running, expect output
Popen is a structure, and not directly iterable. To get the list of lines of the process' standard output, you should convert length_pid_detectmotion_running.stdout to list instead.
But in your case you should just use check_output and split:
output = subprocess.check_output(get_length_pid_running.split())
toks = output.split(":")
first element of toks should be your 23

Using subprocess and .format

So I have this code so far:
from subprocess import call
a, b = 10000000, 10000100
call('samtools faidx file.fa chr22:{}-{}'.format(a, b), shell = True)
but when I run it, the numbers assigned to a and b does not seem to go into the {} brackets as format should do.
Am I using format wrong here, or is my code itself wrong?
(file.fa is a file that holds a DNA sequence for chromosome 22)

Reversing a byte string in Python

I find out the following code in python:
def ExtractShellcodeArm(_arg_name):
ObjDumpOutput(_arg_name)
print("\033[101m\033[1mExtracted Shellcode:\033[0m\n")
proc = subprocess.Popen(['objdump','-d',_arg_name], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if line != b'':
array = line.decode().rstrip().split(':')
if len(array) > 1:
if array[1]:
array2 = array[1].split(' ')
array2 = array2[0].lstrip().rstrip()
if array2:
sc_part = '\t"'
sc_part += '\\x'
sc_part += '\\x'.join(a+b for a,b in zip(array2[::2], array2[1::2]))
sc_part += '"+'
print(sc_part)
else:
break
After I run this code in python3 it gives me the result of the objdump tools like the following:
"\xe2\x8f\x60\x01"+
"\xe1\x2f\xff\x16"+
"\x22\x0c"+
"\x46\x79"+
"\x31\x0e"+
"\x20\x01"+
"\x27\x04"+
"\xdf\x01"+
"\x1b\x24"+
"\x1c\x20"+
"\x27\x01"+
"\xdf\x01"+
"\x6c\x6c\x65\x48"+
"\x6f\x57\x20\x6f"+
"\x0a\x64\x6c\x72"+
But I want it shows the result in the big endian format. How can I change this represantion in python function. for example I want this code shows the result like the following:
"\x01\x60\x8f\xe2"+
"\x16\xff\x2f\xe1"+
"\x0c\x22"+
"\x79\x46"+
...

It's not the prettiest code, but this works:
''.join(a+b for a, b in zip(s[::-2], s[-2::-2]))

You should store each complete opcode (set of bytes) as an element in a list when you parse them, and then iterate over the list, flipping the bytes in the opcode one at a time. For example, rather than opcodes "\xcd\x80" + "\xeb\xfe" use opcodes = ["\xcd\x80", "\xeb\xfe". You should have no problem iterating over the list and reversing each opcode.
Another option is using shell utilities to reverse the bytes before they are received by Python by piping the objdump command to tools like sed and awk to do this by splitting up the bytes on each line into columns and then printing the columns backwards.