This question already has answers here:
Why does passing variables to subprocess.Popen not work despite passing a list of arguments?
(5 answers)
Closed 1 year ago.
I was wondering how I can pass a python variable to subprocess.check_output command.
In this particular case I have lower and upper python variables to be passed to the subprocess.check_output command, but I'm certain the way I have done it below isn't correct because it's not giving me the expected result.
If I input the values for the lower and upper bound values manually it does work.
for qq in range (0, 5, 1):
lo = glob.glob(path2 + "IM" + path1 + "*_GM.nii.gz")
lo = ' '.join(lo)
lower = qq - 0.5
upper = qq + 0.5
subprocess.check_output(['fslstats {} -l lower -u upper -V | cut -d " " -f 1'.format(lo)], shell=True)
Any suggestions how I can pass the lower and upper variables?
Note:
lo= /Users/say/Documents/awIM/network5/awfc_GM.nii.gz
path2=/Users/say/Documents/aw
path1=/network5/awfc
Thanks
Posted Community Wiki because this is a question already asked and answered elsewhere in the knowledgebase.
Doing this correctly (but for the removal of the cut in favor of native-Python string manipulation) might look something like:
glob_str = path2 + "IM" + path1 + "*_GM.nii.gz"
glob_list = glob.glob(glob_str)
if len(glob_list) == 0:
raise Exception("No results found from glob expression %r" % glob_str)
for qq in range (0, 5, 1):
lower = qq - 0.5
upper = qq + 0.5
args = ['fslstats'] + glob_list + [ '-l', str(lower), '-u', str(upper), '-V' ]
### EVERYTHING BELOW HERE IS UNNECESSARILY COMPLICATED BY THE USE OF 'cut'
### CONSIDER REPLACING WITH THE ALTERNATE IMPLEMENTATION BELOW.
p1 = subprocess.Popen(args, stdout=subprocess.PIPE)
p1.stdout.close()
p2 = subprocess.Popen(['cut', '-d', ' ', '-f1'], stdin=p1.stdout)
(stdout, _) = p2.communicate()
if p1.wait() != 0:
raise Exception("fslstats run as %r returned exit status %r" % (args, p1.returncode))
print("Result is: %r" % (stdout.split("\n"),))
To remove cut, you might change everything below the line assigning args as follows:
stdout = subprocess.check_output(args)
first_column = [ line.split()[0] for line in stdout.split('\n') ]
print("Result is: %r" % first_column)
Note:
We're not using shell=True. Keeping this disabled makes for an implementation where you have more control -- a shell isn't doing things behind your back, and you don't need to know how that shell works and how it's implemented to avoid (potentially security-impacting) bugs.
To implement a pipeline without shell=True, we're following the practices documented at https://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline
We're actually passing the values of the lower and upper variables, instead of passing the command lower and upper strings.
We're not joining our glob results into a string (which would break our command if any filenames resulting from that glob contained spaces), but are instead passing the list directly on the argument list for fslstats.
Because you care about the exit status of fslstats, not cut, you need to check that yourself. (Even with shell=True, you get default shell behavior, which returns only the exit status of the last pipeline component).
Related
I have a function that does some thing and displays a line of output mixed between integer and strings.
However, I just want to print out the last part of the output which is the number 5 that comes after the dots:
The number 5 is the value of the OID and it could be 5( as On)
Or 6(as OFF).
Is there any way how can I specify that in print or if condition?
Here is the function:
import subprocess, sys
p = subprocess.Popen(["powershell.exe",
"snmpwalk -v1 -c public 192.168.178.213 .1.3.6.1.4.1.9986.3.22.1.6.1.1.15"],
stdout=sys.stdout)
p.communicate()
def calc_execution():
import subprocess
get_pid_detectmotion = "pgrep -f detectmotion.py"
pidcmd = subprocess.Popen(get_pid_detectmotion.split(), stdout=subprocess.PIPE)
pidcmd, error = pidcmd.communicate()
#print pidcmd
#detectmotion_file_pid = int(out.rstrip())
get_length_pid_running="ps -o etime= -p" + pidcmd
length_pid_detectmotion_running = subprocess.Popen(get_length_pid_running.split())#, int(pidcmd))
print length_pid_detectmotion_running
print list(length_pid_detectmotion_running)
Outputs:
TypeError: 'Popen' object is not iterable
23:15:59
How can I convert the output of length_pid_detectmotion_running to a list, then get the closest value to the left, if there are (3). For example: 23:15:59 I want to print out 23 within a list like length_pid_detectmotion_running[0]
Popen is useful when you want to perform some parallel tasks, like controlled/modified printing line by line while program is running, expect output
Popen is a structure, and not directly iterable. To get the list of lines of the process' standard output, you should convert length_pid_detectmotion_running.stdout to list instead.
But in your case you should just use check_output and split:
output = subprocess.check_output(get_length_pid_running.split())
toks = output.split(":")
first element of toks should be your 23
I find out the following code in python:
def ExtractShellcodeArm(_arg_name):
ObjDumpOutput(_arg_name)
print("\033[101m\033[1mExtracted Shellcode:\033[0m\n")
proc = subprocess.Popen(['objdump','-d',_arg_name], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
line = proc.stdout.readline()
if line != b'':
array = line.decode().rstrip().split(':')
if len(array) > 1:
if array[1]:
array2 = array[1].split(' ')
array2 = array2[0].lstrip().rstrip()
if array2:
sc_part = '\t"'
sc_part += '\\x'
sc_part += '\\x'.join(a+b for a,b in zip(array2[::2], array2[1::2]))
sc_part += '"+'
print(sc_part)
else:
break
After I run this code in python3 it gives me the result of the objdump tools like the following:
"\xe2\x8f\x60\x01"+
"\xe1\x2f\xff\x16"+
"\x22\x0c"+
"\x46\x79"+
"\x31\x0e"+
"\x20\x01"+
"\x27\x04"+
"\xdf\x01"+
"\x1b\x24"+
"\x1c\x20"+
"\x27\x01"+
"\xdf\x01"+
"\x6c\x6c\x65\x48"+
"\x6f\x57\x20\x6f"+
"\x0a\x64\x6c\x72"+
But I want it shows the result in the big endian format. How can I change this represantion in python function. for example I want this code shows the result like the following:
"\x01\x60\x8f\xe2"+
"\x16\xff\x2f\xe1"+
"\x0c\x22"+
"\x79\x46"+
...
It's not the prettiest code, but this works:
''.join(a+b for a, b in zip(s[::-2], s[-2::-2]))
You should store each complete opcode (set of bytes) as an element in a list when you parse them, and then iterate over the list, flipping the bytes in the opcode one at a time. For example, rather than opcodes "\xcd\x80" + "\xeb\xfe" use opcodes = ["\xcd\x80", "\xeb\xfe". You should have no problem iterating over the list and reversing each opcode.
Another option is using shell utilities to reverse the bytes before they are received by Python by piping the objdump command to tools like sed and awk to do this by splitting up the bytes on each line into columns and then printing the columns backwards.
I've run into a couple of issues using arguments within a python script. Can i please get some help or direction to get this code functional? Thank you in advance.
First issue: I am unable to specify multiple arguments at once.
For example I am able to pass a single argument fine:
$ ./my_arg_scenario.py -a
Argument_A
$ ./my_arg_scenario.py -c
Argument_C
$ ./my_arg_scenario.py -d
Argument_D
However, I am looking for a way to pass multiple arguments in any position. Is there a way I can accomplish this?
For example, I would like the below to occur:
./my_arg_scenario.py -a -c -d
Argument_A
Argument_C
Argument_D
# OR
./my_arg_scenario.py -c -a
Argument_C
Argument_A
Second Issue: I am trying to pass both whole numbers and floats in the -b argument. But when I pass a float/decimal I get the below error. Is there a way I can pass both a float and whole number?
This works:
$ ./my_arg_scenario.py -b 5
The number provided is: 5
But this does NOT:
$ ./my_arg_scenario.py -b 5.50
Traceback (most recent call last):
File "./my_arg_scenario.py", line 18, in <module>
if int(sys.argv[2]) not in range(0,11):
ValueError: invalid literal for int() with base 10: '5.50'
Below is my testable code:
#!/usr/local/bin/python3.5
import sys
script_options = ['-a', '-b', '-c', '-d']
manual_flag = ''
build_flag = ''
if len(sys.argv) > 1:
if sys.argv[1] in script_options:
pass
else:
print('\n\t\tParameter "' + sys.argv[1] + '" is an invalid argument.\n')
sys.exit()
if sys.argv[1] == '-a':
print('Argument_A')
sys.exit()
elif sys.argv[1] == '-b':
if int(sys.argv[2]) not in range(0,11):
print('Invalid interval. Please select a value bewteen 1-5s.')
sys.exit()
else:
print('The number provided is: ' + (sys.argv[2]))
elif sys.argv[1] == '-c':
manual_flag = 'Argument_C'
print(manual_flag)
elif sys.argv[1] == '-d':
build_flag ='Argument_D'
print(build_flag)
else:
pass
You didn't actually provide the code you're using (aside from incidentally in the traceback),(Update: Code added later) but the answer is: Stop messing around with parsing sys.argv manually and use the argparse module (or docopt or something that doesn't involve rolling your own switch parsing).
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', action='store_true')
parser.add_argument('-b', metavar='INTERVAL', type=int, choices=range(11))
parser.add_argument('-c', action='store_true')
parser.add_argument('-d', action='store_true')
args = parser.parse_args()
if args.a: print('Argument_A')
if args.b is not None: print('The number provided is:', args.b)
if args.c: print('Argument_C')
if args.d: print('Argument_D')
If you want to accept int or float, the easiest solution is to just make type=float and use a consistent type (but the range check must be done outside the parsing step). If you must allow both, ast.literal_eval or a homegrown argparse type conversion function are options. Since you want a range check too (which range won't handle properly for float values that aren't equal to int values), roll a type checker:
def int_or_float(minval=None, maxval=None):
def checker(val):
try:
val = int(val)
except ValueError:
val = float(val)
if minval is not None and val < minval:
raise argparse.ArgumentTypeError('%r must be >= %r' % (val, minval))
if maxval is not None and val > maxval:
raise argparse.ArgumentTypeError('%r must be <= %r' % (val, maxval))
return val
return checker
Then use it by replacing the definition for -b with:
# Might want int_or_float(0, 10) depending on range exclusivity rules
parser.add_argument('-b', metavar='INTERVAL', type=int_or_float(0, 11))
I've got about 3,500 files that consist of single line character strings. The files vary in size (from about 200b to 1mb). I'm trying to compare each file with each other file and find a common subsequence of length 20 characters between two files. Note that the subsequence is only common between two files during each comparison, and not common among all files.
I've stuggled with this problem a bit, and since I'm not an expert, I've ended up with a bit of an ad-hoc solution. I use itertools.combinations to build a list in Python that ends up with around 6,239,278 combinations. I then pass the files two at a time to a Perl script that acts a wrapper for a suffix tree library written in C called libstree. I've tried to avoid this type of solution but the only comparable C suffix tree wrapper in Python suffers from a memory leak.
So here's my problem. I've timed it, and on my machine, the solution processes about 500 comparisons in 25 seconds. So that means, it'll take around 3 days of continuous processing to complete the task. And then I have to do it all again to look at say 25 characters instead of 20. Please note that I'm way out of my comfort zone and not a very good programmer, so I'm sure there is a much more elegant way to do this. I thought I'd ask it here and produce my code to see if anyone has any suggestion as to how I could complete this task faster.
Python code:
from itertools import combinations
import glob, subprocess
glist = glob.glob("Data/*.g")
i = 0
for a,b in combinations(glist, 2):
i += 1
p = subprocess.Popen(["perl", "suffix_tree.pl", a, b, "20"], shell=False, stdout=subprocess.PIPE)
p = p.stdout.read()
a = a.split("/")
b = b.split("/")
a = a[1].split(".")
b = b[1].split(".")
print str(i) + ":" + str(a[0]) + " --- " + str(b[0])
if p != "" and len(p) == 20:
with open("tmp.list", "a") as openf:
openf.write(a[0] + " " + b[0] + "\n")
Perl code:
use strict;
use Tree::Suffix;
open FILE, "<$ARGV[0]";
my $a = do { local $/; <FILE> };
open FILE, "<$ARGV[1]";
my $b = do { local $/; <FILE> };
my #g = ($a,$b);
my $st = Tree::Suffix->new(#g);
my ($c) = $st->lcs($ARGV[2],-1);
print "$c";
Rather than writing Python to call Perl to call C, I'm sure you would be better off dropping the Python code and writing it all in Perl.
If your files are certain to contain exactly one line then you can read the pairs more simply by writing just
my #g = <>;
I believe the program below performs the same function as your Python and Perl code combined, but I cannot test it as I am unable to install libstree at present.
But as ikegami has pointed out, it would be far better to calculate and store the longest common subsequence for each pair of files and put them into categories afterwards. I won't go on to code this as I don't know what information you need - whether it is just subsequence length or if you need the characters and/or the position of the subsequences as well.
use strict;
use warnings;
use Math::Combinatorics;
use Tree::Suffix;
my #glist = glob "Data/*.g";
my $iterator = Math::Combinatorics->new(count => 2, data => \#glist);
open my $fh, '>', 'tmp.list' or die $!;
my $n = 0;
while (my #pair = $iterator->next_combination) {
$n++;
#ARGV = #pair;
my #g = <>;
my $tree = Tree::Suffix->new(#g);
my $lcs = $tree->lcs;
#pair = map m|/(.+?)\.|, #pair;
print "$n: $pair[0] --- $pair[1]\n";
print $fh, "#pair\n" if $lcs and length $lcs >= 20;
}