So I am trying to execute a file and get the returned value back using the python builtin methods available in the subprocess library.
For example, lets say I want to execute this hello_world python file:
def main():
print("in main")
return("hello world!")
if __name__ == '__main__':
main()
I do not care about getting back the statement in main. What I want to get back is the return value hello world!.
I tried numerous things but non of them worked.
Here's a list of what I tried and their outputs:
args is common for all trials:
args = ['python',hello_cmd]
First trial:
p1 = subprocess.Popen(args, stdout=subprocess.PIPE)
print(p1.communicate())
print("returncode is:")
print(p1.returncode)
output is:
(b'in main\n', None)
returncode is:
0
second trial:
p2 = subprocess.check_output(args,stderr=subprocess.STDOUT)
print(p2)
output is:
b'in main\n'
third trial:
output, result = subprocess.Popen(args, stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell=False).communicate()
print(output)
print(result)
output is:
b'in main\n'
b''
fourth trial:
p4 = subprocess.run(args, stdout=PIPE, stderr=PIPE)
print(p4)
output is:
CompletedProcess(args=['python', '/path/to/file/hello.py'], returncode=0, stdout=b'in main\n', stderr=b'')
fifth trial:
p5 =subprocess.getstatusoutput(args)
print(p5)
output is:
(0, '')
Can anyone help?
The return value of the main function is not the return code that is passed to the OS. To pass a return code to the OS use sys.exit(), which expects an integer. You can pass it a string, but if you do, Python will pass 1 to the OS.
You cannot return strings as return codes it must be an integer. If you want to act differently depending on the process. Try to map your return code to some function in your main program. For example
def execute_sub_program(): ...
# somewhere else:
return_code = execute_sub_program()
if return_code == 0:
# do something ...
elif ...
You can try with subprocess.run().returncode, it gives 0 if successful execution and 1 if failed execution.
driver.py
import subprocess
args = ['python', './hello_cmd.py']
status_code = subprocess.run(args).returncode
print(["Successful execution", "Failed execution"][status_code])
For happy flow (hello_cmd.py):
def main():
print("in main")
return("hello world!")
if __name__ == '__main__':
main()
For failed flow (hello_cmd.py):
def main():
print("in main")
raise ValueError('Failed')
if __name__ == '__main__':
main()
Related
This is my main python script:
import time
import subprocess
def main():
while(True):
a=input("Please enter parameter to pass to subprocess:")
subprocess.Popen(args="python child.py")
print(f"{a} was started")
time.sleep(5)
if __name__ == '__main__':
main()
This is python child script named child.py:
def main(a):
while(True):
print(a)
if __name__ == '__main__':
main(a)
How to pass value to argument a which is in the child subprocess?
You need to use command line arguments, like this;
import time
import subprocess
def main():
while(True):
a=input("Please enter parameter to pass to subprocess:")
subprocess.Popen(["python", "child.py", a])
print(f"{a} was started")
time.sleep(5)
if __name__ == '__main__':
main()
child.py:
import sys
def main(a):
while(True):
print(a)
if __name__ == '__main__':
a = sys.argv[1]
main(a)
You may use subprocess.PIPE to pass data between your main process and spawned subprocess.
Main script:
import subprocess
def main():
for idx in range(3):
a = input(
'Please enter parameter to pass to subprocess ({}/{}): '
.format(idx + 1, 3))
print('Child in progress...')
pipe = subprocess.Popen(
args='python child.py',
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
pipe.stdin.write(str(a).encode('UTF-8'))
pipe.stdin.close()
print('Child output is:')
print(pipe.stdout.read().decode('UTF-8'))
if __name__ == '__main__':
main()
Child script:
import sys
import time
def main(a):
for dummy in range(3):
time.sleep(.1)
print(a)
if __name__ == '__main__':
a = sys.stdin.read()
main(a)
Output:
>>> python main.py
Please enter parameter to pass to subprocess (1/3): qwe
Child in progress...
Child output is:
qwe
qwe
qwe
Please enter parameter to pass to subprocess (2/3): qweqwe
Child in progress...
Child output is:
qweqwe
qweqwe
qweqwe
Please enter parameter to pass to subprocess (3/3): 123
Child in progress...
Child output is:
123
123
123
The easiest way to pass arguments to a child process is to use command line parameters.
The first step is to rewrite child.py so that it accepts command line arguments. There is detailed information about parsing command line arguments in this question: How to read/process command line arguments? For this simple example though, we will simply access the command line arguments through sys.argv.
import sys
def main(a):
while(True):
print(a)
if __name__ == '__main__':
# the first element in the sys.argv list is the name of our program ("child.py"), so
# the argument the parent process sends to the child process will be the 2nd element
a = sys.argv[1]
main(a)
Now child.py can be started with an argument like python child.py foobar and the text "foobar" will be used as the value for the a variable.
With that out of the way, all that's left is to rewrite parent.py and make it pass an argument to child.py. The recommended way to pass arguments with subprocess.Popen is as a list of strings, so we'll do just that:
import time
import subprocess
def main():
while(True):
a = input("Please enter parameter to pass to subprocess:")
subprocess.Popen(["python", "child.py", a]) # pass a as an argument
print(f"{a} was started")
time.sleep(5)
if __name__ == '__main__':
main()
This question already has answers here:
Exit codes in Python
(14 answers)
Closed 4 years ago.
I'd like to return a value from a python subprocess call corresponding to the kind of error returned. E.g.:
test.py
RETCODE_OK = 0
RETCODE_THING_1_FAILED = 1
RETCODE_THING_2_FAILED = 2
def main():
return RETCODE_THING_2_FAILED
if __name__ == '__main__':
main()
Then I'm calling it with subprocess, like so:
>>> import subprocess
>>> proc = subprocess.Popen('python test.py', shell=True)
>>> proc.communicate()
(None, None)
>>> proc.returncode
0
I'd like this to return whatever was returned in main(), in this case 2. I've also got other stuff in the stdout and stderr streams, so I can't just print RETCODE_THING_2_FAILED and get the stdout.
Processes uses exit codes, not return statements.
You should use sys.exit(STATUS) rather than return STATUS statement:
test2.py:
---------
import sys
def main():
sys.exit(RETCODE_THING_2_FAILED)
if __name__ == '__main__':
main()
interpreter:
------------
>>> p = subprocess.Popen('python test2.py')
>>> p.communicate()
(None, None)
>>> p.returncode
2
This is because the process is actually closing/exiting, not returning a value to another function.
Summary: I'd like to write python scripts that act like bash scripts on the command line, but then I'd also like to pipe them together easily in python. Where I'm having trouble is the glue to make the latter happen.
So imagine I wrote two scripts, script1.py and script2.py and I can pipe them together like so:
echo input_string | ./script1.py -a -b | ./script2.py -c -d
How do I get this behavior from within another python file?
Here's the way I know, but I don't like:
arg_string_1 = convert_to_args(param_1, param_2)
arg_string_2 = convert_to_args(param_3, param_4)
output_string = subprocess.check_output("echo " + input_string + " | ./script1.py " + arg_string_1 + " | ./script2.py " + arg_string_2)
If I didn't want to take advantage of multithreading, I could do something like this (?):
input1 = StringIO(input_string)
output1 = StringIO()
script1.main(param_1, param_2, input1, output1)
input2 = StringIO(output1.get_value())
output2 = StringIO()
script2.main(param_3, param_4, input2, output2)
Here's the approach I was trying, but I got stuck at writing the glue. I'd appreciate either learning how to finish my approach below, or suggestions for a better design/approach!
My approach: I wrote script1.py and script2.py to look like:
#!/usr/bin/python3
... # import sys and define "parse_args"
def main(param_1, param_2, input, output):
for line in input:
...
print(stuff, file=output)
if __name__ == "__main__":
parameter_1, parameter_2 = parse_args(sys.argv)
main(parameter_1, parameter_2, sys.stdin, sys.stdout)
Then I wanted to write something like this, but don't know how to finish:
pipe_out, pipe_in = ????
output = StringIO()
thread_1 = Thread(target=script1.main, args=(param_1, param_2, StreamIO(input_string), pipe_out))
thread_2 = Thread(target=script2.main, args=(param_3, param_4, pipe_in, output)
thread_1.start()
thread_2.start()
thread_1.join()
thread_2.join()
output_str = output.get_value()
For the "pipe in", uses sys.stdin with the readlines() method. (Using method read() would read one character at a time.)
For passing information from one thread to another, you can use Queue. You must define one way to signal the end of data. In my example, since all data passed between threads are str, I simply use a None object to signal the end of data (since it cannot appear in the transmitted data).
One could also use more threads, or use different functions in threads.
I did not include the sys.argvin my example to keep it simple. Modifying it to get parameters (parameter1, ...) should be easy.
import sys
from threading import Thread
from Queue import Queue
import fileinput
def stdin_to_queue( output_queue ):
for inp_line in sys.stdin.readlines(): # input one line at at time
output_queue.put( inp_line, True, None ) # blocking, no timeout
output_queue.put( None, True, None ) # signal the end of data
def main1(input_queue, output_queue, arg1, arg2):
do_loop = True
while do_loop:
inp_data = input_queue.get(True)
if inp_data is None:
do_loop = False
output_queue.put( None, True, None ) # signal end of data
else:
out_data = arg1 + inp_data.strip('\r\n').upper() + arg2 # or whatever transformation...
output_queue.put( out_data, True, None )
def queue_to_stdout(input_queue):
do_loop = True
while do_loop:
inp_data = input_queue.get(True)
if inp_data is None:
do_loop = False
else:
sys.stdout.write( inp_data )
def main():
q12 = Queue()
q23 = Queue()
q34 = Queue()
t1 = Thread(target=stdin_to_queue, args=(q12,) )
t2 = Thread(target=main1, args=(q12,q23,'(',')') )
t3 = Thread(target=main1, args=(q23,q34,'[',']') )
t4 = Thread(target=queue_to_stdout, args=(q34,))
t1.start()
t2.start()
t3.start()
t4.start()
main()
Finally, I tested this program (python2) with a text file.
head sometextfile.txt | python script.py
Redirect the return value to stdout depending on whether the script is being run from the command line:
#!/usr/bin/python3
import sys
# Example function
def main(input):
# Do something with input producing stuff
...
return multipipe(stuff)
if __name__ == '__main__':
def multipipe(data):
print(data)
input = parse_args(sys.argv)
main(input)
else:
def multipipe(data):
return data
Each other script will have the same two definitions of multipipe. Now, use multipipe for output.
If you call all the scripts together from the command line $ ./scrip1.py | ./scrip2.py, each will have __name__ == '__main__' and so multipipe will print it all to stdout to be read as an argument by the next script (and return None, so each function returns None, but you're not looking at the return values anyway in this case).
If you call them within some other python script, each function will return whatever you passed to multipipe.
Effectively, you can use your existing functions, just replace print(stuff, file=output) with return multipipe(stuff). Nice and simple.
To use it with multithreading or multiprocessing, set the functions up so that each function returns a single thing, and plug them into a simple function that adds data to a multithreading queue. For an example of such a queueing system, see the sample at the bottom of the Queue docs. With that example, just make sure that each step in the pipeline puts None (or other sentinel value of your choice - I like ... for that since it's extremely rare that you'd pass the Ellipsis object for any reason other than as a marker for its singleton-ness) in the queue to the next one to signify done-ness.
There is a very simple solution using the standard Popen class.
Here's an example:
#this is the master python program
import subprocess
import sys
import os
#note the use of stdin and stdout arguments here
process1 = subprocess.Popen(['./script1.py'], stdin=sys.stdin, stdout=subprocess.PIPE)
process2 = subprocess.Popen(['./script2.py'], stdin=process1.stdout)
process1.wait()
process2.wait()
the two scripts are:
#!/usr/bin/env python
#script1.py
import sys
for line in sys.stdin:
print(line.strip().upper())
Here's the second one
#!/usr/bin/env python
#script2.py
import sys
for line in sys.stdin:
print("<{}>".format(line.strip()))
I have a python code that reads output from command line:
import subprocess
def get_prg_output():
p = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
return out
print get_prg_output()
In my office I want to simulate result in this mode:
def get_prg_output():
return 'ok - program executed'
print get_prg_output()
Is there an elegant way to do this without comment out the original function?
I've try this:
import subprocess
debug = True
if not debug:
def get_prg_output():
p = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
return out
else:
def get_prg_output():
return 'ok - program executed'
print get_prg_output()
but I don't like it.
Thanks
I'd do something like this:
def _get_prg_output_real():
p = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
return out
def _get_prg_output_fake():
return 'ok - program executed'
Here you can toggle between:
get_prg_output = _get_prg_output_fake
or
get_prg_output = _get_prg_output_real
based on user input, or commandline arguments or config files or commenting/uncommenting a single line of code ...
def get_prg_output_fake():
return 'ok - program executed'
then, if you want to enable this simulation, just do
get_prg_output=get_prg_output_fake
I'd use an environment variable (or a command switch) to control it myself so you don't have to make code changes to test.
import os
import subprocess
def _get_prg_output_real():
p = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
return out
def _get_prg_output_fake():
return 'ok - program executed'
if os.environ.get('DEBUG').lower() == 'true':
get_prg_output = _get_prg_output_fake
else:
get_prg_output = get_prg_output_real
This is how my code looks and I get an error, while using Popen
test.py
import subprocess
import sys
def test(jobname):
print jobname
p=subprocess.Popen([sys.executable,jobname,parm1='test',parm2='test1'])
if __name__ == "__main__":
test(r'C:\Python27\test1.py')
test1.py
def test1(parm1,parm2):
print 'test1',parm1
if __name__ = '__main__':
test1(parm1='',parm2='')
Error
Syntax error
In test1.py:
You need two equal signs in :
if __name__ = '__main__':
Use instead
if __name__ == '__main__':
since you want to compare the value of __name__ with the string '__main__', not assign a value to __name__.
In test.py:
parm1='test' is a SyntaxError. You can not to assign a value to a variable in the middle of a list:
p=subprocess.Popen([sys.executable,jobname,parm1='test',parm2='test1'])
It appears you want to feed different values for parm1 and parm2 into the function test1.test1. You can not do that by calling python test1.py since there parm1='' and parm2='' are hard-coded there.
When you want to run a non-Python script from Python, use subprocess. But when you want to run Python functions in a subprocess, use multiprocessing:
import multiprocessing as mp
import test1
def test(function, *args, **kwargs):
print(function.__name__)
proc = mp.Process(target = function, args = args, kwargs = kwargs)
proc.start()
proc.join() # wait for proc to end
if __name__ == '__main__':
test(test1.test1, parm1 = 'test', parm2 = 'test1')