This question already has answers here:
Running shell command and capturing the output
(21 answers)
Closed 2 years ago.
I want to get the stdout in a variable after running the os.system call.
Lets take this line as an example:
batcmd="dir"
result = os.system(batcmd)
result will contain the error code (stderr 0 under Windows or 1 under some linux for the above example).
How can I get the stdout for the above command without using redirection in the executed command?
If all you need is the stdout output, then take a look at subprocess.check_output():
import subprocess
batcmd="dir"
result = subprocess.check_output(batcmd, shell=True)
Because you were using os.system(), you'd have to set shell=True to get the same behaviour. You do want to heed the security concerns about passing untrusted arguments to your shell.
If you need to capture stderr as well, simply add stderr=subprocess.STDOUT to the call:
result = subprocess.check_output([batcmd], stderr=subprocess.STDOUT)
to redirect the error output to the default output stream.
If you know that the output is text, add text=True to decode the returned bytes value with the platform default encoding; use encoding="..." instead if that codec is not correct for the data you receive.
These answers didn't work for me. I had to use the following:
import subprocess
p = subprocess.Popen(["pwd"], stdout=subprocess.PIPE)
out = p.stdout.read()
print out
Or as a function (using shell=True was required for me on Python 2.6.7 and check_output was not added until 2.7, making it unusable here):
def system_call(command):
p = subprocess.Popen([command], stdout=subprocess.PIPE, shell=True)
return p.stdout.read()
import subprocess
string="echo Hello world"
result=subprocess.getoutput(string)
print("result::: ",result)
I had to use os.system, since subprocess was giving me a memory error for larger tasks. Reference for this problem here. So, in order to get the output of the os.system command I used this workaround:
import os
batcmd = 'dir'
result_code = os.system(batcmd + ' > output.txt')
if os.path.exists('output.txt'):
fp = open('output.txt', "r")
output = fp.read()
fp.close()
os.remove('output.txt')
print(output)
I would like to expand on the Windows solution. Using IDLE with Python 2.7.5, When I run this code from file Expts.py:
import subprocess
r = subprocess.check_output('cmd.exe dir',shell=False)
print r
...in the Python Shell, I ONLY get the output corresponding to "cmd.exe"; the "dir" part is ignored. HOWEVER, when I add a switch such as /K or /C ...
import subprocess
r = subprocess.check_output('cmd.exe /K dir',shell=False)
print r
...then in the Python Shell, I get all that I expect including the directory listing. Woohoo !
Now, if I try any of those same things in DOS Python command window, without the switch, or with the /K switch, it appears to make the window hang because it is running a subprocess cmd.exe and it awaiting further input - type 'exit' then hit [enter] to release. But with the /K switch it works perfectly and returns you to the python prompt. Allrightee then.
Went a step further...I thought this was cool...When I instead do this in Expts.py:
import subprocess
r = subprocess.call("cmd.exe dir",shell=False)
print r
...a new DOS window pops open and remains there displaying only the results of "cmd.exe" not of "dir". When I add the /C switch, the DOS window opens and closes very fast before I can see anything (as expected, because /C terminates when done). When I instead add the /K switch, the DOS window pops open and remain, AND I get all the output I expect including the directory listing.
If I try the same thing (subprocess.call instead of subprocess.check_output) from a DOS Python command window; all output is within the same window, there are no popup windows. Without the switch, again the "dir" part is ignored, AND the prompt changes from the python prompt to the DOS prompt (since a cmd.exe subprocess is running in python; again type 'exit' and you will revert to the python prompt). Adding the /K switch prints out the directory listing and changes the prompt from python to DOS since /K does not terminate the subprocess. Changing the switch to /C gives us all the output expected AND returns to the python prompt since the subprocess terminates in accordance with /C.
Sorry for the long-winded response, but I am frustrated on this board with the many terse 'answers' which at best don't work (seems because they are not tested - like Eduard F's response above mine which is missing the switch) or worse, are so terse that they don't help much at all (e.g., 'try subprocess instead of os.system' ... yeah, OK, now what ??). In contrast, I have provided solutions which I tested, and showed how there are subtle differences between them. Took a lot of time but...
Hope this helps.
commands also works.
import commands
batcmd = "dir"
result = commands.getoutput(batcmd)
print result
It works on linux, python 2.7.
Related
Thank you in advance for the time you'll give to read this question. I am learning Python and I looked up a lot before asking here, please forgive me for the newbie question.
So I created this script in python 3 using subprocess module to search for another python script's PID, while only knowing the beginning of the script's name and terminate it nicely.
Basically I run python clocks on my LCD screen through Raspberry and I2C, and I terminate the script, clear the LCD and turn it off. This "off" script code is provided below.
The issue is that when I run it from the directory it sits in with a:
python3 off.py
It works perfectly, getting parsing and terminating the PID, then turning off the LCD display.
Ideally I want to trigger it through telegram-cli because I did it in bash and it worked nicely, I find it to be a nice feature. In python it fails.
So I tested and it appears that when I try to launch it from another directory like this:
python3 ~/code/off.py
The grep subprocess returns more than the one PID it returns normally when launched from the script residing directory. For instance (with python3 -v):
kill: failed to parse argument: '25977
26044'
The second PID number is from a sub process created by the script, I can't seem to find what it is as it terminates when the script ends but fails it initial purpose.
Any help in understanding what is happening here would be really appreciated.
I came so far, as show below, from two ugly lines of bash mixed with a call to an dummy four lines python scripts, so I really feel I am getting close to a proper way of achieving my first real python script.
I tried to decompose the script line by line in the interpreter and could not reproduce the error, everything behave as expected. I only get this double PID result when running the script from an outer location.
Thank you in advance for any helpful insight on how to understand what is happening!
#!/usr/bin/env python3
import subprocess
import I2C_LCD_driver
import string
# Defining variables for searched strings and string encoding
searched_process_name = 'lcd_'
cut_grep_out_of_results = 'grep'
result_string_encoding = 'utf-8'
mylcd = I2C_LCD_driver.lcd()
LCD_NOBACKLIGHT = 0x00
run = True
def kill_script():
# Listing processes and getting the searched process
ps_process = subprocess.Popen(["ps", "aux"], stdout=subprocess.PIPE)
grep_process = subprocess.Popen(["grep", "-i", searched_process_name], stdin=ps_process.stdout, stdout=subprocess.PIPE)
# The .stdout.close() lines below allow the previous process to receive a SIGPIPE if the next process exits.
ps_process.stdout.close()
# Cleaning the result until only the PID number is returned in a string
grep_cutout = subprocess.Popen(["grep", "-v", cut_grep_out_of_results], stdin=grep_process.stdout, stdout=subprocess.PIPE)
grep_process.stdout.close()
awk = subprocess.Popen(["cut", "-c", "10-14"], stdin=grep_cutout.stdout, stdout=subprocess.PIPE)
grep_cutout.stdout.close()
output = awk.communicate()[0]
clean_output = output.decode(result_string_encoding)
clean_output_no_new_line = clean_output.rstrip()
clean_output_no_quote = clean_output_no_new_line.replace("'", '')
PID = clean_output_no_quote
# Terminating the LCD script process
subprocess.Popen(["kill", "-9", PID])
while run:
kill_script()
# Cleaning and shutting off LCD screen
mylcd.lcd_clear()
mylcd.lcd_device.write_cmd(LCD_NOBACKLIGHT)
break
I found out the reason of this weird comportment. An error on my end:
I forgot I called some directories with a name including the characters string I was running grep -i against provoking the double result when running the script from outside its directory using its full path.
Turns out the script runs pretty well using subprocess.
So in the end, I renamed the scripts I wanted to terminate with disp_ rather than lcd_ and added shell=False to my subprocesses to make sure there were no risk of unwantedly sending the output to bash while the running the script.
I am running spyder on windows 10 and when I attempt to run a command similar to the following:
cmd = 'python /path/to/program.py arg1 arg2'
subprocess.run(cmd,shell=True)
The script is being run as expected but I would like to see what is being printed to screen by the executed command in the spyder ipython console. I know the program is printing things to screen as expected by other methods (running the program from a shell) so there is not an error in the script I am running.
How do I go about enabling printing for the subprocess?
The output comes in a stream called stdout. To capture it, you need to redirect it to a pipe, which then is terminated in the calling process. subprocess.run(...) has builtin support for handling this:
import subprocess
cmd = 'python /path/to/program.py arg1 arg2'.split()
proc = subprocess.run(cmd, stdout=subprocess.PIPE, universal_newlines=True)
print(proc.stdout)
As can be seen, the output is caught in the CompletedProcess object (proc) and then accessed as member data.Also, to make the output into text (a string) rather than a bytearray, I have passed the parameter universal_newlines=True.
A caveat, though, is that subprocess.run(...) runs to completion before it returns control. Therefore, this does not allow for capturing the output "live" but rather after the whole process has finsihed. If you want live capture, you must instead use subprocess.Popen(...) and then use .communicate() or some other means of communication to catch the output from the subprocess.
Another comment I like to make, is that using shell=True is not recommended. Specifically not when handling unknown or not trusted input. It leaves the interpretation of cmd to the shell which can lead to all kind of security breaches and bad behavior. Instead, split cmd into a list (e.g. as I have done) and then pass that list to subprocess.run(...) and leave out shell=True.
I'm struggling to get some python script to start a subprocess, wait until it completes and then retrieve the required data. I'm quite new to Python.
The command I wish to run as a subprocess is
./bin.testing/Eva -t --suite="temp0"
Running that command by hand in the Linux terminal produces:
in terminal mode
Evaluation error = 16.7934
I want to run the command as a python sub-process, and receive the output back. However, everything I try seems to skip the second line (ultimately, it's the second line that I want.) At the moment, I have this:
def job(self,fen_file):
from subprocess import Popen, PIPE
from sys import exit
try:
eva=Popen('{0}/Eva -t --suite"{0}"'.format(self.exedir,fen_file),shell=True,stdout=PIPE,stderr=PIPE)
stdout,stderr=eva.communicate()
except:
print ('Error running test suite '+fen_file)
exit("Stopping")
print(stdout)
.
.
.
return 0
All this seems to produce is
in terminal mode
0
with the important line missing. The print statement is just so I can see what I am getting back from the sub-process -- the intention is that it will be replaced with code that processes the number from the second line and returns the output (here I'm just returning 0 just so I can get this particular bit to work first. The caller of this function prints the result, which is why there is a zero at the end of the output.) exedir is just the directory of the executable for the sub-process, and fen-file is just an ascii file that the sub-process needs. I have tried removing the 'in terminal mode' from the source code of the sub-process and re compiling it, but that doesn't work -- it still doesn't return the important second line.
Thanks in advance; I expect what I am doing wrong is really very simple.
Edit: I ought to add that the subprocess Eva can take a second or two to complete.
Since the 2nd line is an error message, it's probably stored in your stderr variable!
To know for sure you can print your stderr in your code, or you can run the program on the command line and see if the output is split into stdout and stderr. One easy way is to do ./bin.testing/Eva -t --suite="temp0" > /dev/null. Any messages you get are stderr since stdout is redirected to /dev/null.
Also, typically with Popen the shell=True option is discouraged unless really needed. Instead pass a list:
[os.path.join(self.exedir, 'Eva'), '-t', '--suite=' + fen_file], shell=False, ...
This can avoid problems down the line if one of your arguments would normally be interpreted by the shell. (Note, I removed the ""'s, because the shell would normally eat those for you!)
Try using subprocess check_output.
output_lines = subprocess.check_output(['./bin.testing/Eva', '-t', '--suite="temp0"'])
for line in output_lines.splitlines():
print(line)
I am trying to run my tcl script through python subprocess as follow:
import subprocess
>>> subprocess.Popen(["tclsh", "tcltest.tcl"])
<subprocess.Popen object at 0x0000000001DD4DD8>
>>> subprocess.Popen(["tclsh", "tcltest.tcl"], shell=True )
<subprocess.Popen object at 0x0000000002B34550>
I don't know if it is working or not, since I don't see any anything from it!
my tcl script also has some packages from my company that causes errors when I use Tkinter, Tk, and eval,
import Tkinter
import socket
def TCLRun():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 5006))
root = Tkinter.Tk()
## root.tk.eval('package require Azimuth-Sdk')
tcl_script ="""
##package require Company-Sdk
## set players [ace_azplayer_list_players]
set players 2
puts $players
## if { $players != "" } {
## foreach player $players {
## set cmd ace_azplayer_remove_player
## if { [catch { [ $cmd $player ] } err] } {
## puts " $cmd $player - $err"
## sleep 1
## }
## }
## } """
# call the Tkinter tcl interpreter
root.tk.call('eval', tcl_script)
root.mainloop()
gives me this error
import TCLCall
>>> TCLCall.TCLRun()
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
TCLCall.TCLRun()
File "C:\Users\XXX\Desktop\PKT\TCLCall.py", line 24, in TCLRun
root.tk.call('eval', tcl_script)
TclError: can not find channel named "stdout"
that's why I switched to subprocess. at least it doesn't give me error!
any idea how to run my tcl script with internal required package through python?!
Thanks
To get the output from using subprocess.Popen, you can try the following:
import subprocess
p = subprocess.Popen(
"tclsh tcltest.tcl",
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
print stdout
print stderr
It's entirely possible that the script you're running with subprocess.Popen is also generating an error, but isn't displaying since you aren't explicitly looking for it.
Edit:
To prevent some information from being lost in the comments below:
You probably have several potential errors here, or things you can try.
Either your tcl script by itself isn't able to import teapot properly, or some sort of interaction between the tcl script and the python script isn't working properly, or subprocess.Popen isn't correctly finding the teapot package from your path.
I would try debugging your programs in that order. First confirm that your tcl script works without using python or subprocess.Popen and just directly run it from the command line (for example, C:\Users\blah tclsh tcltest.tcl)
Then, after you've made sure your script work, bring in Python. From what I'm seeing, you don't have any problem with the python stuff, but with either your tcl script or some issue with your path.
The whole point of subprocess.Popen is redirection of standard channels, so you can handle output programmatically instead of seeing it on your own standard output. Did you try handling it? How?
Maybe you don't need redirection at all: then os.system("tclsh tcltest.tcl") should be enough. Or maybe subprocess.Popen has other advantages for you -- then figure out how to disable redirection, or how to redirect child's stdout to your own stdout.
I think I might have a solution for you. Or at least a different method to try. This example opens the TCL shell as a process. Then you pump commands to it as though you are on command line. Since you said your command line worked, I would think that this would too. This works for me on Windows with Python 3.7.6 and TCL 8.5.
There needs to be a little trickery. I've found solutions that require threads and all sorts of other overhead to get this job done but they just fell flat. What I came up with is simple and synchronous.
stdout.readline() will block. So if your TCL command doesn't throw anything back, you are dead in the water.
SO, you force something to come back by appending an innocuous TCL puts command that communicates back to the Python script that the job is done.
The other trick is that you need to "puts []" the command to force the output back into stdout.
If you need to source more TCL files, then add those in before you make your final call to what you are trying to run. Whatever you need to do on the cli, you do through this process.
My code has this all wrapped in a class with methods. I brought them all together here in-line for the example. Leave the errorInfo code in while debugging and remove as you see fit.
Note: You can also use TCL "info body " to read a script into a string variable and then run each line through the process. In essence, if in a Python debugging session, stepping through TCL. Sorry, this does not work too well. The workaround for comments does not work for opening curly braces.
Hope this helps someone out there.
EDIT: Using multi-line string handles comment lines.
import subprocess
print("------")
shell_path = r"<YOUR PATH TO THE TCL INTERPRETER SHELL>"
tcl_shell = subprocess.Popen(shell_path,
stdin =subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines = True,
bufsize = 0)
#
# Use ONE of the "command"s below. I have them here in series for exmaples.
#
# Simple
command = r"set temp 5"
# Multiline with error --> This will give an error of course to see the error extraction in action
command = r"""
set temp 5
set temp2 [expr temp + 5]
"""
# Multiline working
command = r"""
set temp 5
set temp2 [expr $temp + 5]
"""
# More output
command = r"""
puts "Starting process"
set temp 5
puts $temp
set temp2 [expr $temp + 5]
"""
# Comments handled
command = r"# comment!"
# Be sure to leave the newline to handle comments
tcl_shell.stdin.write(f"""puts [{command}
] \nputs \"tcl_shell_cmd_complete\"\n""")
# NOTE: tcl_shell.stdin.flush() does not seem to be needed. Consider if needed.
result = ""
line = tcl_shell.stdout.readline()
while line != "tcl_shell_cmd_complete\n":
result += line
line = tcl_shell.stdout.readline()
print(f"tcl_shell sent:\n{command}")
print(f"tcl_shell result:\n{result}".rstrip())
command_error_check = "puts $errorInfo"
tcl_shell.stdin.write(f"{command_error_check} \nputs \"tcl_shell_cmd_complete\"\n")
resultErr = ""
line = tcl_shell.stdout.readline()
while line != "tcl_shell_cmd_complete\n":
resultErr += line
line = tcl_shell.stdout.readline()
print(f"tcl_shell error info:\n{resultErr}")
tcl_shell.stdin.close()
tcl_shell.terminate()
tcl_shell.wait(timeout=0.5)
print("------")
I am running a sub-program using subprocess.popen. When I start my Python program from the command window (cmd.exe), the program writes some info and dates in the window as the program evolves.
When I run my Python code not in a command window, it opens a new command window for this sub-program's output, and I want to avoid that. When I used the following code, it doesn't show the cmd window, but it also doesn't print the status:
p = subprocess.Popen("c:/flow/flow.exe", shell=True, stdout=subprocess.PIPE)
print p.stdout.read()
How can I show the sub-program's output in my program's output as it occurs?
Use this:
cmd = subprocess.Popen(["c:/flow/flow.exe"], stdout=subprocess.PIPE)
for line in cmd.stdout:
print line.rstrip("\n")
cmd.wait() # you may already be handling this in your current code
Note that you will still have to wait for the sub-program to flush its stdout buffer (which is commonly buffered differently when not writing to a terminal window), so you may not see each line instantaneously as the sub-program prints it (this depends on various OS details and details of the sub-program).
Also notice how I've removed the shell=True and replaced the string argument with a list, which is generally recommended.
Looking for a recipe to process Popen data asynchronously I stumbled upon http://code.activestate.com/recipes/576759-subprocess-with-async-io-pipes-class/
This looks quite promising, however I got the impression that there might be some typos in it. Not tried it yet.
It is an old post, but a common problem with a hard to find solution. Try this: http://code.activestate.com/recipes/440554-module-to-allow-asynchronous-subprocess-use-on-win/