I am trying to run my tcl script through python subprocess as follow:
import subprocess
>>> subprocess.Popen(["tclsh", "tcltest.tcl"])
<subprocess.Popen object at 0x0000000001DD4DD8>
>>> subprocess.Popen(["tclsh", "tcltest.tcl"], shell=True )
<subprocess.Popen object at 0x0000000002B34550>
I don't know if it is working or not, since I don't see any anything from it!
my tcl script also has some packages from my company that causes errors when I use Tkinter, Tk, and eval,
import Tkinter
import socket
def TCLRun():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 5006))
root = Tkinter.Tk()
## root.tk.eval('package require Azimuth-Sdk')
tcl_script ="""
##package require Company-Sdk
## set players [ace_azplayer_list_players]
set players 2
puts $players
## if { $players != "" } {
## foreach player $players {
## set cmd ace_azplayer_remove_player
## if { [catch { [ $cmd $player ] } err] } {
## puts " $cmd $player - $err"
## sleep 1
## }
## }
## } """
# call the Tkinter tcl interpreter
root.tk.call('eval', tcl_script)
root.mainloop()
gives me this error
import TCLCall
>>> TCLCall.TCLRun()
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
TCLCall.TCLRun()
File "C:\Users\XXX\Desktop\PKT\TCLCall.py", line 24, in TCLRun
root.tk.call('eval', tcl_script)
TclError: can not find channel named "stdout"
that's why I switched to subprocess. at least it doesn't give me error!
any idea how to run my tcl script with internal required package through python?!
Thanks
To get the output from using subprocess.Popen, you can try the following:
import subprocess
p = subprocess.Popen(
"tclsh tcltest.tcl",
shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
print stdout
print stderr
It's entirely possible that the script you're running with subprocess.Popen is also generating an error, but isn't displaying since you aren't explicitly looking for it.
Edit:
To prevent some information from being lost in the comments below:
You probably have several potential errors here, or things you can try.
Either your tcl script by itself isn't able to import teapot properly, or some sort of interaction between the tcl script and the python script isn't working properly, or subprocess.Popen isn't correctly finding the teapot package from your path.
I would try debugging your programs in that order. First confirm that your tcl script works without using python or subprocess.Popen and just directly run it from the command line (for example, C:\Users\blah tclsh tcltest.tcl)
Then, after you've made sure your script work, bring in Python. From what I'm seeing, you don't have any problem with the python stuff, but with either your tcl script or some issue with your path.
The whole point of subprocess.Popen is redirection of standard channels, so you can handle output programmatically instead of seeing it on your own standard output. Did you try handling it? How?
Maybe you don't need redirection at all: then os.system("tclsh tcltest.tcl") should be enough. Or maybe subprocess.Popen has other advantages for you -- then figure out how to disable redirection, or how to redirect child's stdout to your own stdout.
I think I might have a solution for you. Or at least a different method to try. This example opens the TCL shell as a process. Then you pump commands to it as though you are on command line. Since you said your command line worked, I would think that this would too. This works for me on Windows with Python 3.7.6 and TCL 8.5.
There needs to be a little trickery. I've found solutions that require threads and all sorts of other overhead to get this job done but they just fell flat. What I came up with is simple and synchronous.
stdout.readline() will block. So if your TCL command doesn't throw anything back, you are dead in the water.
SO, you force something to come back by appending an innocuous TCL puts command that communicates back to the Python script that the job is done.
The other trick is that you need to "puts []" the command to force the output back into stdout.
If you need to source more TCL files, then add those in before you make your final call to what you are trying to run. Whatever you need to do on the cli, you do through this process.
My code has this all wrapped in a class with methods. I brought them all together here in-line for the example. Leave the errorInfo code in while debugging and remove as you see fit.
Note: You can also use TCL "info body " to read a script into a string variable and then run each line through the process. In essence, if in a Python debugging session, stepping through TCL. Sorry, this does not work too well. The workaround for comments does not work for opening curly braces.
Hope this helps someone out there.
EDIT: Using multi-line string handles comment lines.
import subprocess
print("------")
shell_path = r"<YOUR PATH TO THE TCL INTERPRETER SHELL>"
tcl_shell = subprocess.Popen(shell_path,
stdin =subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines = True,
bufsize = 0)
#
# Use ONE of the "command"s below. I have them here in series for exmaples.
#
# Simple
command = r"set temp 5"
# Multiline with error --> This will give an error of course to see the error extraction in action
command = r"""
set temp 5
set temp2 [expr temp + 5]
"""
# Multiline working
command = r"""
set temp 5
set temp2 [expr $temp + 5]
"""
# More output
command = r"""
puts "Starting process"
set temp 5
puts $temp
set temp2 [expr $temp + 5]
"""
# Comments handled
command = r"# comment!"
# Be sure to leave the newline to handle comments
tcl_shell.stdin.write(f"""puts [{command}
] \nputs \"tcl_shell_cmd_complete\"\n""")
# NOTE: tcl_shell.stdin.flush() does not seem to be needed. Consider if needed.
result = ""
line = tcl_shell.stdout.readline()
while line != "tcl_shell_cmd_complete\n":
result += line
line = tcl_shell.stdout.readline()
print(f"tcl_shell sent:\n{command}")
print(f"tcl_shell result:\n{result}".rstrip())
command_error_check = "puts $errorInfo"
tcl_shell.stdin.write(f"{command_error_check} \nputs \"tcl_shell_cmd_complete\"\n")
resultErr = ""
line = tcl_shell.stdout.readline()
while line != "tcl_shell_cmd_complete\n":
resultErr += line
line = tcl_shell.stdout.readline()
print(f"tcl_shell error info:\n{resultErr}")
tcl_shell.stdin.close()
tcl_shell.terminate()
tcl_shell.wait(timeout=0.5)
print("------")
Related
This question already has answers here:
Running shell command and capturing the output
(21 answers)
Closed 2 years ago.
I want to get the stdout in a variable after running the os.system call.
Lets take this line as an example:
batcmd="dir"
result = os.system(batcmd)
result will contain the error code (stderr 0 under Windows or 1 under some linux for the above example).
How can I get the stdout for the above command without using redirection in the executed command?
If all you need is the stdout output, then take a look at subprocess.check_output():
import subprocess
batcmd="dir"
result = subprocess.check_output(batcmd, shell=True)
Because you were using os.system(), you'd have to set shell=True to get the same behaviour. You do want to heed the security concerns about passing untrusted arguments to your shell.
If you need to capture stderr as well, simply add stderr=subprocess.STDOUT to the call:
result = subprocess.check_output([batcmd], stderr=subprocess.STDOUT)
to redirect the error output to the default output stream.
If you know that the output is text, add text=True to decode the returned bytes value with the platform default encoding; use encoding="..." instead if that codec is not correct for the data you receive.
These answers didn't work for me. I had to use the following:
import subprocess
p = subprocess.Popen(["pwd"], stdout=subprocess.PIPE)
out = p.stdout.read()
print out
Or as a function (using shell=True was required for me on Python 2.6.7 and check_output was not added until 2.7, making it unusable here):
def system_call(command):
p = subprocess.Popen([command], stdout=subprocess.PIPE, shell=True)
return p.stdout.read()
import subprocess
string="echo Hello world"
result=subprocess.getoutput(string)
print("result::: ",result)
I had to use os.system, since subprocess was giving me a memory error for larger tasks. Reference for this problem here. So, in order to get the output of the os.system command I used this workaround:
import os
batcmd = 'dir'
result_code = os.system(batcmd + ' > output.txt')
if os.path.exists('output.txt'):
fp = open('output.txt', "r")
output = fp.read()
fp.close()
os.remove('output.txt')
print(output)
I would like to expand on the Windows solution. Using IDLE with Python 2.7.5, When I run this code from file Expts.py:
import subprocess
r = subprocess.check_output('cmd.exe dir',shell=False)
print r
...in the Python Shell, I ONLY get the output corresponding to "cmd.exe"; the "dir" part is ignored. HOWEVER, when I add a switch such as /K or /C ...
import subprocess
r = subprocess.check_output('cmd.exe /K dir',shell=False)
print r
...then in the Python Shell, I get all that I expect including the directory listing. Woohoo !
Now, if I try any of those same things in DOS Python command window, without the switch, or with the /K switch, it appears to make the window hang because it is running a subprocess cmd.exe and it awaiting further input - type 'exit' then hit [enter] to release. But with the /K switch it works perfectly and returns you to the python prompt. Allrightee then.
Went a step further...I thought this was cool...When I instead do this in Expts.py:
import subprocess
r = subprocess.call("cmd.exe dir",shell=False)
print r
...a new DOS window pops open and remains there displaying only the results of "cmd.exe" not of "dir". When I add the /C switch, the DOS window opens and closes very fast before I can see anything (as expected, because /C terminates when done). When I instead add the /K switch, the DOS window pops open and remain, AND I get all the output I expect including the directory listing.
If I try the same thing (subprocess.call instead of subprocess.check_output) from a DOS Python command window; all output is within the same window, there are no popup windows. Without the switch, again the "dir" part is ignored, AND the prompt changes from the python prompt to the DOS prompt (since a cmd.exe subprocess is running in python; again type 'exit' and you will revert to the python prompt). Adding the /K switch prints out the directory listing and changes the prompt from python to DOS since /K does not terminate the subprocess. Changing the switch to /C gives us all the output expected AND returns to the python prompt since the subprocess terminates in accordance with /C.
Sorry for the long-winded response, but I am frustrated on this board with the many terse 'answers' which at best don't work (seems because they are not tested - like Eduard F's response above mine which is missing the switch) or worse, are so terse that they don't help much at all (e.g., 'try subprocess instead of os.system' ... yeah, OK, now what ??). In contrast, I have provided solutions which I tested, and showed how there are subtle differences between them. Took a lot of time but...
Hope this helps.
commands also works.
import commands
batcmd = "dir"
result = commands.getoutput(batcmd)
print result
It works on linux, python 2.7.
Thank you in advance for the time you'll give to read this question. I am learning Python and I looked up a lot before asking here, please forgive me for the newbie question.
So I created this script in python 3 using subprocess module to search for another python script's PID, while only knowing the beginning of the script's name and terminate it nicely.
Basically I run python clocks on my LCD screen through Raspberry and I2C, and I terminate the script, clear the LCD and turn it off. This "off" script code is provided below.
The issue is that when I run it from the directory it sits in with a:
python3 off.py
It works perfectly, getting parsing and terminating the PID, then turning off the LCD display.
Ideally I want to trigger it through telegram-cli because I did it in bash and it worked nicely, I find it to be a nice feature. In python it fails.
So I tested and it appears that when I try to launch it from another directory like this:
python3 ~/code/off.py
The grep subprocess returns more than the one PID it returns normally when launched from the script residing directory. For instance (with python3 -v):
kill: failed to parse argument: '25977
26044'
The second PID number is from a sub process created by the script, I can't seem to find what it is as it terminates when the script ends but fails it initial purpose.
Any help in understanding what is happening here would be really appreciated.
I came so far, as show below, from two ugly lines of bash mixed with a call to an dummy four lines python scripts, so I really feel I am getting close to a proper way of achieving my first real python script.
I tried to decompose the script line by line in the interpreter and could not reproduce the error, everything behave as expected. I only get this double PID result when running the script from an outer location.
Thank you in advance for any helpful insight on how to understand what is happening!
#!/usr/bin/env python3
import subprocess
import I2C_LCD_driver
import string
# Defining variables for searched strings and string encoding
searched_process_name = 'lcd_'
cut_grep_out_of_results = 'grep'
result_string_encoding = 'utf-8'
mylcd = I2C_LCD_driver.lcd()
LCD_NOBACKLIGHT = 0x00
run = True
def kill_script():
# Listing processes and getting the searched process
ps_process = subprocess.Popen(["ps", "aux"], stdout=subprocess.PIPE)
grep_process = subprocess.Popen(["grep", "-i", searched_process_name], stdin=ps_process.stdout, stdout=subprocess.PIPE)
# The .stdout.close() lines below allow the previous process to receive a SIGPIPE if the next process exits.
ps_process.stdout.close()
# Cleaning the result until only the PID number is returned in a string
grep_cutout = subprocess.Popen(["grep", "-v", cut_grep_out_of_results], stdin=grep_process.stdout, stdout=subprocess.PIPE)
grep_process.stdout.close()
awk = subprocess.Popen(["cut", "-c", "10-14"], stdin=grep_cutout.stdout, stdout=subprocess.PIPE)
grep_cutout.stdout.close()
output = awk.communicate()[0]
clean_output = output.decode(result_string_encoding)
clean_output_no_new_line = clean_output.rstrip()
clean_output_no_quote = clean_output_no_new_line.replace("'", '')
PID = clean_output_no_quote
# Terminating the LCD script process
subprocess.Popen(["kill", "-9", PID])
while run:
kill_script()
# Cleaning and shutting off LCD screen
mylcd.lcd_clear()
mylcd.lcd_device.write_cmd(LCD_NOBACKLIGHT)
break
I found out the reason of this weird comportment. An error on my end:
I forgot I called some directories with a name including the characters string I was running grep -i against provoking the double result when running the script from outside its directory using its full path.
Turns out the script runs pretty well using subprocess.
So in the end, I renamed the scripts I wanted to terminate with disp_ rather than lcd_ and added shell=False to my subprocesses to make sure there were no risk of unwantedly sending the output to bash while the running the script.
I am attempting to call a bash script via the subprocess Popen function passes in a for loop. My intent is that with each iteration, a new string commit from an array out is passed as an argument to the Popen command. The command invokes a bash script that outputs a text identified by the variable commit and greps certain lines from that particular text. However, I can't get the output to flush out in the Python for loop. Right now, only the grepped data from the final commit in out is being passed into my final data structure (a pandas dataframe).
accuracy_dictionary = {}
for commit in out:
accuracy_dictionary.setdefault(commit, {})
p2 = subprocess.Popen(['~/Desktop/find_accuracies.sh', commit], encoding='utf-8', shell=True, stdout=subprocess.PIPE)
outputstring = p2.stdout.read()
# This part below is less critical to the problem at hand
# I'm putting the data from each file in a dictionary
for acc_type_line in outputstring.split('\n'):
accuracy = acc_type_line.split(': ')
if accuracy != ['']:
acc_type = accuracy[0]
value = accuracy[1]
accuracy_dictionary[commit][acc_type] = float(value)
acc_data = pd.DataFrame.from_dict(accuracy_dictionary).T
Here is the bash script that is being called:
"find_accuracies.sh":
#!/bin/sh
COMMIT=$1
git show $COMMIT:blahblahfolder/blahblah.txt | grep --line-buffered 'accuracy'
acc_data returns a dataframe of nrows=len(out) populated by unique commits, but the value is the exact same for all rows for each acc_type
For example, my output looks like this:
How can I call the file "find_accuracies.sh" with the subprocess command and have it flush the unique values of each file for each commit?
I hope this help addressing the immediate problem you're seeing: Here you should really use communicate with subprocess.PIPE as it waits for the command to finish and give give you all of its output:
outputstring = p2.communicate()[0]
You can also use convenient method like check_output to the same effect:
outputstring = subprocess.check_output(['~/Desktop/find_accuracies.sh', commit],
encoding='utf-8', shell=True)
Or also in py3 use run should also do:
p2 = subprocess.run(['~/Desktop/find_accuracies.sh', commit],
encoding='utf-8', shell=True, stdout=subprocess.PIPE)
outputstring = p2.stdout
Now few more comments, hints and suggestions:
I am a little surprised it works for you as using shell=True and list of arguments should (see the paragraph starting with "On POSIX with shell=True") make your commit argument of the underlying sh wrapped around your script call and not of the script itself. In any case you can (and I would suggest to) actually drop the shell and leave HOME resolution to python:
from pathlib import Path
executable = Path.home().joinpath('Desktop/find_accuracies.sh')
p2 = subprocess.run([executable, commit],
encoding='utf-8', stdout=subprocess.PIPE)
outputstring = p2.stdout
You can (or must for py <3.5) also use os.path.expanduser('~/Desktop/find_accuracies.sh') instead of Path.home() to get script executable. On the other hand for >=3.7 you could replace stdout=subprocess.PIPE with capture_output=True.
And last but not least. It seems a bit unnecessary to call a bash script (esp. double wrapped in sh call like in the original example) just to run git through grep when we already have a python script to process the information. I would actually try to run the corresponding git command directly getting the bulk of its output and process its output in the python script itself to get the bits of interest.
I'm struggling to get some python script to start a subprocess, wait until it completes and then retrieve the required data. I'm quite new to Python.
The command I wish to run as a subprocess is
./bin.testing/Eva -t --suite="temp0"
Running that command by hand in the Linux terminal produces:
in terminal mode
Evaluation error = 16.7934
I want to run the command as a python sub-process, and receive the output back. However, everything I try seems to skip the second line (ultimately, it's the second line that I want.) At the moment, I have this:
def job(self,fen_file):
from subprocess import Popen, PIPE
from sys import exit
try:
eva=Popen('{0}/Eva -t --suite"{0}"'.format(self.exedir,fen_file),shell=True,stdout=PIPE,stderr=PIPE)
stdout,stderr=eva.communicate()
except:
print ('Error running test suite '+fen_file)
exit("Stopping")
print(stdout)
.
.
.
return 0
All this seems to produce is
in terminal mode
0
with the important line missing. The print statement is just so I can see what I am getting back from the sub-process -- the intention is that it will be replaced with code that processes the number from the second line and returns the output (here I'm just returning 0 just so I can get this particular bit to work first. The caller of this function prints the result, which is why there is a zero at the end of the output.) exedir is just the directory of the executable for the sub-process, and fen-file is just an ascii file that the sub-process needs. I have tried removing the 'in terminal mode' from the source code of the sub-process and re compiling it, but that doesn't work -- it still doesn't return the important second line.
Thanks in advance; I expect what I am doing wrong is really very simple.
Edit: I ought to add that the subprocess Eva can take a second or two to complete.
Since the 2nd line is an error message, it's probably stored in your stderr variable!
To know for sure you can print your stderr in your code, or you can run the program on the command line and see if the output is split into stdout and stderr. One easy way is to do ./bin.testing/Eva -t --suite="temp0" > /dev/null. Any messages you get are stderr since stdout is redirected to /dev/null.
Also, typically with Popen the shell=True option is discouraged unless really needed. Instead pass a list:
[os.path.join(self.exedir, 'Eva'), '-t', '--suite=' + fen_file], shell=False, ...
This can avoid problems down the line if one of your arguments would normally be interpreted by the shell. (Note, I removed the ""'s, because the shell would normally eat those for you!)
Try using subprocess check_output.
output_lines = subprocess.check_output(['./bin.testing/Eva', '-t', '--suite="temp0"'])
for line in output_lines.splitlines():
print(line)
I need to open an R script and supply it with input formulated by a separate python script. The subprocess module seems to be a good way to do this.
I have encountered some puzzling results though, namely that I can apparently write once and only once via p.stdin. Here is what I have so far:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['r --no-save'],stdin=PIPE,stdout=PIPE,stderr=PIPE,shell=True)
p.stdin.write("source('myrscript.R')\n")
p.stdin.write('myfirstinput')
What happens when I run this code is that the first instance of stdin.write() performs as expected (and opens my R script), but the second line does nothing, and the subprocess (really, the R script) exits with an error, indicating that the subprocessed received no input where input was expected and therefore terminated.
N.B. - In a perfect world, I would just interact directly through R, but this particular script requires complex inputs that cannot be entered directly for practical purposes. Also, rpy / rpy2 is not an option, because end-users of this script will not necessarily have access to that module or its dependencies. rscript is also not an option (for many reasons, but mainly because of variability in the end-users R configurations).
Finally, p.communicate is not an option, because apparently that will close the process after writing and I need to keep it open.
Thanks in advance
What you need is to call .communicate():
from subprocess import Popen, PIPE, STDOUT
p = Popen(
['r', '--nosave'],
stdin=PIPE,
stdout=PIPE,
stderr=PIPE)
p.stdin.write("source('myrscript.R')\n")
p.stdin.write('myfirstinput\n')
p.stdin.write('q\n')
stdout, stderr = p.communicate()
print '---STDOUT---'
print stdout
print '---STDERR---'
print stderr
print '---'
Discussion
I don't use the shell=True and it seems working with my fake R script since I don't have R install in my system. You might or might not need it.
I prefer breaking the command line up into a list of string as shown, but a single string such as r --nosave will work as well; just don't do them both at the same time.
Don't forget that stdin.write() does not write the new line character \n, you have to supply that yourself.
Update
My first attempt was off the mark, I hope this second attempt gets closer. As J.F. Sebastian suggested, you might want to use pexpect:
import pexpect
import sys
if __name__ == '__main__':
prompt = '> ' # Don't know what the R prompt looks like
lines = ['one', 'two', 'three']
r = pexpect.spawn('r --no-save', logfile=sys.stdout)
for line in lines:
r.expect(prompt)
r.sendline(line)
# If you want to interact with your script, use these two lines
# Otherwise, comment them out
r.logfile = None # Turn off logging to sys.stdout
r.interact()
Discussion
You might need to install pexpect. I did it with pip install pexpect
If you don't want to interact with the system, comment out the last two line, but make sure to send some signal for the R script to exit.
spawn() returns a spawn object, see doc here.