We Keep Coding

Executing script inside python subprocess and getting the output

I am trying to get execute one script with some arguments and getting its out to a buffer for further processing. The original code snippet which works now as below-
cmd = "/pathToScript/myscript.sh --endpoint "+ ip_addr + " --register '"+ reg_token + "' --output-file " + pathToOutputFile
os.system(cmd)
The above cmd runs and creates the output file where it write the data generated by myhelper script.
Now I want to run the same script, but instead of writing to file, I am expecting the output to a buffer as returned value.
I rewrite the code line as below, but it does not generates the data into buffer.
creds = subprocess.run(
["/pathToScript/myscript.sh", "--endpoint {0}".format(endpoint_ipaddr), "--register {0}".format(reg_token)],
check=True,
stdout=subprocess.PIPE,
universal_newlines=True
).stdout
print("creds")
I tried with both subprocess.run() and call() command, but both did not return anything. Is my modified code is correct with respect to executing the script and getting the output collected to returned variable?
Appreciate your help on this.
thank you

How can I run a binary executable with input file (bash command) in Python?

I have a binary executable named as "abc" and I have a input file called as "input.txt". I can run these with following bash command:
./abc < input.txt
How can I run this bash command in Python, I tried some ways but I got errors.
Edit:
I also need the store the output of the command.
Edit2:
I solved with this way, thanks for the helps.
input_path = path of the input.txt file.
out = subprocess.Popen(["./abc"],stdin=open(input_path),stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout,stderr = out.communicate()
print(stdout)

use os.system
import os
os.system("echo test from shell");

Using subprocess is the best way to invoke system commands and executables. It provides better control than os.system() and is intended to replace it. The python documentation link below provides additional information.
https://docs.python.org/3/library/subprocess.html
Here is a bit of code that uses subprocess to read output from head to return the first 100 rows from a txt file and process it row by row. It gives you the output (out) and any errors (err).
mycmd = 'head -100 myfile.txt'
(out, err) = subprocess.Popen(mycmd, stdout=subprocess.PIPE, shell=True).communicate()
myrows = str(out.decode("utf-8")).split("\n")
for myrow in myrows:
# do something with myrow

This can be done with os module. The following code works perfectly fine.
import os
path = "path of the executable 'abc' and 'input.txt' file"
os.chdir(path)
os.system("./abc < input.txt")
Hope this works :)

Running a batch file with subprocess

I'd like to execute a simple batch file using Python. But I'm getting some error back from the process saying the file, directory or disc name is not right. I guess the best way to start is to show the code:
import subprocess as sp
from pathlib import Path
file = Path(r'C:\Program Files (x86)\test.bat')
p = sp.Popen(['"' + str(file) + '"'], stdin=sp.PIPE, stdout=sp.PIPE, stderr=sp.PIPE,
shell=True, universal_newlines=True)
outs, errs = p.communicate('', timeout=5)
print(outs, '\n-----\n', errs)
I extended this with appending to system path and changing the working directory:
import os
import sys
sys.path.append(file.parent)
os.chdir(file.parent)
The batch file contains just a few echo commands for debugging. So I'd expect the code above to print the contents of the echoes. I've verified that I'm able to call it inside a command prompt from any folder. Previously I was getting some file permission error (WinError 5), so that might be related especially as the file is in Program Files. This error was not from the process, but Python itself.
I also tried it with an executable, and a similar error was raised: WinError 2: the system cannot find the file specified. Any idea where I'm stumbling?
Attempts
When the shell=True keyword is removed, the WinError 5 is back
When Popen is called with ['cmd'] and the batch file is run with p.communicate('"' + str(file) + '"\n', timeout=5), no errors are thrown, and the output contains the echoes. However batch files should run without explicitly opening a command prompt, I presume.

Use a command-line string instead of an args list when passing shell=True or when running a batch script with the default shell=False.
On Windows, Popen processes an args list into a command line that's compatible with VC++ argument parsing. But cmd.exe don't use VC++ argument parsing rules, and even for an executable that does use VC++ rules (e.g. python.exe), the extra quotes you're adding (i.e. '"' + str(file) + '"') get backslash-escaped as literal quotation marks in the command line.

Running bash command on server

I am trying to run the bash command pdfcrack in Python on a remote server. This is my code:
bashCommand = "pdfcrack -f pdf123.pdf > myoutput.txt"
import subprocess
process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
I, however, get the following error message:
Non-option argument myoutput2.txt
Error: file > not found
Can anybody see my mistake?

The first argument to Popen is a list containing the command name and its arguments. > is not an argument to the command, though; it is shell syntax. You could simply pass the entire line to Popen and instruct it to use the shell to execute it:
process = subprocess.Popen(bashCommand, shell=True)
(Note that since you are redirecting the output of the command to a file, though, there is no reason to set its standard output to a pipe, because there will be nothing to read.)
A better solution, though, is to let Python handle the redirection.
process = subprocess.Popen(['pdfcrack', '-f', 'pdf123.pdf'], stdout=subprocess.PIPE)
with open('myoutput.txt', 'w') as fh:
for line in process.stdout:
fh.write(line)
# Do whatever else you want with line
Also, don't use str.split as a replacement for the shell's word splitting. A valid command line like pdfcrack -f "foo bar.pdf" would be split into the incorrect list ['pdfcrack', '-f', '"foo', 'bar.pdf"'], rather than the correct list ['pdfcrack', '-f', 'foo bar.pdf'].

> is interpreted by shell, but not valid otherwise.
So, that would work (don't split, use as-is):
process = subprocess.Popen(bashCommand, shell=True)
(and stdout=subprocess.PIPE isn't useful since all output is redirected to the output file)
But it could be better with native python for redirection to output file and passing arguments as list (handles quote protection if needed)
with open("myoutput.txt","w") as f:
process = subprocess.Popen(["pdfcrack","-f","pdf123.pdf"], stdout=subprocess.PIPE)
f.write(process.read())
process.wait()

Your mistake is > in command.
It doesn't treat this as redirection to file because normally bash does it and now you run it without using bash.
Try with shell=True if you whan to use bash. And then you don't have to split command into list.
subprocess.Popen("pdfcrack -f pdf123.pdf > myoutput.txt", shell=True)

Python subprocess library: Running grep command from Python

I am trying to run grep command from my Python module using the subprocess library. Since, I am doing this operation on the doc file, I am using Catdoc third party library to get the content in a plan text file. I want to store the content in a file. I don't know where I am going wrong but the program fails to generate a plain text file and eventually to get the grep result. I have gone through the error log but its empty. Thanks for all the help.
def search_file(name, keyword):
#Extract and save the text from doc file
catdoc_cmd = ['catdoc', '-w' , name, '>', 'testing.txt']
catdoc_process = subprocess.Popen(catdoc_cmd, stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
output = catdoc_process.communicate()[0]
grep_cmd = []
#Search the keyword through the text file
grep_cmd.extend(['grep', '%s' %keyword , 'testing.txt'])
print grep_cmd
p = subprocess.Popen(grep_cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
stdoutdata = p.communicate()[0]
print stdoutdata

On UNIX, specifying shell=True will cause the first argument to be treated as the command to execute, with all subsequent arguments treated as arguments to the shell itself. Thus, the > won't have any effect (since with /bin/sh -c, all arguments after the command are ignored).
Therefore, you should actually use
catdoc_cmd = ['catdoc -w "%s" > testing.txt' % name]
A better solution, though, would probably be to just read the text out of the subprocess' stdout, and process it using re or Python string operations:
catdoc_cmd = ['catdoc', '-w' , name]
catdoc_process = subprocess.Popen(catdoc_cmd, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
for line in catdoc_process.stdout:
if keyword in line:
print line.strip()

I think you're trying to pass the > to the shell, but that's not going to work the way you've done it. If you want to spawn a process, you should arrange for its standard out to be redirected. Fortunately, that's really easy to do; all you have to do is open the file you want the output to go to for writing and pass it to popen using the stdout keyword argument, instead of PIPE, which causes it to be attached to a pipe which you can read with communicate().