We Keep Coding

how can I redirect an external command output to a file using python

What would be the best way to capture the output of an external command in python?
I'm using Popen from the subprocess method to run my command which takes approx 3 minutes to complete but when I'm trying to cat the same file where I've captured the output there is no output I think it is because of some buffer issue.
with open(os.path.join(path, file), 'w') as fp:
with contextlib.redirect_stdout(fp):
sys.stdout = fp
subprocess.run(command, shell=True)
sys.stdout.close()
or is there any way to capture sbuild output using os.popen or subprocess.getstatusoutput()

Redirect subprocess.run output to file

I've seen a lot of other pages here that talk output grep output to a file but I can't seem to make any of those work.
I have
subprocess.run(['grep', '-o', searchFor, filename])
if this wasn't a subprocess I would use something like
grep -o searchfor filename >> outputfile.txt
and when I try to use > or >> or anything else in the subprocess, I can't get it to output to a simple txt file. I assume that this is because most all of the pages I've seen here are writing to a file from regular command compared to me trying within the subprocess. My guess is that my syntax is wrong. Where should > or >> or whatever I should be using go? I've tried after the ] and before and many other combinations.

Open the file in write (>) or append (>>) mode, and assign the descriptor associated with it to stdout in subprocess.run call.
with open('outputfile.txt', 'w') as fd:
subprocess.run(['grep', '-o', searchFor, filename], stdout=fd)

How can I run a binary executable with input file (bash command) in Python?

I have a binary executable named as "abc" and I have a input file called as "input.txt". I can run these with following bash command:
./abc < input.txt
How can I run this bash command in Python, I tried some ways but I got errors.
Edit:
I also need the store the output of the command.
Edit2:
I solved with this way, thanks for the helps.
input_path = path of the input.txt file.
out = subprocess.Popen(["./abc"],stdin=open(input_path),stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout,stderr = out.communicate()
print(stdout)

use os.system
import os
os.system("echo test from shell");

Using subprocess is the best way to invoke system commands and executables. It provides better control than os.system() and is intended to replace it. The python documentation link below provides additional information.
https://docs.python.org/3/library/subprocess.html
Here is a bit of code that uses subprocess to read output from head to return the first 100 rows from a txt file and process it row by row. It gives you the output (out) and any errors (err).
mycmd = 'head -100 myfile.txt'
(out, err) = subprocess.Popen(mycmd, stdout=subprocess.PIPE, shell=True).communicate()
myrows = str(out.decode("utf-8")).split("\n")
for myrow in myrows:
# do something with myrow

This can be done with os module. The following code works perfectly fine.
import os
path = "path of the executable 'abc' and 'input.txt' file"
os.chdir(path)
os.system("./abc < input.txt")
Hope this works :)

Reading from stdin with a system argv

I am attempting to cat a CSV file into stdout and then pipe the printed output as input into a python program that also takes a system argument vector with 1 argument. I ran into an issue I think directly relates to how Python's fileinput.input() function reacts with regards to occupying the stdin file descriptor.
generic_user% cat my_data.csv | python3 my_script.py myarg1
Here is a sample Python program:
import sys, fileinput
def main(argv):
print("The program doesn't even print this")
data_list = []
for line in fileinput.input():
data_list.append(line)
if __name__ == "__main__":
main(sys.argv)
If I attempt to run this sample program with the above terminal command and no argument myarg1, the program is able to evaluate and parse the stdin for the data output from the CSV file.
If I run the program with the argument myarg1, it will end up throwing a FileNotFoundError directly related to myarg1 not existing as a file.
FileNotFoundError: [Errno 2] No such file or directory: 'myarg1'
Would someone be able to explain in detail why this behavior takes place in Python and how to handle the logic such that a Python program can first handle stdin data before argv overwrites the stdin descriptor?

You can read from the stdin directly:
import sys
def main(argv):
print("The program doesn't even print this")
data_list = []
for line in iter(sys.stdin):
data_list.append(line)
if __name__ == "__main__":
main(sys.argv)
You are trying to access a file which has not been yet created, hence fileinput cannot open it, but since you are piping the data you have no need for it.

This is by design. The conceptors of fileinput thought that there were use cases where reading from stdin would be non sense and just provided a way to specifically add stdin to the list of files. According to the reference documentation:
import fileinput
for line in fileinput.input():
process(line)
This iterates over the lines of all files listed in sys.argv[1:], defaulting to sys.stdin if the list is empty. If a filename is '-', it is also replaced by sys.stdin.
Just keep your code and use: generic_user% cat my_data.csv | python3 my_script.py - myarg1
to read stdin before myarg1 file or if you want to read it after : ... python3 my_script.py myarg1 -

fileinput implements a pattern common for Unix utilities:
If the utility is called with commandline arguments, they are files to read from.
If it is called with no arguments, read from standard input.
So fileinput works exactly as intended. It is not clear what you are using commandline arguments for, but if you don't want to stop using fileinput, you should modify sys.argv before you invoke it.
some_keyword = sys.argv[1]
sys.argv = sys.argv[:1] # Retain only argument 0, the command name
for line in fileinput.input():
...

Execute zgrep command and write results to a file

I have a folder containing lots of files like file_1.gz to file_250.gz and increasing.
A zgrep command which searches through them is like:
zgrep -Pi "\"name\": \"bob\"" ../../LM/DATA/file_*.gz
I want to execute this command in a python subprocess like:
out_file = os.path.join(out_file_path, file_name)
search_command = ['zgrep', '-Pi', '"name": "bob"', '../../LM/DATA/file_*.gz']
process = subprocess.Popen(search_command, stdout=out_file)
The problem is the out_file is created but it is empty and these errors are raised:
<type 'exceptions.AttributeError'>
'str' object has no attribute 'fileno'
What is the solution?

You need to pass a file object:
process = subprocess.Popen(search_command, stdout=open(out_file, 'w'))
Citing the manual, emphasis mine:
stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, an existing file descriptor (a positive integer), an existing file object, and None. PIPE indicates that a new pipe to the child should be created. With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent.
Combined with LFJ's answer - using the convenience functions is recommended, and you need to use shell=True to make the wildcard (*) work:
subprocess.call(' '.join(search_command), stdout=open(out_file, 'w'), shell=True)
Or when you're using shell anyways, you can use the shell redirection as well:
subprocess.call("%s > %s" % (' '.join(search_command), out_file), shell=True)

There are two issues:
you should pass something with a valid .fileno() method instead of the filename
the shell expands * but subprocess does not invoke the shell unless you ask. You could use glob.glob() to expand the file patterns manually.
Example:
#!/usr/bin/env python
import os
from glob import glob
from subprocess import check_call
search_command = ['zgrep', '-Pi', '"name": "bob"']
out_path = os.path.join(out_file_path, file_name)
with open(out_path, 'wb', 0) as out_file:
check_call(search_command + glob('../../LM/DATA/file_*.gz'),
stdout=out_file)

if your want to execute a shell command and get the output, try to use subprocess.check_output(). it is very simple, and you could save the output to a file easily.
command_output = subprocess.check_output(your_search_command, shell=True)
with open(out_file, 'a') as f:
f.write(command_output)

My problem consist of two parts:
First part is answered by #liborm as well
The second part is related to the files that zgrep tries to search in. when we write a command like zgrep "pattern" path/to/files/*.gz the bash automatically removes the *.gz by all files ends with .gz. When i run the command in a subprocess no one replaced the *.gz by real file, in consequence the error gzip: ../../LM/DATA/file_*.gz: No such file or directory raises. So solved it by:
for file in os.listdir(archive_files_path):
if file.endswith(".gz"):
search_command.append(os.path.join(archive_files_path, file))