I am writing a script which would run a Linux command and write a string (up to EOL) to stdin and read a string (until EOL) from stdout. The easiest illustration would be cat - command:
p=subprocess.Popen(['cat', '-'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
stringin="String of text\n"
p.stdin.write=(stringin)
stringout=p.stout.read()
print(stringout)
I aim to open the cat - process once and use it to write a string multiple times to its stdin every time getting a string from its stdout.
I googled quite a bit and a lot of recipes don't work, because the syntax is incompatible through different python versions (I use 3.4). That is my first python script from scratch and I find the python documentation to be quite confusing so far.
Thank you for your solution Salva.
Unfortunately communicate() closes the cat - process. I did not find any solution with subprocess to communicate with the cat - without having to open a new cat - for every call. I found an easy solution with pexpect though:
import pexpect
p = pexpect.spawn('cat -')
p.setecho(False)
def echoback(stringin):
p.sendline(stringin)
echoback = p.readline()
return echoback.decode();
i = 1
while (i < 11):
print(echoback("Test no: "+str(i)))
i = i + 1
In order to use pexpect Ubuntu users will have to install it through pip. If you wish to install it for python3.x, you will have to install pip3 (python3-pip) first from the Ubuntu repo.
Well you need to communicate with the process:
from subprocess import Popen, PIPE
s = Popen(['cat', '-'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
input = b'hello!' # notice the input data are actually bytes and not text
output, errs = s.communicate(input)
To use unicode strings, you would need to encode() the input and decode() the output:
from subprocess import Popen, PIPE
s = Popen(['cat', '-'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
input = 'España'
output, errs = s.communicate(input.encode())
output, errs = output.decode(), errs.decode()
Related
This question already has answers here:
Store output of subprocess.Popen call in a string [duplicate]
(15 answers)
Closed 5 months ago.
I am trying to use Popen to scp a file from my laptop to a device on my network. The process is pretty straight foward...I can get the file to transfer but I cant get the output from the command to display. I am specificlly looking for the percentage complete. Here is what I have:
from subprocess import Popen, STDOUT, PIPE
scp_command = 'scp -i c:<local_key> <filepath to local file> <user>#<destination_device>:\path'
local_scp_command = Popen(scp_command, text=True, stout=PIPE)
output = local_scp_transfer.communicate
print(output)
I have tried a number of different combinations of stdout and printing the output. I cant even remember all the ways I have tried this. I imagine that there is something kind of easy that I am missing. I am pretty new at programming so even the easy things are compliacted for me.
Thank you so much for all your help!
Use poll() to determine whether or not the process has finished and read a line:
from subprocess import Popen, STDOUT, PIPE
import shlex
scp_command = 'scp -i c:<local_key> <filepath to local file> <user>#<destination_device>:\path'
local_scp_command = Popen(shlex.split(scp_command), text=True, stdout=PIPE)
while local_scp_command.poll() is None and line := local_scp_command.stdout.readline():
print(line)
I added a shlex.split because that's the proper format for Popen.
I wrote a python script that uses subprocess.Popen to call command line tool look to do the binary search on a file.
For example
p = subprocess.Popen('look -b "abc" testfile.txt',executable='/bin/bash', stdout=subprocess.PIPE, stderr=STDOUT, shell=True)
out, err = p.communicate()
result = out.decode()
print(result)
What this snippet of code does is that it calls the system command look to perform a binary search the file called testfile.txt for the string abc.
It works fine if you just have this snippet of code.
However, when your memory is loaded with some large files, it becomes significantly slow.
For example, if you do:
a = read_a_large_file() #Like GBs of data
p = subprocess.Popen('look -b "abc" testfile.txt',executable='/bin/bash', stdout=subprocess.PIPE, stderr=STDOUT, shell=True)
out, err = p.communicate()
result = out.decode()
print(result)
a[0]
The subprocess part takes a very long time to execute. Running the look command is very fast in shell as it performs binary search on sorted files.
Any help will be appreciated! Thanks!
I'm trying to get the output of a command's STDOUT with the HandBrakeCLI program when encoding a video. I can't seem to get python to handle its output on the standard output stream. I've tried the following codes:
import subprocess
import sys
encode = subprocess.check_output("HandBrakeCLI -i video.mkv -o out.mp4", shell=True, stderr=subprocess.STDOUT, universal_newlines=True)
print(encode)
This printed nothing as well as this which I also tried:
import subprocess
import sys
encode = subprocess.Popen("HandBrakeCLI -i video.mkv -o out.mp4", stdout=subprocess.PIPE, stderr = subprocess.PIPE, shell=True, universal_newlines=True)
print(encode.stdout.read())
As stated before, both will result in no output. This application is the type that will update text on a single line in bash as it's encoding. I'm not sure if that type of data stream creates a problem with python or not.
It seems HandBrakeCLI changes its output depending on whether it prints to a terminal. Either specify a command-line flag to force the necessary output or you could trick it by providing a pseudo-tty (if your system supports it) using pexpect or pty module directly.
Code examples on how to get output from a subprocess using pexpect, pty modules:
Last unbuffered line can't be read
Python subprocess readlines() hangs
I'm trying to talk to a child process using the python subprocess.Popen() call. In my real code, I'm implementing a type of IPC, so I want to write some data, read the response, write some more data, read the response, and so on. Because of this, I cannot use Popen.communicate(), which otherwise works well for the simple case.
This code shows my problem. It never even gets the first response, hangs at the first "Reading result". Why? How can I make this work as I expect?
import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
stdout = subprocess.PIPE,
stdin = subprocess.PIPE)
p.stdin.write("abc\n")
print "Reading result:"
print p.stdout.readline()
p.stdin.write("cat\n")
print "Reading result:"
print p.stdout.readline()
sed's output is buffered and only outputs its data until enough has been cumulated or the input stream is exhausted and closed.
Try this:
import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
stdout = subprocess.PIPE,
stdin = subprocess.PIPE)
p.stdin.write("abc\n")
p.stdin.write("cat\n")
p.stdin.close()
print "Reading result 1:"
print p.stdout.readline()
print "Reading result 2:"
print p.stdout.readline()
Be aware that this cannot be done reliably which huge data as wriring to stdin blocks once the buffer is full. The best way to do is using communicate().
I would try to use Popen().communicate() if you can as it does a lot of nice things for you, but if you need to use Popen() exactly as you described, you'll need to set sed to flush its buffer after newlines with the -l option:
p = subprocess.Popen(['sed', '-l', 's/a/x/g'],
stdout=subprocess.PIPE,
stdin=subprocess.PIPE)
and your code should work fine
I am writing a python program in linux and in part of it running the pdftotext executable to convert a pdf text. The code I am currently using is given below.
pdfData = currentPDF.read()
tf = os.tmpfile()
tf.write(pdfData)
tf.seek(0)
out, err = subprocess.Popen(["pdftotext", "-", "-"], stdin = tf, stdout=subprocess.PIPE ).communicate()
This works fine, but now I want to run the pdftotext executable with the -layout option (preserves layout of document). I tried replacing the "-" with layout, replacing "pdftotext" with "pdftotext -layout" etc. None of it works. They all give me an empty text. Since the input is being piped in via the temp file, I am having trouble figureing out the argument list. Most of the documentation on Popen assumes all the parameters are being passed in through the argument list, but in my case the input is being passed in through the temp file.
Any help would be greatly appreciated.
This works for me:
out, err = subprocess.Popen(
["pdftotext", '-layout', "-", "-"], stdin = tf, stdout=subprocess.PIPE ).communicate()
Although I couldn't find explicit confirmation in the man page, I believe the first - tells pdftotext to expect PDF-file to come from stdin, and the second - tells pdftotext to expect text-file to be sent to stdout.
You can pass the full command in string with shell=True:
out, err = subprocess.Popen('pdftotext -layout - -', shell=True, stdin=tf, stdout=subprocess.PIPE).communicate()