I am trying to spawn a process using Popen and send it a particular string to its stdin.
I have:
pipe = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE)
pipe.communicate( my_stdin_str.encode(encoding='ascii') )
pipe.stdin.close()
However, the second line actually escapes the whitespace in my_stdin_str. For example, if I have:
my_stdin_str="This is a string"
The process will see:
This\ is\ a\ string
How can I prevent this behaviour?
I can't reproduce it on Ubuntu:
from subprocess import Popen, PIPE
shell_cmd = "perl -pE's/.\K/-/g'"
p = Popen(shell_cmd, shell=True, stdin=PIPE)
p.communicate("This $PATH is a string".encode('ascii'))
In this case shell=True is unnecessary:
from subprocess import Popen, PIPE
cmd = ["perl", "-pE" , "s/.\K/-/g"]
p = Popen(cmd, stdin=PIPE)
p.communicate("This $PATH is a string".encode('ascii'))
Both produce the same output:
T-h-i-s- -$-P-A-T-H- -i-s- -a- -s-t-r-i-n-g-
Unless you know you need it for some reason, don't run with "shell=True" in general (which, without testing, sounds like what's going on here).
Related
I have this command which I can get the external IP address returned from Unix shell, so I can use it in my server:
ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'
on my mac, terminal returns:
192.168.1.3
How do I get this outputted in the python script? I have tried:
import subprocess
command = ['ifconfig', '|', 'sed', '-En', 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p']
p = subprocess.Popen(command, stdout=subprocess.PIPE)
text = p.stdout.read()
retcode = p.wait()
and I got error says
ifconfig: interface | does not exist
Thanks in advance!
Your shell command is invoking two commands, the output of ifconfig is used as input to sed. You could emulate this using subprocess but the sed call is just doing some text manipulation, so a cleaner approach would just be to use Python for that step. For example:
import re
import subprocess
pattern = r'inet (?:addr:)?(?!127\.0\.0\.1)((?:\d+\.){3}\d+)'
p = subprocess.Popen(['ifconfig'], stdout=subprocess.PIPE)
text = re.search(pattern, p.stdout.read()).group(1)
retcode = p.wait()
ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'
you are trying to envoke 2 shell commands, ifconfig and sed, which is okay. but, these are shell commands and the shell keyword argument must be set to true when calling the subprocess.Poen.
use the communictae method, it better to use here. and dispatch the command as a string not list.
import subprocess
command = ' '.join(['ifconfig', '|', 'sed', '-En', 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'])
p = subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
stdoutdata, stderrdata = p.communicate() #this is blocking
for line in stdoutdata:
#do some thing with line
Popen.communicate(input=None)
Interact with process: Send data to stdin. Read data from stdout and
stderr, until end-of-file is reached. Wait for process to terminate.
The optional input argument should be a string to be sent to the child
process, or None, if no data should be sent to the child.
Have been trying to get something like this to work for a while, the below doesn't seem to be sending the correct arg to the c program arg_count, which outputs argc = 1. When I'm pretty sure I would like it to be 2. ./arg_count -arg from the shell outputs 2...
I have tried with another arg (so it would output 3 in the shell) and it still outputs 1 when calling via subprocess.
import subprocess
pipe = subprocess.Popen(["./args/Release/arg_count", "-arg"], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = pipe.communicate()
result = out.decode()
print "Result : ",result
print "Error : ",err
Any idea where im falling over? I'm running linux btw.
From the documentation:
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence.
Thus,
pipe = subprocess.Popen("./args/Release/arg_count -arg", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
should give you what you want.
If shell=True then your call is equivalent to:
from subprocess import Popen, PIPE
proc = Popen(['/bin/sh', '-c', "./args/Release/arg_count", "-arg"],
stdout=PIPE, stderr=PIPE)
i.e., -arg is passed to the shell itself and not your program. Drop shell=True to pass -arg to the program:
proc = Popen(["./args/Release/arg_count", "-arg"],
stdout=PIPE, stderr=PIPE)
If you don't need to capture stderr separately from stdout then you could use check_output():
from subprocess import check_output, STDOUT
output = check_output(["./args/Release/arg_count", "-arg"]) # or
output_and_errors = check_output(["./args/Release/arg_count", "-arg"],
stderr=STDOUT)
Im trying to get a linux binary to send its standard output to a variable by using subprocess. But just keep getting tracebacks.
>>> import subprocess
>>>nmap -sn -Pn todd.ns.cloudflare.com --script dns-check-zone --script-args='dns-check-zone.domain=www.macmonster.com
Any Ideas (oh and Im using Python2.7).
Ideally I would like to avoid using Shell=true to avoid any security concerns.
Thanks,
shlex to the rescue!
The module shlex will take a string containing the whole shell command and split it up exactly how Popen and check_output expect it. Like this:
import shlex, subprocess
cmd = "/usr/bin/nmap -sn -Pn todd.ns.cloudflare.com --script dns-check-zone --script-args='dns-check-zone.domain=www.macmonster.com'"
args = shlex.split(cmd)
output = subprocess.check_output(args)
When you look at contents of args you'll see:
>>> print args
['/usr/bin/nmap', '-sn', '-Pn', 'todd.ns.cloudflare.com', '--script', 'dns-check-zone', '--script-args=dns-check-zone.domain=www.macmonster.com']
Note that shlex split up the option "--script dns-check-zone" into two tokens. On the other hand it kept "--script-args='dns-check-zone.domain=www.macmonster.com'", but removed the single-quotes.
import subprocess
output = subprocess.check_output(["/usr/bin/nmap", "-sP", "-n", "172.16.1.0/24"])
output = subprocess.check_output(["/usr/bin/nmap", "-sP", "-n", "172.16.1.0/24"], stderr=subprocess.STDOUT)
Have you tried this:
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
var = p.stdout.read()
print var
I would have used communicate, but it returns an odd list-type thing.
I am trying to convert several pieces of html to latex using python and pandoc and I have got stuck with a couple of problems.
To communicate my python script with pandoc I use subprocess.Popen, redirecting stdout to a file I am saving for including it in a latex template.
If I use the classic way of implementing Popen
from subprocess import Popen, PIPE, STDOUT
filedesc = open('myfile.tex','w')
args = ['pandoc', '-f', 'html', '-t', 'latex']
p = Popen(args, stdout=PIPE, stdin=PIPE, stderr=STDOUT)
outp, err = p.communicate(input=html)
filedesc.write(outp)
I get the lines with an additional new line where there shouldn't be any:
> \textbf{M. John Harrison} (Rugby, Warckwickshire, 1945) is a contemporary
>
> English writer.
This is (misteriously?) easy to solve by changing the stdout=PIPE to the file descriptor:
from subprocess import Popen, PIPE, STDOUT
filedesc = open('myfile.tex','w')
args = ['pandoc', '-f', 'html', '-t', 'latex']
p = Popen(args, stdout=filedesc, stdin=PIPE, stderr=STDOUT)
outp, err = p.communicate(input=html)
# not needed
# filedesc.write(outp)
But if I want to use a string buffer, the same problem occurs, since i cannot use it as the stdout parameter.
Any idea on how to stop Popen/pandoc from doing this?
Thanks!
Well, it seems to be a "kind of bug" in python's PIPE (???).
I am executing this code in a Windows system. This means that when a new line is entered, they are in the CR+LF (\r\n) style rather than the (cleaner) LF (\n) new line in unix-style.
At the time I introduce a large html text to be converted by pandoc, the output is returned by the pipe to the command line. Thus, every time the standard column width is reached, an ugly "new line" character is introduced. In my case, a CR+LF. This was making my output look so weird.
The dirty solution I have implemented is to add a replace('\r\n','\n') before writing the output but I am not sure if it's the most elegant one.
from subprocess import Popen, PIPE, STDOUT
html = '<p><b>Some random html code</b> longer than 80 columns ... </p>'
filedesc = open('myfile.tex','w')
args = ['pandoc', '-f', 'html', '-t', 'latex']
p = Popen(args, stdout=PIPE, stdin=PIPE, stderr=STDOUT)
outp, err = p.communicate(input=html)
filedesc.write(outp.replace('\r\n','\n'))**strong text**
There's a file named startup.cmd that sets some environment variables, runs some preparation commands, then does:
start "startup" cmd /k
Which opens a command shell named startup. The manual process I'm trying to automate is to then enter the following command into this shell: get startup.xml. I thought the correct way to do this in Python would be something like this:
import subprocess
p = subprocess.Popen('startup.cmd', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
getcommand = 'get startup.xml'
servercommand = 'startserver'
p.stdin.write(getcommand)
p.stdin.write(startserver)
(stdoutdata, stderrdata) = p.communicate()
print stdoutdata
print stderrdata
But those commands don't seem to be executing in the shell. What am I missing? Also, the command shell appears regardless of whether shell is set to True or False.
I found this warning in subprocess's document,
Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
So my suggestion is to use communicate to send your command.
import subprocess
p = subprocess.Popen('startup.cmd', shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
command = 'get startup.xml\n'
command += 'startserver\n'
(stdoutdata, stderrdata) = p.communicate(command)
print stdoutdata
print stderrdata
This is a new process, so one cannot communicate directly with Popen.