We Keep Coding

Can not pass special character to subprocess in python

I have this command which I can get the external IP address returned from Unix shell, so I can use it in my server:
ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'
on my mac, terminal returns:
192.168.1.3
How do I get this outputted in the python script? I have tried:
import subprocess
command = ['ifconfig', '|', 'sed', '-En', 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p']
p = subprocess.Popen(command, stdout=subprocess.PIPE)
text = p.stdout.read()
retcode = p.wait()
and I got error says
ifconfig: interface | does not exist
Thanks in advance!

Your shell command is invoking two commands, the output of ifconfig is used as input to sed. You could emulate this using subprocess but the sed call is just doing some text manipulation, so a cleaner approach would just be to use Python for that step. For example:
import re
import subprocess
pattern = r'inet (?:addr:)?(?!127\.0\.0\.1)((?:\d+\.){3}\d+)'
p = subprocess.Popen(['ifconfig'], stdout=subprocess.PIPE)
text = re.search(pattern, p.stdout.read()).group(1)
retcode = p.wait()

ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'
you are trying to envoke 2 shell commands, ifconfig and sed, which is okay. but, these are shell commands and the shell keyword argument must be set to true when calling the subprocess.Poen.
use the communictae method, it better to use here. and dispatch the command as a string not list.
import subprocess
command = ' '.join(['ifconfig', '|', 'sed', '-En', 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'])
p = subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
stdoutdata, stderrdata = p.communicate() #this is blocking
for line in stdoutdata:
#do some thing with line
Popen.communicate(input=None)
Interact with process: Send data to stdin. Read data from stdout and
stderr, until end-of-file is reached. Wait for process to terminate.
The optional input argument should be a string to be sent to the child
process, or None, if no data should be sent to the child.

Unix Popen.communicate not able to gzip large file

I need to gzip files of size more than 10 GB using python on top of shell commands and hence decided to use subprocess Popen.
Here is my code:
outputdir = '/mnt/json/output/'
inp_cmd='gzip -r ' + outputdir
pipe = Popen(["bash"], stdout =PIPE,stdin=PIPE,stderr=PIPE)
cmd = bytes(inp_cmd.encode('utf8'))
stdout_data,stderr_data = pipe.communicate(input=cmd)
It is not gzip-ing the files within output directory.
Any way out?

The best way is to use subprocess.call() instead of subprocess.communicate().
call() waits till the command is executed completely while in Popen(), one has to extrinsically use wait() method for the execution to finish.

Have you tried it like this:
output_dir = "/mnt/json/output/"
cmd = "gzip -r {}".format(output_dir)
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
shell=True,
)
out, err = proc.communicate()

Send input (command) with communicate to created Subprocess Python

I have created a Subprocess object. The subprocess invokes a shell, I need to send the shell command provided below to it. The code I've tried:
from subprocess import Popen, PIPE
p = Popen(["code.exe","25"],stdin=PIPE,stdout=PIPE,stderr=PIPE)
print p.communicate(input='ping 8.8.8.8')
The command doesn't execute, nothing is being input into the shell. Thanks in advance.

If I simulate code.exe to read the arg and then process stdin:
#!/usr/bin/env bash
echo "arg: $1"
echo "stdin:"
while read LINE
do
echo "$LINE"
done < /dev/stdin
and slightly update your code:
import os
from subprocess import Popen, PIPE
cwd = os.getcwd()
exe = os.path.join(cwd, 'foo.sh')
p = Popen([exe, '25'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
out, err = p.communicate(input='aaa\nbbb\n')
for line in out.split('\n'):
print(line)
Then the spawned process outputs:
arg: 25
stdin:
aaa
bbb
If input is changed without a \n though:
out, err = p.communicate(input='aaa')
Then it doesn't appear:
arg: 25
stdin:
Process finished with exit code 0
So you might want to look closely at the protocol between both ends of the pipe. For example this might be enough:
input='ping 8.8.8.8\n'
Hope that helps.

Python messing vim when spawning

I'm trying to run a program from python and print the output based on its exit status. The code below is outputting directly to my vim screen (messing it) instead of opening a shell with the output.
python << EOF
import subprocess
import vim
cmd = "BAD_COMMAND"
p = subprocess.Popen(cmd, shell=True)
retcode = p.poll()
if retcode > 0:
output_of_error = p.communicate()[0]
vim.command("!echo show errors here")
EOF

You want to redirect your subprocess to a pipe:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
communicate() will then return a 2-tuple containing the contents of stdout and stderr of the process.

Help with wrapping a command line tool in Python

I'm really stuck with a problem I'm hoping someone can help me with. I'm trying to create a wrapper in Python3.1 for a command line program called spooky. I can successfully run this program on the command line like this:
$ spooky -a 4 -b .97
My first Python wrapper attempt for spooky looked like this:
import subprocess
start = "4"
end = ".97"
spooky_path = '/Users/path/to/spooky'
cmd = [spooky_path, '-a', start, '-b', end]
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
process.wait()
print('Done')
The above code prints Done, but does not execute the program spooky
Next I tried to just execute the program on the command line like this:
$ /Users/path/to/spooky -a 4 -b .97
The above code also fails, and provides no helpful errors.
My question is: How can I get Python to run this program by sending spooky -a 4 -b .97 to the command line? I would VERY much appreciate any help you can provide. Thanks in advance.

You need to drop the stdout=subprocess.PIPE. Doing that disconnects the stdout of your process from Python's stdout and makes it retrievable using the Popen.communicate() function, like so:
import subprocess
spooky_path = 'ls'
cmd = [spooky_path, '-l']
process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
output = process.communicate()[0]
print "Output:", output
process.wait()
print('Done')
To make it print directly you can use it without the stdout argument:
process = subprocess.Popen(cmd)
Or you can use the call function:
process = subprocess.call(cmd)

Try making your command into a single string:
cmd = 'spooky_path -a start -b end'
process = subprocess.Popen(cmd, shell=True)