I have created a Subprocess object. The subprocess invokes a shell, I need to send the shell command provided below to it. The code I've tried:
from subprocess import Popen, PIPE
p = Popen(["code.exe","25"],stdin=PIPE,stdout=PIPE,stderr=PIPE)
print p.communicate(input='ping 8.8.8.8')
The command doesn't execute, nothing is being input into the shell. Thanks in advance.
If I simulate code.exe to read the arg and then process stdin:
#!/usr/bin/env bash
echo "arg: $1"
echo "stdin:"
while read LINE
do
echo "$LINE"
done < /dev/stdin
and slightly update your code:
import os
from subprocess import Popen, PIPE
cwd = os.getcwd()
exe = os.path.join(cwd, 'foo.sh')
p = Popen([exe, '25'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
out, err = p.communicate(input='aaa\nbbb\n')
for line in out.split('\n'):
print(line)
Then the spawned process outputs:
arg: 25
stdin:
aaa
bbb
If input is changed without a \n though:
out, err = p.communicate(input='aaa')
Then it doesn't appear:
arg: 25
stdin:
Process finished with exit code 0
So you might want to look closely at the protocol between both ends of the pipe. For example this might be enough:
input='ping 8.8.8.8\n'
Hope that helps.
Related
I have this command which I can get the external IP address returned from Unix shell, so I can use it in my server:
ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'
on my mac, terminal returns:
192.168.1.3
How do I get this outputted in the python script? I have tried:
import subprocess
command = ['ifconfig', '|', 'sed', '-En', 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p']
p = subprocess.Popen(command, stdout=subprocess.PIPE)
text = p.stdout.read()
retcode = p.wait()
and I got error says
ifconfig: interface | does not exist
Thanks in advance!
Your shell command is invoking two commands, the output of ifconfig is used as input to sed. You could emulate this using subprocess but the sed call is just doing some text manipulation, so a cleaner approach would just be to use Python for that step. For example:
import re
import subprocess
pattern = r'inet (?:addr:)?(?!127\.0\.0\.1)((?:\d+\.){3}\d+)'
p = subprocess.Popen(['ifconfig'], stdout=subprocess.PIPE)
text = re.search(pattern, p.stdout.read()).group(1)
retcode = p.wait()
ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'
you are trying to envoke 2 shell commands, ifconfig and sed, which is okay. but, these are shell commands and the shell keyword argument must be set to true when calling the subprocess.Poen.
use the communictae method, it better to use here. and dispatch the command as a string not list.
import subprocess
command = ' '.join(['ifconfig', '|', 'sed', '-En', 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'])
p = subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
stdoutdata, stderrdata = p.communicate() #this is blocking
for line in stdoutdata:
#do some thing with line
Popen.communicate(input=None)
Interact with process: Send data to stdin. Read data from stdout and
stderr, until end-of-file is reached. Wait for process to terminate.
The optional input argument should be a string to be sent to the child
process, or None, if no data should be sent to the child.
I have a .war file I'd like to launch via python. I want it to run in the background so no log messages appear in my terminal. Also I would love to have the actual log output put into a logfile. This is the python code I use to solve this.
I had no luck yet. The process is not detached because I cannot run other shell commands after executing the script. The logfile is created but no log-output is appended there.
EDIT: To make things more clear. I want to enhance this script to run multiple java processes in the end. Therefore this python script should spawn those java processes and die in the end. How to achieve exactly that including the functionality of redirecting stdout to a file?
#!/usr/bin/env python3
import subprocess
import re
platformDir = "./platform/"
fe = "frontend-webapp-0.5.0.war"
logfile = open("frontend-log", 'w')
process = subprocess.Popen(['java', '-jar', platformDir + fe],
stdout=subprocess.PIPE)
for line in process.stdout:
logFile.write(line)
Here is how you can try it using Python 3:
import sys
from subprocess import Popen, PIPE, STDOUT
platformDir = "./platform/"
fe = "frontend-webapp-0.5.0.war"
logfile = open("frontend-log", 'ab')
p = Popen(['java', '-jar', platformDir + fe], stdout=PIPE, stderr=STDOUT, bufsize=1)
for line in p.stdout:
sys.stdout.buffer.write(line)
logfile.write(line)
I actually just simply had to give stdout the file handle like stdout=logFile. The solution with the for loop through the stdout would leave me in the python script process all the time
import sys
from subprocess import Popen, PIPE, STDOUT
platformDir = "./platform/"
fe = "frontend-webapp-0.5.0.war"
logfile = open("frontend-log", 'ab')
p = Popen(['java', '-jar', platformDir + fe], stdout=logfile, stderr=STDOUT, bufsize=1)
I have a custom input method and I have a python module to communicate with it. I'm trying to control the shell with it so everything from local stdout is printed on the remote device and everything sent from the remote device goes into local stdin, so that remote device can control the input given to the program, like if there was an input function inside the program the remote device can answer to that too (like in ssh).
I used python subprocess to control the stdin and stdout:
#! /usr/bin/python
from subprocess import Popen, PIPE
import thread
from mymodule import remote_read, remote_write
def talk2proc(dap):
while True:
try:
remote_write(dap.stdout.read())
incmd = remote_read()
dap.stdin.write(incmd)
except Exception as e:
print (e)
break
while True:
cmd = remote_read()
if cmd != 'quit':
p = Popen(['bash', '-c', '"%s"'%cmd], stdout=PIPE, stdin=PIPE, stderr=PIPE)
thread.start_new_thread(talk2proc, (p,))
p.wait()
else:
break
But it doesn't work, what should I do?
p.s.
is there a difference for windows?
I had this problem, I used this for STDIN
from subprocess import call
call(['some_app', 'param'], STDIN=open("a.txt", "rb"))
a.txt
:q
This I used for a git wrapper, this will enter the data line wise whenever there is an interrupt in some_app that is expecting and user input
There is a difference for Windows. This line won't work in Windows:
p = Popen(['bash', '-c', '"%s"'%cmd], stdout=PIPE, stdin=PIPE, stderr=PIPE)
because the equivalent of 'bash' is 'cmd.exe'.
I have a python script that calls a shell scrips, that in turn calls a .exe called iv4_console. I need to print the stdout of iv4_console for debugging purposes. I used this:
Python:
import sys
import subprocess
var="rW015005000000"
proc = subprocess.Popen(["c.sh", var], shell=True, stdout=subprocess.PIPE)
output = ''
for line in iter(proc.stdout.readline, ""):
print line
output += line
Shell:
start_dir=$PWD
release=$1
echo Release inside shell: $release
echo Directory: $start_dir
cd $start_dir
cd ../../iv_system4/ports/visualC12/Debug
echo Debug dir: $PWD
./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua ../../../../../logs/$release/VASP_DUN722_20160307_Krk_Krk_113048_092_1_$release.dvl &>../../../../FCW/ObjectDetectionTest/VASP_DUN722_20160307_Krk_Krk_113048_092_1_$release.xml
./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua ../../../../../logs/$release/VASP_FL140_20170104_C60_Checkout_afterIC_162557_001_$release.dvl &>../../../../FCW/ObjectDetectionTest/VASP_FL140_20170104_C60_Checkout_afterIC_162557_001_$release.xml
exit
But this didn't work, it prints nothing. What do you think?
See my comment, best approach (i.m.o) would be to just use python only.
However, in answer of your question, try:
import sys
import subprocess
var="rW015005000000"
proc = subprocess.Popen(["/bin/bash", "/full/path/to/c.sh"], stdout=subprocess.PIPE)
# Best to always avoid shell=True because of security vulnerabilities.
proc.wait() # To make sure the shell script does not continue running indefinitely in the background
output, errors = proc.communicate()
print(output.decode())
# Since subprocess.communicate() returns a bytes-string, you can use .decode() to print the actual output as a string.
You can use
import subprocess
subprocess.call(['./c.sh'])
to call the shell script in python file
or
import subprocess
import shlex
subprocess.call(shlex.split('./c.sh var'))
I am using Popen to call a shell script that is continuously writing its stdout and stderr to a log file. Is there any way to simultaneously output the log file continuously (to the screen), or alternatively, make the shell script write to both the log file and stdout at the same time?
I basically want to do something like this in Python:
cat file 2>&1 | tee -a logfile #"cat file" will be replaced with some script
Again, this pipes stderr/stdout together to tee, which writes it both to stdout and my logfile.
I know how to write stdout and stderr to a logfile in Python. Where I'm stuck is how to duplicate these back to the screen:
subprocess.Popen("cat file", shell=True, stdout=logfile, stderr=logfile)
Of course, I could just do something like this, but is there any way to do this without tee and shell file descriptor redirection?:
subprocess.Popen("cat file 2>&1 | tee -a logfile", shell=True)
You can use a pipe to read the data from the program's stdout and write it to all the places you want:
import sys
import subprocess
logfile = open('logfile', 'w')
proc=subprocess.Popen(['cat', 'file'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in proc.stdout:
sys.stdout.write(line)
logfile.write(line)
proc.wait()
UPDATE
In python 3, the universal_newlines parameter controls how pipes are used. If False, pipe reads return bytes objects and may need to be decoded (e.g., line.decode('utf-8')) to get a string. If True, python does the decode for you
Changed in version 3.3: When universal_newlines is True, the class uses the encoding locale.getpreferredencoding(False) instead of locale.getpreferredencoding(). See the io.TextIOWrapper class for more information on this change.
To emulate: subprocess.call("command 2>&1 | tee -a logfile", shell=True) without invoking the tee command:
#!/usr/bin/env python2
from subprocess import Popen, PIPE, STDOUT
p = Popen("command", stdout=PIPE, stderr=STDOUT, bufsize=1)
with p.stdout, open('logfile', 'ab') as file:
for line in iter(p.stdout.readline, b''):
print line, #NOTE: the comma prevents duplicate newlines (softspace hack)
file.write(line)
p.wait()
To fix possible buffering issues (if the output is delayed), see links in Python: read streaming input from subprocess.communicate().
Here's Python 3 version:
#!/usr/bin/env python3
import sys
from subprocess import Popen, PIPE, STDOUT
with Popen("command", stdout=PIPE, stderr=STDOUT, bufsize=1) as p, \
open('logfile', 'ab') as file:
for line in p.stdout: # b'\n'-separated lines
sys.stdout.buffer.write(line) # pass bytes as is
file.write(line)
Write to terminal byte by byte for interactive applications
This method write any bytes it gets to stdout immediately, which more closely simulates the behavior of tee, especially for interactive applications.
main.py
#!/usr/bin/env python3
import os
import subprocess
import sys
with subprocess.Popen(sys.argv[1:], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as proc, \
open('logfile.txt', 'bw') as logfile:
while True:
byte = proc.stdout.read(1)
if byte:
sys.stdout.buffer.write(byte)
sys.stdout.flush()
logfile.write(byte)
# logfile.flush()
else:
break
exit_status = proc.returncode
sleep.py
#!/usr/bin/env python3
import sys
import time
for i in range(10):
print(i)
sys.stdout.flush()
time.sleep(1)
First we can do a non-interactive sanity check:
./main.py ./sleep.py
And we see it counting to stdout on real time.
Next, for an interactive test, you can run:
./main.py bash
Then the characters you type appear immediately on the terminal as you type them, which is very important for interactive applications. This is what happens when you run:
bash | tee logfile.txt
Also, if you want the output to show on the ouptut file immediately, then you can also add a:
logfile.flush()
but tee does not do this, and I'm afraid it would kill performance. You can test this out easily with:
tail -f logfile.txt
Related question: live output from subprocess command
Tested on Ubuntu 18.04, Python 3.6.7.