I am new to the subprocess module, and after reading the Python documents among many other websites (including Stack Overflow), I am struggling to find simplified examples of .Popen, .communicate, and other such useful classes. Often, the examples never use each class in succession, only on their own. Also, many examples are Linux-based, such as the use of ["ls", "-l"], which makes it quite difficult to understand for Windows users.
After several hours playing with this module, I have encountered several issues, which could be best illustrated through the method of opening and communicating with a simple .exe command line program.
For example, say that the program is called "numbers.exe", and that it asks the following questions:
>>> Question 1) Choose 1, 2 or 3
>>> Question 2) Choose 4, 5 or 6
>>> You have answered #(Q1) and #(Q2)
>>> *Use these values in an iterative sequence displaying each stage in the iteration*
I then want to manipulate this program automatically, i.e. I want python to input 2 and 6 without me having to do anything, but still print the questions. I then want to be able to view the iteration in python.
The first consideration here is that I could use:
from subprocess import Popen, PIPE
numprog = subprocess.call('numbers.exe')
print(numprog.communicate())
However, this just opens the program, and I would still have to enter 2 and 6 myself. To automate the process, I believe I must use Popen, alongside stdin, stdout and stderr. This is where I encounter issues. I understand I must use Popen to begin communicating with the input (stdin), output (stdout) and error (stderr) pipes:
from subprocess import Popen, PIPE
numcomms = Popen('numbers.exe', stdout=PIPE, stdin=PIPE, stderr=PIPE)
I am unsure what to do from here. Using numcomms.stdout.read() causes the program to just linger, and using numcomms.stdin.write(2) throws out an error that int values cannot be used. The numprog.communicate class seems to require you to enter the values yourself.
From what I can see, the pseudo code would be along the lines:
>>> Open numbers.exe stdin, stdout and stderr pipes using Popen
>>> Print first question using stdout
>>> Enter "2" using stdin
>>> Print second question using stdout
>>> Enter "6" using stdin
>>> Receive a string saying "You have answered 2 and 6" using stdout
>>> Display the results of each stage of the iteration using stdout
How would I go about writing this?
Really appreciate the help, thank you!
Edit: Edited the question to describe the iterative sequence issue. Michael has suggested a good resolution to the inputting issue, however I am having trouble printing the results of the iteration.
The problem with subprocess.stdin.write is probably that you need to provide a string instead of an integer, as Steve Barnes pointed out correcly.
However, for your simple case their might be an easier solution. The communicate method has an optional parameter for input, so this should work:
from subprocess import Popen, PIPE
numcomms = Popen('numbers.exe', stdout=PIPE, stdin=PIPE, stderr=PIPE)
out, err = numcomms.communicate("2\n6\n")
The output of the program should be in out afterwards, and can easily be splitted using out.splitlines().
I think that your problem with the subprocess.stdin.write is that the program will be expecting a string probably terminated with a newline i.e. '2\n' rather than an int.
Related
Well the first problem I run into is that I have no idea how to respond to a command prompt.
bat_location = "F:/SteamLibrary/steamapps/common/Terraria"
os.chdir(bat_location)
os.system("TerrariaServer.exe -steam -lobby friends -config serverconfig.txt")
all of this works, but then when I want to respond to the command prompt which asks me which world I want to run ( the worlds are indexed by numbers from 1 - n (amount of worlds)) I dont know how to respond to it.
I've looked all over google but the code doesnt seem to work.
So basically what I need is when a cmd asks me for example :
Choose World:
I want to automatically respond with the number 10.
os.system("10")
this doesnt seem to do anything, I've also tried a lot with subprocesses but im clearly lost.
Any help is apriciated!
EDIT NR.1 :
Welp, now I've tried this :
bat_location = r'F:\SteamLibrary\steamapps\common\Terraria'
with Popen('TerrariaServer.exe -steam -lobby friends -config serverconfig.txt',
cwd=f'{bat_location}', stdin=PIPE, shell=True) as proc:
proc.stdin.write(b'10\n')
and all it does, im guessing by the response, it just loops around and around.
EDIT NR.2 :
I WILL CLOSE THIS AND START A NEW THREAD, SINCE MY PROBLEM COMPLETELY DERIVED FROM THE ORIGINAL.
From your last few comments, I realized the problem you were having with Popen. When you pass stdout=PIPE and stderr=PIPE, the outputs of the process are captured by the pipes, so you won't see them unless you read from the pipes.
So here's a simple example that you should be able to work with:
import subprocess
from subprocess import PIPE
from textwrap import dedent
with open('tmp.py', 'w') as f:
f.write(dedent("""
print(input())
print(input())
"""))
with subprocess.Popen(['python3', 'tmp.py'], stdin=PIPE) as proc:
proc.stdin.write(b'Hello, world!\n') # write to the process' input
proc.stdin.write(b'Good bye, world!\n') # write to the process' input
If you want to read the data from the function in Python, you can use stdout=PIPE, then use proc.stdout.read and the like, but you may have to be careful about how you get data from the blocking read functions.
I currently try to do a Buffer Overflow attack to a simple C Program.
This Program takes 2 inputs via C's scanf function. The First input is secure, the second is not. So I found my shellcode, the padding and the new return adress. It should spawn a simple shell. So i tried it with the following:
#!/usr/bin/env python
...
p = subprocess.Popen(["/home/user/Desktop/A3/1/exploitme"], shell=True, stdout = subprocess.PIPE, stdin = subprocess.PIPE, stderr=subprocess.PIPE)
p.stdin.write('A\n')
p.stdin.write(OVERFLOWCODE)
stdout, stderr = p.communicate()
print(stdout)
But i don't see the //bin/sh terminal.
However I think the problem is not the Code i use (it should work fine), the Problem seems to be that like the command itself says that it just creates a subprocess where just the programm can communicate with, and not me.
So my question is, how can I make it possible to run the programm, enter two strings to the scanf's and then take control over the Program? Something like spawn a new Terminal and pipe the two commands in there.
The Problem is I can't just enter the Code manually because there is something like ASLR (it creates a buffer with size of srand of unix time in seconds), so I must calculate the return adress every second new.
Any Ideas to make this work?
I have a python 3 script that accepts input from the user, this input is piped into a subprocess, where a shell has been spawned. Originally I was going to put this code together with a socket, to be able to make my own non-serious remote administration tool, however this proves too hard for my level currently. Code:
import subprocess
p1 = subprocess.Popen(["/bin/sh"], stderr = subprocess.PIPE, stdin = subprocess.PIPE, stdout = subprocess.PIPE, encoding = "utf-8")
command = input("Command: ")
p1.stdin.write(command)
p1.stdout.read()
Problem: Nothing gets printed out. I have searched endless hours
online for a reason, over multiple days, but all of them don't seem to
work, and/or advise using communicate() which is something I do not
want to do. When thinking ahead if I am able to implement this with a
socket, I can't have the process closing after each command. I have
also tried flushes everywhere, before write, inbetween the read, after
the read, pretty much everywhere you can think of. It should be simple
enough, without me having to look deep into the io module or the rules
of buffering (too late now). I have been struggling with this for days
on end.
You have a couple of issues, I will try to draw the path, I hope I am not misleading you.
First of all, disclaimer: executing user input is inherently unsafe. If that's not an issue, let's keep on your problem.
I would start by doing
p1 = subprocess.Popen(["/bin/sh"], stderr=subprocess.PIPE, stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True, bufsize=1)
Having a line-buffered PIPE is a good idea, in this scenario.
Then, remember that you have to press enter after the command, so:
command = input("Command: ") + "\n"
The write is correct,
p1.stdin.write(command)
But the, the read is VERY dangerous and you will trivially become deadlocked. Because the stdout is open (without an EOF or anything like that) you should read conservatively, and still, you will have problems on that.
A first idea would be to read lines:
p1.stdout.readline()
But, if you don't know how many lines you can read... then that's a problem. In fact, that specific problem has already been asked.
If you want to experiment, just open an interactive python interpreter, send a "ls\n" command and perform readline(). It will work, until you read the last line, and then... it will wait forever. That is bad behaviour. So you will need to solve that.
I need to run a external exe file inside a python script. I need two things out of this.
Get whatever the exe outputs to the stdout (stderr).
exe stops executing only after I press the enter Key. I can't change this behavior. I need the script the pass the enter Key input after it gets the output from the previous step.
This is what I have done so far and I am not sure how to go after this.
import subprocess
first = subprocess.Popen(["myexe.exe"],shell=True,stdout=subprocess.PIPE)
from subprocess import Popen, PIPE, STDOUT
first = Popen(['myexe.exe'], stdout=PIPE, stderr=STDOUT, stdin=PIPE)
while first.poll() is None:
data = first.stdout.read()
if b'press enter to' in data:
first.stdin.write(b'\n')
first.stdin.close()
first.stdout.close()
This pipes stdin as well, do not forget to close your open file handles (stdin and stdout are also file handles in a sense).
Also avoid shell=True if at all possible, I use it a lot my self but best practices say you shouldn't.
I assumed python 3 here and stdin and stdout assumes bytes data as input and output.
first.poll() will poll for a exit code of your exe, if none is given it means it's still running.
Some other tips
one tedious thing to do can be to pass arguments to Popen, one neat thing to do is:
import shlex
Popen(shlex.split(cmd_str), shell=False)
It preserves space separated inputs with quotes around them, for instance python myscript.py debug "pass this parameter somewhere" would result in three parameters from sys.argv, ['myscript.py', 'debug', 'pass this parameter somewhere'] - might be useful in the future when working with Popen
Another thing that would be good is to check if there's output in stdout before reading from it, otherwise it might hang the application. To do this you could use select.
Or you could use pexpect which is often used with SSH since it lives in another user space than your application when it asks for input, you need to either fork your exe manually and read from that specific pid with os.read() or use pexpect.
I need to open an R script and supply it with input formulated by a separate python script. The subprocess module seems to be a good way to do this.
I have encountered some puzzling results though, namely that I can apparently write once and only once via p.stdin. Here is what I have so far:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['r --no-save'],stdin=PIPE,stdout=PIPE,stderr=PIPE,shell=True)
p.stdin.write("source('myrscript.R')\n")
p.stdin.write('myfirstinput')
What happens when I run this code is that the first instance of stdin.write() performs as expected (and opens my R script), but the second line does nothing, and the subprocess (really, the R script) exits with an error, indicating that the subprocessed received no input where input was expected and therefore terminated.
N.B. - In a perfect world, I would just interact directly through R, but this particular script requires complex inputs that cannot be entered directly for practical purposes. Also, rpy / rpy2 is not an option, because end-users of this script will not necessarily have access to that module or its dependencies. rscript is also not an option (for many reasons, but mainly because of variability in the end-users R configurations).
Finally, p.communicate is not an option, because apparently that will close the process after writing and I need to keep it open.
Thanks in advance
What you need is to call .communicate():
from subprocess import Popen, PIPE, STDOUT
p = Popen(
['r', '--nosave'],
stdin=PIPE,
stdout=PIPE,
stderr=PIPE)
p.stdin.write("source('myrscript.R')\n")
p.stdin.write('myfirstinput\n')
p.stdin.write('q\n')
stdout, stderr = p.communicate()
print '---STDOUT---'
print stdout
print '---STDERR---'
print stderr
print '---'
Discussion
I don't use the shell=True and it seems working with my fake R script since I don't have R install in my system. You might or might not need it.
I prefer breaking the command line up into a list of string as shown, but a single string such as r --nosave will work as well; just don't do them both at the same time.
Don't forget that stdin.write() does not write the new line character \n, you have to supply that yourself.
Update
My first attempt was off the mark, I hope this second attempt gets closer. As J.F. Sebastian suggested, you might want to use pexpect:
import pexpect
import sys
if __name__ == '__main__':
prompt = '> ' # Don't know what the R prompt looks like
lines = ['one', 'two', 'three']
r = pexpect.spawn('r --no-save', logfile=sys.stdout)
for line in lines:
r.expect(prompt)
r.sendline(line)
# If you want to interact with your script, use these two lines
# Otherwise, comment them out
r.logfile = None # Turn off logging to sys.stdout
r.interact()
Discussion
You might need to install pexpect. I did it with pip install pexpect
If you don't want to interact with the system, comment out the last two line, but make sure to send some signal for the R script to exit.
spawn() returns a spawn object, see doc here.