I'm making a shell with python. So far I have gotten cd to work (not pretty I know, but it's all I need for now). When I su root (for example) I get a root shell, but I can't capture the output I receive after running a command. However the shell does accept my commands, as when I type exit it exits. Is there a way to capture the output of a 'new' shell?
import os, subprocess
while True:
command = input("$ ")
if len(command.split(" ")) >= 2:
print(command.split(" ")[0]) #This line is for debugging
if command.split(" ")[0] == "cd" or command.split(" ")[1] == "cd":
os.chdir(command.split(" ")[command.split(" ").index("cd") + 1])
continue
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE, universal_newlines=True)
output, error = process.communicate()
print(output.strip("\n"))
EDIT: To make my request a bit more precise, I'd like a way to authenticate as another user from a python script, basically catching the authentication, doing it in the background and then starting a new subprocess.
You really need to understand how subprocess.Popen works. This command executes a new sub-process (on a Unix machine, calls fork and then exec). The new sub-process is a separate process. Your code just calls communicate once and then discards of it.
If you just create a new shell by calling subprocess.Popen and then running su <user> inside of it, the shell will be closed right after that and the next time, you'll be running the command using the same (original) user again.
What you want is probably to create a single subprocess at the beginning of your application and then be a sort of a proxy between the user and the underlying process, and then just keep writing to its stdin and reading from stdout.
Here's an example:
import os, subprocess
process = subprocess.Popen(["bash"], stdin=subprocess.PIPE,
stdout=subprocess.PIPE, universal_newlines=True)
while True:
command = input("$ ")
process.stdin.write(command + "\n")
process.stdin.flush()
output = process.stdout.readline()
print(output.strip("\n"))
(I removed the cd command parsing bit because it wasn't constructive to understanding the solution here, but you can definitely add specific handlers for specific inputs that wrap the underlying shell)
Related
Is there any way to use Popen with interactive commands? I mean nslookup, ftp, powershell... I read the whole subprocess documentation several times but I can't find the way.
What I have (removing the parts of the project which aren't of interest here) is:
from subprocess import call, PIPE, Popen
command = raw_input('>>> ')
command = command.split(' ')
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
execution = process.stdout.read()
error = process.stderr.read()
output = execution + error
process.stderr.close()
process.stdout.close()
print(output)
Basically, when I try to print the output with a command like dir, the output is a string, so I can work with the .read() on it. But when I try to use nslookup for example, the output isn't a string, so it can't be read, and the script enters in a deadlock.
I know that I can invoke nslookup in non-interactive mode, but that's not the point. I want to remove all the chances of a deadlock, and make it works with every command you can run in a normal cmd.
The real way the project works is through sockets, so the raw_input is a s.recv() and the output is sending back the output, but I have simplified it to focus on the problem.
I would like to run an interactive program using python's subprocess. This interactive program starts and runs for sometime. I would like to read the stdout and process it. After running for sometime, it waits with the prompt for user input. This prompt for instance can be "tool_name> ". I wanted to detect this situation and source another script. Here is some basic code, I could come up with. Would be great, if someone can comment further on if this is the correct way to do it.
import subprocess
p = subprocess.Popen(['cli_program' 'a.txt'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
while True:
line = p.stdout.readline()
if not line: #this is true only when application exits
break
elif 'tool_name> ' in line: #wanted to detect the wait state
p.stdin.write('source b.txt\n')
p.stdin.flush()
else:
print line
Here, "cli_program" is my interactive program which will start executing the script a.txt and waits with the prompt "tool_name> ".
I am working on executing the shell script from Python and so far it is working fine. But I am stuck on one thing.
In my Unix machine I am executing one command in the background by using & like this. This command will start my app server -
david#machineA:/opt/kml$ /opt/kml/bin/kml_http --config=/opt/kml/config/httpd.conf.dev &
Now I need to execute the same thing from my Python script but as soon as it execute my command it never goes to else block and never prints out execute_steps::Successful, it just hangs over there.
proc = subprocess.Popen("/opt/kml/bin/kml_http --config=/opt/kml/config/httpd.conf.dev &", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, executable='/bin/bash')
if proc.returncode != 0:
logger.error("execute_steps::Errors while executing the shell script: %s" % stderr)
sleep(0.05) # delay for 50 ms
else:
logger.info("execute_steps::Successful: %s" % stdout)
Anything wrong I am doing here? I want to print out execute_steps::Successful after executing the shell script in the background.
All other command works fine but only the command which I am trying to run in background doesn't work fine.
There's a couple things going on here.
First, you're launching a shell in the background, and then telling that shell to run the program in the background. I don't know why you think you need both, but let's ignore that for now. In fact, by adding executable='/bin/bash' on top of shell=True, you're actually trying to run a shell to run a shell to run the program in the background, although that doesn't actually quite work.*
Second, you're using PIPE for the process's output and error, but then not reading them. This can cause the child to deadlock. If you don't want the output, use DEVNULL, not PIPE. If you want the output to process yourself, use proc.communicate().**, or use a higher-level function like check_output. If you just want it to intermingle with your own output, just leave those arguments off.
* If you're using the shell because kml_http is a non-executable script that has to be run by /bin/bash, then don't use shell=True for that, or executable, just make make /bin/bash the first argument in the command line, and /opt/kml/bin/kml_http the second. But this doesn't seem likely; why would you install something non-executable into a bin directory?
** Or you can read it explicitly from proc.stdout and proc.stderr, but that gets more complicated.
At any rate, the whole point of executing something in the background is that it keeps running in the background, and your script keeps running in the foreground. So, you're checking its returncode before it's finished, and then moving on to whatever's next in your code, and never coming back again.
It seems like you want to wait for it to be finished. In that case, don't run it in the background—use proc.wait, or just use subprocess.call() instead of creating a Popen object. And don't use & either, of course. While we're at it, don't use the shell, either:
retcode = subprocess.call(["/opt/kml/bin/kml_http",
"--config=/opt/kml/config/httpd.conf.dev"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if retcode != 0:
# etc.
Now, you won't get to that if statement until kml_http finishes running.
If you want to wait for it to be finished, but at the same time keep doing other stuff, then you're trying to do two things at once in your program, which means you need a thread to do the waiting:
def run_kml_http():
retcode = subprocess.call(["/opt/kml/bin/kml_http",
"--config=/opt/kml/config/httpd.conf.dev"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
if retcode != 0:
# etc.
t = threading.Thread(target=run_kml_http)
t.start()
# Now you can do other stuff in the main thread, and the background thread will
# wait around until kml_http is finished and execute the `if` statement whenever
# that happens
You're using stderr=PIPE, stdout=PIPE which means that rather than letting the stdin and stdout of the child process be forwarded to the current process' standard output and error streams, they are being redirected to a pipe which you must read from in your python process (via proc.stdout and proc.stderr.
To "background" a process, simply omit the usage of PIPE:
#!/usr/bin/python
from subprocess import Popen
from time import sleep
proc = Popen(
['/bin/bash', '-c', 'for i in {0..10}; do echo "BASH: $i"; sleep 1; done'])
for x in range(10):
print "PYTHON: {0}".format(x)
sleep(1)
proc.wait()
which will show the process being "backgrounded".
I'm trying to run a Powershell subprocess from Python. I need to send Powershell code from Python to the child process. I've got this far:
import subprocess
import time
args = ["powershell", "-NoProfile", "-InputFormat None", "-NonInteractive"]
startTime = time.time()
process = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.stdin.write("Write-Host 'FINISHED';".encode("utf-8"))
result = ''
while 'FINISHED' not in result:
result += process.stdout.read(32).decode('utf-8')
if time.time() > startTime + 5:
raise TimeoutError(result)
print(result)
This times out, because nothing ever gets written to stdout. I think the Write-Host cmdlet never gets executed. Even the simple bash/Cygwin code echo "Write-Host 'FINISHED';" | powershell doesn't seem to do the job.
For comparison, sending the code block using the -Command flag works correctly.
How can I convince Powershell to run the code which I'm sending to stdin?
There a couple of things you can consider:
Invoke PowerShell in a mode where you provide it with a script file which it should execute. Write this script file prior to calling the subprocess. Use the -File <FilePath> parameter for PowerShell (cf. the docs)
If you really want to go with the stdin technique, you might be missing a newline character after the command. If this does not help, you might need to send another control character that tells PowerShell that input EOF is reached. You definitely need to consult the PowerShell docs for finding out how to 'terminate' commands on stdin. One thing you definitely need is the -Command - arguments: The value of Command can be "-", a string. or a script block. If the value of Command is "-", the command text is read from standard input. You may also want to look at this little hack: https://stackoverflow.com/a/13877874/145400
If you only want to execute one command, you can simplify your code by using out, err = subprocess.communicate(in)
I had trouble with a similar task, but I was able to solve it.
First my example code:
import subprocess
args = ["powershell.exe", "-Command", r"-"]
process = subprocess.Popen(args, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
process.stdin.write(b"$data = Get-ChildItem C:\\temp\r\n")
process.stdin.write(b"Write-Host 'Finished 1st command'\r\n")
process.stdin.write(b"$data | Export-Clixml -Path c:\\temp\state.xml\r\n")
process.stdin.write(b"Write-Host 'Finished 2nd command'\r\n")
output = process.communicate()[0]
print(output.decode("utf-8"))
print("done")
The main issue was the correct argument list args. It is required to start the powershell with the -Command-flag, followed by "-" as indicated by Jan-Philipp.
Another mystery was the end-of-line character that is required to get the stuff executed. \r\n works quite well.
Getting the output of the Powershell is still an issue. But if you don't care about realtime, you can collect the output after finishing all executions by calling
output = process.communicate()[0]
However, the active Powershell will be terminated afterwards.
I'm writing a simple wrapper over python debugger (pdb) and I need to parse pdb output. But I have a problem reading text from process pipe.
Example of my code:
import subprocess, threading, time
def readProcessOutput(process):
while not process.poll():
print(process.stdout.readline())
process = subprocess.Popen('python -m pdb script.py', shell=True, universal_newlines=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
read_thread = threading.Thread(target=readProcessOutput, args=(process,))
read_thread.start()
while True:
time.sleep(0.5)
When i execute given command (python -m pdb script.py) in OS shell I get results like this:
> c:\develop\script.py(1)<module>()
-> print('hello, world!')
(Pdb)
But when i run my script i get only two lines, but can't get pdb prompt. Writing commands to stdin after this has no effect. So my question is:
why I cannot read third line? How can I avoid this problem and get correct output?
Platform: Windows XP, Python 3.3
The third line can not be read by readline() because it is not terminated yet by the end of line. You see usually the cursor after "(pdb) " until you write anything + enter.
The communication to processes that have some prompt is usually more complicated. It proved to me to write also an independent thread for data writer first for easier testing the communication in order to be sure that the main thread never freezes if too much is tried to be written or read. Then it can be simplified again.