I am trying to execute a linux command through Python.
Here, for one of the command (cryptsetup luksChangeKey) I need to pass two keys [current key and new key] when prompted by the command through STDIN.
I tried using communicate() for the same and not able to pass both the keys.
Is there any other option available in python for the above scenario ?
Sample Code:
import subprocess
cmd = 'cryptsetup --batch-mode --key-file - luksChangeKey
/dev/multiplelv_pool_VG_13341/lv4'
process = subprocess.Popen(shlex.split(cmd), stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.stdin.write("old123\nnew123\n")
process.communicate()
process.stdin.close()
If we manually execute the cryptsetup command, then it will prompt for old key and then the new key to be provided.
I am trying to simulate the same through the Python code.
Here current password is old123 and new password is new123.
My expectation of using '\n' in stdin.write was to split the password into two lines. However, POpen is taking the entire
line as current password and resulting in error
I am using Python 2.7
This work for me:
other.py
a = input()
b = input()
print(a, b)
program.py
import subprocess
cmd = r'python other.py'
process = subprocess.Popen(cmd, stdin=subprocess.PIPE)
process.communicate("input1\ninput2".encode('utf-8'))
process.stdin.close()
# print: >>> input1 input2
The subprocess documentation warns about the use of stdin.write.
Also, I don't know how your code doesn't throw an error, since you are passing a string to stdin and not a bytes-like object? This is what happens for me with your code:
TypeError: a bytes-like object is required, not 'str'
This is not a Python problem but a cryptsetup limitation. It is explicit in the manpage (emphasize mine):
--key-file, -d<br/>
use file as key material.
... If the key file is "-", stdin will be used. With the "-" key file reading will not stop when new line character is detected.
and later:
Notes on Password Processing
...
From stdin: Reading will continue until EOF
Workarounds:
According to this post on SuperUser, it used to be possible to pass both the old and new passwords in the same keyfile, but it would no longer be possible. A trick would be to use stdin for the old key, and stdout for the new one. Python will refuse to write on a process.stdout file because it is only opened for reading, but it is still possible to write on it at the fileno level. Code would become:
import subprocess
cmd = 'cryptsetup --batch-mode --key-file - luksChangeKey
/dev/multiplelv_pool_VG_13341/lv4 /dev/fd/1' # read the new key from /dev/fd/1
process = subprocess.Popen(shlex.split(cmd), stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
os.write(process.stdout.fileno(), 'new123')
process.communicate('old123')
Beware: untested because I currently have no Linux system with LUKS.
A way to trick cryptsetup would be to use the pexpect-module (needs to be installed separately). This runs a subprocess faking a controlling terminal. It then allows to wait for prompts and issue user input when prompted. This is e.g. a way to control SSH and enter a password instead of having to resort to public keys etc.
Related
Is there any way to use Popen with interactive commands? I mean nslookup, ftp, powershell... I read the whole subprocess documentation several times but I can't find the way.
What I have (removing the parts of the project which aren't of interest here) is:
from subprocess import call, PIPE, Popen
command = raw_input('>>> ')
command = command.split(' ')
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
execution = process.stdout.read()
error = process.stderr.read()
output = execution + error
process.stderr.close()
process.stdout.close()
print(output)
Basically, when I try to print the output with a command like dir, the output is a string, so I can work with the .read() on it. But when I try to use nslookup for example, the output isn't a string, so it can't be read, and the script enters in a deadlock.
I know that I can invoke nslookup in non-interactive mode, but that's not the point. I want to remove all the chances of a deadlock, and make it works with every command you can run in a normal cmd.
The real way the project works is through sockets, so the raw_input is a s.recv() and the output is sending back the output, but I have simplified it to focus on the problem.
I've managed to get the cmd being opened by python. However, using runas administrator comes with a password check before cmd.exe is executed.
I'm using this to open cmd...
import subprocess
subprocess.call(["runas", "/user:Administrator", "cmd.exe"])
I'm looking for a way to automatically enter the password into the runas.exe prompt which opens when i run the code. Say if i were to create var = "test" and add it after import subprocess how would i make it so that this variable is passed to and seen as an input to the runas.exe?
The solution would require only python modules which are in version 3.4 or higher.
Update
I have found some code which appears to input straight into runas.exe. However, the apparent input is \x00\r\n when in the code the input is supposed to be test I am fairly certain that if i can get the input to be test then the code will be successful.
The code is as follows :
import subprocess
args = ['runas', '/user:Administrator', 'cmd.exe']
proc = subprocess.Popen(args,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
proc.stdin.write(b'test\n')
proc.stdin.flush()
stdout, stderr = proc.communicate()
print (stdout)
print (stderr)
Although not an answer to your question, this can be a solution to your problem. Use psexec instead of runas. You can run it like this:
psexec -u user -p password cmd
(or run it from Python using subprocess.Popen or something else)
This piece of code actually works (tested on a Windows 2008 server). I've used it to call runas for a different user and pass his password. A new command prompt opened with new user context, without needing to enter password.
Note that you have to install pywin32 to have access to the win32 API.
The idea is:
to Popen the runas command, without any input redirection, redirecting output
read char by char until we encounter ":" (last char of the password prompt).
send key events to the console using win32 packages, with the final \r to end the password input.
(adapted from this code):
import win32console, win32con, time
import subprocess
username = "me"
domain = "my_domain"
password ="xxx"
free_console=True
try:
win32console.AllocConsole()
except win32console.error as exc:
if exc.winerror!=5:
raise
## only free console if one was created successfully
free_console=False
stdin=win32console.GetStdHandle(win32console.STD_INPUT_HANDLE)
p = subprocess.Popen(["runas",r"/user:{}\{}".format(domain,username),"cmd.exe"],stdout=subprocess.PIPE)
while True:
if p.stdout.read(1)==b":":
for c in "{}\r".format(password): # end by CR to send "RETURN"
## write some records to the input queue
x=win32console.PyINPUT_RECORDType(win32console.KEY_EVENT)
x.Char=unicode(c) # remove unicode for python 3
x.KeyDown=True
x.RepeatCount=1
x.VirtualKeyCode=0x0
x.ControlKeyState=win32con.SHIFT_PRESSED
stdin.WriteConsoleInput([x])
p.wait()
break
I'm trying to write a Python script that starts a subprocess, and writes to the subprocess stdin. I'd also like to be able to determine an action to be taken if the subprocess crashes.
The process I'm trying to start is a program called nuke which has its own built-in version of Python which I'd like to be able to submit commands to, and then tell it to quit after the commands execute. So far I've worked out that if I start Python on the command prompt like and then start nuke as a subprocess then I can type in commands to nuke, but I'd like to be able to put this all in a script so that the master Python program can start nuke and then write to its standard input (and thus into its built-in version of Python) and tell it to do snazzy things, so I wrote a script that starts nuke like this:
subprocess.call(["C:/Program Files/Nuke6.3v5/Nuke6.3", "-t", "E:/NukeTest/test.nk"])
Then nothing happens because nuke is waiting for user input. How would I now write to standard input?
I'm doing this because I'm running a plugin with nuke that causes it to crash intermittently when rendering multiple frames. So I'd like this script to be able to start nuke, tell it to do something and then if it crashes, try again. So if there is a way to catch a crash and still be OK then that'd be great.
It might be better to use communicate:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['myapp'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
stdout_data = p.communicate(input='data_to_write')[0]
"Better", because of this warning:
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
To clarify some points:
As jro has mentioned, the right way is to use subprocess.communicate.
Yet, when feeding the stdin using subprocess.communicate with input, you need to initiate the subprocess with stdin=subprocess.PIPE according to the docs.
Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.
Also qed has mentioned in the comments that for Python 3.4 you need to encode the string, meaning you need to pass Bytes to the input rather than a string. This is not entirely true. According to the docs, if the streams were opened in text mode, the input should be a string (source is the same page).
If streams were opened in text mode, input must be a string. Otherwise, it must be bytes.
So, if the streams were not opened explicitly in text mode, then something like below should work:
import subprocess
command = ['myapp', '--arg1', 'value_for_arg1']
p = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.communicate(input='some data'.encode())[0]
I've left the stderr value above deliberately as STDOUT as an example.
That being said, sometimes you might want the output of another process rather than building it up from scratch. Let's say you want to run the equivalent of echo -n 'CATCH\nme' | grep -i catch | wc -m. This should normally return the number characters in 'CATCH' plus a newline character, which results in 6. The point of the echo here is to feed the CATCH\nme data to grep. So we can feed the data to grep with stdin in the Python subprocess chain as a variable, and then pass the stdout as a PIPE to the wc process' stdin (in the meantime, get rid of the extra newline character):
import subprocess
what_to_catch = 'catch'
what_to_feed = 'CATCH\nme'
# We create the first subprocess, note that we need stdin=PIPE and stdout=PIPE
p1 = subprocess.Popen(['grep', '-i', what_to_catch], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# We immediately run the first subprocess and get the result
# Note that we encode the data, otherwise we'd get a TypeError
p1_out = p1.communicate(input=what_to_feed.encode())[0]
# Well the result includes an '\n' at the end,
# if we want to get rid of it in a VERY hacky way
p1_out = p1_out.decode().strip().encode()
# We create the second subprocess, note that we need stdin=PIPE
p2 = subprocess.Popen(['wc', '-m'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# We run the second subprocess feeding it with the first subprocess' output.
# We decode the output to convert to a string
# We still have a '\n', so we strip that out
output = p2.communicate(input=p1_out)[0].decode().strip()
This is somewhat different than the response here, where you pipe two processes directly without adding data directly in Python.
Hope that helps someone out.
Since subprocess 3.5, there is the subprocess.run() function, which provides a convenient way to initialize and interact with Popen() objects. run() takes an optional input argument, through which you can pass things to stdin (like you would using Popen.communicate(), but all in one go).
Adapting jro's example to use run() would look like:
import subprocess
p = subprocess.run(['myapp'], input='data_to_write', capture_output=True, text=True)
After execution, p will be a CompletedProcess object. By setting capture_output to True, we make available a p.stdout attribute which gives us access to the output, if we care about it. text=True tells it to work with regular strings rather than bytes. If you want, you might also add the argument check=True to make it throw an error if the exit status (accessible regardless via p.returncode) isn't 0.
This is the "modern"/quick and easy way to do to this.
One can write data to the subprocess object on-the-fly, instead of collecting all the input in a string beforehand to pass through the communicate() method.
This example sends a list of animals names to the Unix utility sort, and sends the output to standard output.
import sys, subprocess
p = subprocess.Popen('sort', stdin=subprocess.PIPE, stdout=sys.stdout)
for v in ('dog','cat','mouse','cow','mule','chicken','bear','robin'):
p.stdin.write( v.encode() + b'\n' )
p.communicate()
Note that writing to the process is done via p.stdin.write(v.encode()). I tried using
print(v.encode(), file=p.stdin), but that failed with the message TypeError: a bytes-like object is required, not 'str'. I haven't figured out how to get print() to work with this.
You can provide a file-like object to the stdin argument of subprocess.call().
The documentation for the Popen object applies here.
To capture the output, you should instead use subprocess.check_output(), which takes similar arguments. From the documentation:
>>> subprocess.check_output(
... "ls non_existent_file; exit 0",
... stderr=subprocess.STDOUT,
... shell=True)
'ls: non_existent_file: No such file or directory\n'
I have a executable, called tsfoil2.exe, and I want to operate this .exe from my python environment.
I'm running Python 2.7.3, with Spyder 2.1.11 on Windows 7.
In order to operate the .exe, it requires some input, the default hard drive ('I:\'), a name for the outputfile ('test'), and a name for the input file ('SC20610.inp').
One of my colleagues advised me to use os.system, and supply this with a temporary input file, that contains all the arguments.
f = open('temp', 'w')
f.write('I:\ \n')
f.write('test \n')
f.write('SC20610.inp\n')
f.close()
I then supply this file with arguments to the .exe in the following way:
os.system("tsfoil2.exe < temp")
This all works, but the program requires a 'ENTER' to close. For some reason, the .exe is repeatedly asking to 'Press the ENTER key to exit'. Even, when I press the enter key in my Spyder-console, the program does not terminate.
Is there a way to give the 'ENTER' key as an interactive input to .exe?
I've tried to use the SendKeys class, but as the program does not terminate, it does not reach the line of code that contains the SendKeys command.
I've also tried to include it in the arguments-file, but this does not work either.
Furthermore I've also found out that it might be beneficial to switch to subprocesses command, as it might give me more command over the execution, but I haven't been able to run the executable with the input files.
Is it possible to provide the necessary 'ENTER' using os.system, or should I switch to subprocess, and if so, how can I construct a method similar to the os.system("tsfoil2.exe < temp") I'm using now.
I've tried this:
import subprocess as sub
f = open('temp', 'w')
f.write('I:\ \n')
f.write('test \n')
f.write('SC20610.inp\n')
f.close()
proc=sub.Popen(["tsfoil2.exe","temp"], shell=True)
and this
import subprocess as sub
p=sub.Popen('tsfoil2.exe')
p.communicate(input='I:' )
But, the program does not respond to the arguments given.
MWE:
import os
f = open('temp', 'w')
f.write('I:\ \n')
f.write('test \n')
f.write('SC20610.inp\n')
f.close()
os.system("tsfoil2.exe < temp")
Both the program can be found via http://www.dept.aoe.vt.edu/~mason/Mason_f/tsfoil2.exe, the input file can be found via http://www.dept.aoe.vt.edu/~mason/Mason_f/SC20610.inp.
I hope everything is clear, and you can help me out.
'Press the ENTER key to exit' means that the programs expects a newline.
I see no blank line at the end of the temp file. Also, you might have meant 'I:\\\n' -- you need to use '\\' in a Python string literal if you want \ in the output.
The question is what tsfoil2.exe considers a newline e.g., b'\r\n' or just b'\n' and where it expects to receive it: from stdin (getchar()) or directly from console (getch()).
Assuming that the program expects b'\r\n' from stdin on Windows:
import os
from subprocess import CalledProcessError, Popen, PIPE
cmd = "tsfoil2.exe"
input_data = os.linesep.join(['I:\\', 'test', 'SC20610.inp', os.linesep])
p = Popen(cmd, stdin=PIPE, bufsize=0)
p.communicate(input_data.encode('ascii'))
if p.returncode != 0:
raise CalledProcessError(p.returncode, cmd)
How it works
os.linesep == "\r\n" on Windows. "\n".join(["a", "b"]) == "a\nb".
Each process may have three standard streams: stdin, stdout, stderr. In Python, they are represented as sys.stdin, sys.stdout, sys.stderr. You can read input from stdin and write to stdout, stderr e.g., input function reads from stdin and print writes to stdout by default. stderr may be used to write error messages.
stdin=PIPE tells Popen to create a pipe between the parent process where it is called and the new child process ("tsfoil2.exe") and redirect the subprocess' stdin. p.communicate(data) writes data to p.stdin, closes it and waits for the child process to finish. p.returncode contains the exit status of the child process. Usually non-zero status means failure.
It emulates the shell pipeline without actually spawning the shell:
$ echo input data | tsfoil2.exe
If input is expected directly from console, you could try SendKeys module or its pure Python implementation SendKeys-ctypes:
from SendKeys import SendKeys
SendKeys(r"""
{LWIN}
{PAUSE .25}
r
C:\Path\To\tsfoil2.exe{ENTER}
{PAUSE 1}
I:\{ENTER}
{PAUSE 1}
test{ENTER}
{PAUSE 1}
SC20610.inp{ENTER}
{PAUSE 1}
{ENTER}
""")
I haven't tested it.
I'm trying to write a Python script that starts a subprocess, and writes to the subprocess stdin. I'd also like to be able to determine an action to be taken if the subprocess crashes.
The process I'm trying to start is a program called nuke which has its own built-in version of Python which I'd like to be able to submit commands to, and then tell it to quit after the commands execute. So far I've worked out that if I start Python on the command prompt like and then start nuke as a subprocess then I can type in commands to nuke, but I'd like to be able to put this all in a script so that the master Python program can start nuke and then write to its standard input (and thus into its built-in version of Python) and tell it to do snazzy things, so I wrote a script that starts nuke like this:
subprocess.call(["C:/Program Files/Nuke6.3v5/Nuke6.3", "-t", "E:/NukeTest/test.nk"])
Then nothing happens because nuke is waiting for user input. How would I now write to standard input?
I'm doing this because I'm running a plugin with nuke that causes it to crash intermittently when rendering multiple frames. So I'd like this script to be able to start nuke, tell it to do something and then if it crashes, try again. So if there is a way to catch a crash and still be OK then that'd be great.
It might be better to use communicate:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['myapp'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
stdout_data = p.communicate(input='data_to_write')[0]
"Better", because of this warning:
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
To clarify some points:
As jro has mentioned, the right way is to use subprocess.communicate.
Yet, when feeding the stdin using subprocess.communicate with input, you need to initiate the subprocess with stdin=subprocess.PIPE according to the docs.
Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.
Also qed has mentioned in the comments that for Python 3.4 you need to encode the string, meaning you need to pass Bytes to the input rather than a string. This is not entirely true. According to the docs, if the streams were opened in text mode, the input should be a string (source is the same page).
If streams were opened in text mode, input must be a string. Otherwise, it must be bytes.
So, if the streams were not opened explicitly in text mode, then something like below should work:
import subprocess
command = ['myapp', '--arg1', 'value_for_arg1']
p = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.communicate(input='some data'.encode())[0]
I've left the stderr value above deliberately as STDOUT as an example.
That being said, sometimes you might want the output of another process rather than building it up from scratch. Let's say you want to run the equivalent of echo -n 'CATCH\nme' | grep -i catch | wc -m. This should normally return the number characters in 'CATCH' plus a newline character, which results in 6. The point of the echo here is to feed the CATCH\nme data to grep. So we can feed the data to grep with stdin in the Python subprocess chain as a variable, and then pass the stdout as a PIPE to the wc process' stdin (in the meantime, get rid of the extra newline character):
import subprocess
what_to_catch = 'catch'
what_to_feed = 'CATCH\nme'
# We create the first subprocess, note that we need stdin=PIPE and stdout=PIPE
p1 = subprocess.Popen(['grep', '-i', what_to_catch], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# We immediately run the first subprocess and get the result
# Note that we encode the data, otherwise we'd get a TypeError
p1_out = p1.communicate(input=what_to_feed.encode())[0]
# Well the result includes an '\n' at the end,
# if we want to get rid of it in a VERY hacky way
p1_out = p1_out.decode().strip().encode()
# We create the second subprocess, note that we need stdin=PIPE
p2 = subprocess.Popen(['wc', '-m'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# We run the second subprocess feeding it with the first subprocess' output.
# We decode the output to convert to a string
# We still have a '\n', so we strip that out
output = p2.communicate(input=p1_out)[0].decode().strip()
This is somewhat different than the response here, where you pipe two processes directly without adding data directly in Python.
Hope that helps someone out.
Since subprocess 3.5, there is the subprocess.run() function, which provides a convenient way to initialize and interact with Popen() objects. run() takes an optional input argument, through which you can pass things to stdin (like you would using Popen.communicate(), but all in one go).
Adapting jro's example to use run() would look like:
import subprocess
p = subprocess.run(['myapp'], input='data_to_write', capture_output=True, text=True)
After execution, p will be a CompletedProcess object. By setting capture_output to True, we make available a p.stdout attribute which gives us access to the output, if we care about it. text=True tells it to work with regular strings rather than bytes. If you want, you might also add the argument check=True to make it throw an error if the exit status (accessible regardless via p.returncode) isn't 0.
This is the "modern"/quick and easy way to do to this.
One can write data to the subprocess object on-the-fly, instead of collecting all the input in a string beforehand to pass through the communicate() method.
This example sends a list of animals names to the Unix utility sort, and sends the output to standard output.
import sys, subprocess
p = subprocess.Popen('sort', stdin=subprocess.PIPE, stdout=sys.stdout)
for v in ('dog','cat','mouse','cow','mule','chicken','bear','robin'):
p.stdin.write( v.encode() + b'\n' )
p.communicate()
Note that writing to the process is done via p.stdin.write(v.encode()). I tried using
print(v.encode(), file=p.stdin), but that failed with the message TypeError: a bytes-like object is required, not 'str'. I haven't figured out how to get print() to work with this.
You can provide a file-like object to the stdin argument of subprocess.call().
The documentation for the Popen object applies here.
To capture the output, you should instead use subprocess.check_output(), which takes similar arguments. From the documentation:
>>> subprocess.check_output(
... "ls non_existent_file; exit 0",
... stderr=subprocess.STDOUT,
... shell=True)
'ls: non_existent_file: No such file or directory\n'