I created a class method (this will only run on Linux) that sends a list of commands to a remote computer over SSH and returns the output using subprocess.Popen:
def remoteConnection(self, list_of_remote_commands):
ssh = subprocess.Popen(["ssh", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0)
# send ssh commands to stdin
for command in list_of_remote_commands:
ssh.stdin.write(command + "\n")
ssh.stdin.close()
output_dict = {'stdin': list(ssh.stdin), 'stdout': list(ssh.stdout), 'stderr': list(ssh.stderr)}
return output_dict
Whilst I'm still getting to grips with the subprocess module I'd read quite a bit about Popen and no one ever mentioned closing it (SSH Connection with Python 3.0, Proper way to close all files after subprocess Popen and communicate, https://docs.python.org/2/library/subprocess.html) so I assumed that that wasn't a problem.
However when testing this out in ipython outside of a function I noticed that the variable ssh still seemed active. I tried closing ssh.stdin, ssh.stdout and ssh.stderr and even ssh.close(), ssh.terminate() and ssh.kill() but nothing seemed to close it. I thought perhaps it doesn't matter but my function will be called many times for months or even years so I don't want it to spawn a new process everytime it is run otherwise I'm going to quickly use up my maximum processes limit. So I use ssh.pid to find the PID and look it up using ps aux | grep PID and it's still there even after doing all of the above.
I also tried:
with subprocess.Popen(["ssh", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0) as shh:
instead of:
ssh = subprocess.Popen(["ssh", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0)
I also remember solving a similar problem a while back using ssh -T but even:
ssh = subprocess.Popen(["ssh", "-T", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0)
Didn't work.
I'm sure I would have found something about closing Popen if I needed to but then why is the process still open on my computer - can anyone help me understand what's going on here?
In your case, you have a deadlock here:
output_dict = {'stdin': list(ssh.stdin), 'stdout': list(ssh.stdout), 'stderr': list(ssh.stderr)}
Mostly because list(ssh.stdin) blocks forever: trying to read standard input of a process doesn't work (there's also an extra risk because you redirected both standard output & error to different pipes without using threading to consume them)
You mean to use ssh.communicate, passing the whole input as argument. Simply do:
command_input = "".join(["{}\n".format(x) for x in list_of_remote_commands])
output,error = ssh.communicate(command_input) # may need .encode() for python 3
return_code = ssh.wait()
then
output_dict = {'stdin': list_of_commands, 'stdout': output.splitlines(), 'stderr': error.splitlines()}
I may add that in the particular ssh case, using paramiko module is better (python paramiko ssh) and avoids using subprocess completely.
Related
I try to run the code on Ubuntu, which works in Windows:
import subprocess
ARGS = ["ping", "google.com"]
process = subprocess.Popen(ARGS, stdout=subprocess.PIPE)
data = process.communicate()
print(data)
When run under Windows, this code prints out tuples of ping packages. But the same code running on Ubuntu just hangs and nothing else!
Can anyone explain to me why?
After testing on Windows, I saw that ping in Windows sent 4 packets and stopped. ping in Ubuntu, on the other hand, sent infinite packets until quitting the program.
The process.communicate() command will wait for end-of-file (EOF) signal, which it gets to in Windows,obviously, but won't get to in Ubuntu. The solution is making ping in Ubuntu terminate. You can limit the number of packets ping sends with the -c flag. Doing ARGS = ["ping", "-c", "4", "google.com"] should get your program to print the results of 4 packets being sent.
The default behavior of ping on most platforms is to just send more packets forever. This will look like it "hangs" because your Python code is waiting for the subprocess to finish, but it never will (though eventually you will fill up memory with the buffered output).
You want to add an option to limit the number of packets, something like
result = subprocess.run(
['ping', '-c', '4', 'google.com'],
capture_output=True, check=True, text=True)
data = result.stdout
Notice also the use of subprocess.run() in preference over the lower-level Popen function. Like the documentation already tells you, you don't want to manage the Popen object yourself if you can avoid it.
Is there any way to use Popen with interactive commands? I mean nslookup, ftp, powershell... I read the whole subprocess documentation several times but I can't find the way.
What I have (removing the parts of the project which aren't of interest here) is:
from subprocess import call, PIPE, Popen
command = raw_input('>>> ')
command = command.split(' ')
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
execution = process.stdout.read()
error = process.stderr.read()
output = execution + error
process.stderr.close()
process.stdout.close()
print(output)
Basically, when I try to print the output with a command like dir, the output is a string, so I can work with the .read() on it. But when I try to use nslookup for example, the output isn't a string, so it can't be read, and the script enters in a deadlock.
I know that I can invoke nslookup in non-interactive mode, but that's not the point. I want to remove all the chances of a deadlock, and make it works with every command you can run in a normal cmd.
The real way the project works is through sockets, so the raw_input is a s.recv() and the output is sending back the output, but I have simplified it to focus on the problem.
Basically I'm trying to automate some Linux installers (and other tasks) using the subprocess library (Popen).
In the past I've been able to open processes like this:
self.process = subprocess.Popen( self.executable,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True, shell=True)
output, cmdError = self.process.communicate()
I can then print output or cmdError for error messages and this works well for single processes or commands.
But when I need to interact with a subprocess and examine the output, it is very difficult, here is my code for doing this:
def ExecProcessWithAnswers(self):
self.process = subprocess.Popen( self.executable,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
stdin=subprocess.PIPE,
universal_newlines=True, shell=True)
while self.process.poll() is None:
print self.process.stdout.readline()
Basically the idea is that I would poll the output (stdout pipe) and then send commands when a certain input is requested from the installer (stdin).
I've tried flushing the buffer, using 3 different ways to read/iterate the output from stdout, but all of them just block or only give me a small fraction of the output text. On the other hand, if I use the communicate method, I get all of the text I expect, but it terminates the process. I've also had EOF errors and other random things.
I've read around: some guides say this is a bug in 2.6.x but it is still in 2.7.x - apparently the stdout is buffered and cannot be changed. I've tried many different ways of parsing the output from various threads here but I still can't get this to work on 2.7.X.
Surely someone must know how to interact with a subprocess? Is my only option here to use pexcept?
I can't really switch to Python 3.x.x within my environment. I was hoping this would be fairly straight forward :(
Cheers
Edit: I've also tried removing the different Pipes, writing to files, changing the buffer size on popen, disabling the shell and universal newlines, etc.
I am creating a movie controller (Pause/Stop...) using python where I ssh into a remote computer, and issue commands into a named pipe like so
echo -n q > ~/pipes/pipename
I know this works if I ssh via the terminal and do it myself, so there is no problem with the setup of the named pipe redirection. My problem is that setting up an ssh session takes time (1-3 seconds), whereas I want the pause command to be instantaneous. Therefore, I thought of setting up a persistent pipe like so:
controller = subprocess.Popen ( "ssh -T -x <hostname>", shell = True, close_fds = True, stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIPE )
Then issue commands to it like so
controller.stdin.write ( 'echo -n q > ~/pipes/pipename' )
I think the problem is that ssh is interactive so it expects a carriage return. This is where my problems begin, as nearly everyone who has asked this question has been told to use an existing module:
Vivek's answer
Chakib's Answer
shx2's Answer
Crafty Thumber's Answer
Artyom's Answer
Jon W's Answer
Which is fine, but I am so close. I just need to know how to include the carriage return, otherwise, I have to go learn all these other modules, which mind you is not trivial (for example, right now I can't figure out how pexpect uses either my /etc/hosts file or my ssh keyless authentications).
To add a newline to the command, you will need to add a newline to the string:
controller.stdin.write('\n')
You may also need to flush the pipe:
controller.stdin.flush()
And of course the controller has to be ready to receive new data, or you could block forever trying to send it data. (And if the reason it's not ready is that it's blocking forever waiting for you to read from its stdout, which is possible on some platforms, you're deadlocked unrecoverably.)
I'm not sure why it's not working the way you have it set up, but I'll take a stab at this. I think what I would do is change the Popen call to:
controller = subprocess.Popen("ssh -T -x <hostname> \"sh -c 'cat > ~/pipes/pipename'\"", ...
And then simply controller.stdin.write('q').
I'm using Windows 7, and I've tried this under Python 2.6.6 and Python 3.2.
So I'm trying to call this command line from Python:
netstat -ano | find ":80"
under Windows cmd, this line works perfectly fine.
So,
1st attempt:
output = subprocess.Popen(
[r'netstat -ano | find ":80"'],
stdout=subprocess.PIPE,
shell=True
).communicate()
An error is raised that 'find' actually didn't receive correct parameter (e.g. 'find ":80" \'):
Access denied - \
2nd attempt:
#calling netstat
cmd_netstat = subprocess.Popen(
['netstat','-ano'],
stdout = subprocess.PIPE
)
#pipelining netstat result into find
cmd_find = subprocess.Popen(
['find','":80"'],
stdin = cmd_netstat.stdout,
stdout = subprocess.PIPE
)
Again, the same error is raised.
Access denied - \
What did I do wrong? :(
EDIT:
3rd attempt (As #Pavel Repin suggested):
cmd_netstat = subprocess.Popen(
['cmd.exe', '-c', 'netstat -ano | find ":80"'],
stdout=subprocess.PIPE
).communicate()
Unfortunately, subprocess with ['cmd.exe','-c'] results in something resembling deadlock or a blank cmd window. I assume '-c' is ignored by cmd, resulting in communicate() waiting indefinitely for cmd termination. Since this is Windows, my bet bet is cmd only accepts parameter starting with slash (/). So I substituted '-c' with '/c':
cmd_netstat = subprocess.Popen(
['cmd.exe', '/c', 'netstat -ano | find ":80"'],
stdout=subprocess.PIPE
).communicate()
And...back to the same error:
Access denied - \
EDIT:
I gave up, I'll just process the string returned by 'netstat -ano' in Python. Might this be a bug?
What I suggest is that you do the maximum inside Python code. So, you can execute the following command:
# executing the command
import subprocess
output = subprocess.Popen(['netstat', '-ano'], stdout=subprocess.PIPE).communicate()
and then by parsing the output:
# filtering the output
valid_lines = [ line for line in output[0].split('\r\n') if ':80' in line ]
You will get a list of lines. On my computer, the output looks like this for port number 1900 (no html connexion active):
[' UDP 127.0.0.1:1900 *:* 1388', ' UDP 192.xxx.xxx.233:1900 *:* 1388']
In my opinion, this is easier to work with.
Note that :
option shell=True is not mandatory, but a command-line window is opened-closed quickly. See what suits you the most, but take care of command injection;
list of Popen arguments shall be a list of string. Quoting of the list parts is not necessary, subprocess will take care of it for you.
Hope this helps.
EDIT: oops, I missed the last line of the edit. Seems you've already got the idea on your own.
So I revisited this question, and found two solutions (I switched to Python 2.7 sometime ago, so I'm not sure about Python 2.6, but it should be the same.):
Replace find with findstr, and remove doublequotes
output = subprocess.Popen(['netstat','-ano','|','findstr',':80'],
stdout=subprocess.PIPE,
shell=True)
.communicate()
But this doesn't explain why "find" cannot be used, so:
Use string parameter instead of list
output = subprocess.Popen('netstat -ano | find ":80"',
stdout=subprocess.PIPE,
shell=True)
.communicate()
or
pipeout = subprocess.Popen(['netstat', '-ano'],
stdout = subprocess.PIPE)
output = subprocess.Popen('find ":80"',
stdin = pipeout.stdout,
stdout = subprocess.PIPE)
.communicate()
The problem arise from the fact that: ['find','":80"'] is actually translated into ['find,'\":80\"'].
Thus the following command is executed in Windows command shell:
>find \":80\"
Access denied - \
Proof:
Running:
output = subprocess.Popen(['echo','find','":80"'],
stdout=subprocess.PIPE,
shell=True)
.communicate()
print output[0]
returns:
find \":80\"
Running:
output = subprocess.Popen('echo find ":80"',
stdout=subprocess.PIPE,
shell=True)
.communicate()
print output[0]
returns:
find ":80"
New answer, after reading this old question again: this may be due to the two following facts:
The pipe operator executes the following commands in a sub-shell; see for instance this interesting consequence).
Python itself uses the pipe as a way to get the results back:
Note that (...) to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.
Not sure if this 'conflict' is kind of a bug, or a design choice though.