I am trying to download a file in all the machine and therefore I have created a python script. It uses the module paramiko
just a snippet from the code:
from paramiko import SSHClient, AutoAddPolicy
ssh = SSHClient()
ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(AutoAddPolicy())
ssh.connect(args.ip,username=args.username,password=args.password)
stdin, stdout, stderr = ssh.exec_command("wget xyz")
print(stdout.read())
The output will be printed after the download has been completed.!
Is there a way I can print the output real time?
Edit: I have looked at this answer and applied something like this:
def line_buffered(f):
line_buf = ""
print "start"
print f.channel.exit_status_ready()
while not f.channel.exit_status_ready():
print("ok")
line_buf += f.read(size=1)
if line_buf.endswith('\n'):
yield line_buf
line_buf = ''
in, out, err = ssh.exec_command("wget xyz")
for l in line_buffered(out):
print l
But, It is not printing the data real time.! It waits for the file to download and then prints the whole status of the download.
Also, I have tried this command : echo one && sleep 5 && echo two && sleep 5 && echo three and the output is realtime with line_buffered function. But, for wget command, it is not working..
wget's progress information outputs to stderr so you should write line_buffered(err).
Or you can use exec_command("wget xyz", get_pty=True) which would combine stdout and stderr.
Related
Problem Statement:
Execute a groovy script in shell using Python
The Shell would request for password and display "Password:"
Need to check if that is displayed and provide a password.
Verify the output as "Approved"
My code looks something like this:
#!/usr/bin/env python
import subprocess
import time
process = subprocess.Popen(['groovy', 'some.groovy -u param1 -h param2-p param3'],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
time.sleep(5)
stdout, stderr = process.communicate()
print stdout
#Now id the output says "Password:", I need to provide it and then again check if shell displays "Approved"
I am trying to write to a custom program's stdin with paramiko. Here is a minimal (non-)working example:
~/stdin_to_file.py:
#! /usr/bin/python
import time, sys
f = open('/home/me/LOG','w')
while True:
sys.stdin.flush()
data = sys.stdin.read()
f.write(data+'\n\n')
f.flush()
time.sleep(0.01)
Then I do these commands in IPython:
import paramiko
s = paramiko.client.SSHClient
s.load_system_host_keys()
s.connect('myserver')
stdin, stdout, stderr = s.exec_command('/home/me/stdin_to_file.py')
stdin.write('Hello!')
stdin.flush()
Unfortunately, nothing then appears in ~/LOG. However, if I do
$ ~/stdin_to_file.py < some_other_file
The contents of some_other_file appear in ~/LOG.
Can anyone suggest where I've gone wrong? It seems like I'm doing the logical thing. None of these work either:
stdin.channel.send('hi')
using the get_pty parameter
sending the output of cat - to stdin_to_file.py
sys.stdin.read() will keep reading until EOF so in your paramiko script you need to close the stdin (returned from exec_command()). But how?
1. stdin.close() would not work.
According to Paramiko's doc (v1.16):
Warning: To correctly emulate the file object created from a socket’s makefile() method, a Channel and its ChannelFile should be able to be closed or garbage-collected independently. Currently, closing the ChannelFile does nothing but flush the buffer.
2. stdin.channel.close() also has problem.
Since stdin, stdout and stderr all share one single channel, stdin.channel.close() will also close stdout and stderr which is not expected.
3. stdin.channel.shutdown_write()
The correct solution is to use stdin.channel.shutdown_write() which disallows writing to the channel but still allows reading from the channel so stdout.read() and stderr.read() would still work.
See following example to see the difference between stdin.channel.close() and stdin.channel.shutdown_write().
[STEP 101] # cat foo.py
import paramiko, sys, time
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy() )
ssh.connect(hostname='127.0.0.1', username='root', password='password')
cmd = "sh -c 'read v; sleep 1; echo $v'"
stdin, stdout, stderr = ssh.exec_command(cmd)
if sys.argv[1] == 'close':
stdin.write('hello world\n')
stdin.flush()
stdin.channel.close()
elif sys.argv[1] == 'shutdown_write':
stdin.channel.send('hello world\n')
stdin.channel.shutdown_write()
else:
raise Exception()
sys.stdout.write(stdout.read() )
[STEP 102] # python foo.py close # It outputs nothing.
[STEP 103] # python foo.py shutdown_write # This works fine.
hello world
[STEP 104] #
I am trying to run several commands in a single ssh session and save the output of each command in a different file.
The code that works (but it saves all the output in a single file)
conn = Popen(['ssh',host, "ls;uname -a;pwd"], stdin=PIPE, stdout = open ('/output.txt','w'))
mypassword = conn.communicate('password')
Codes that I am trying to work but not working...
cmd = ['ls', 'pwd', 'uname']
conn = Popen(['ssh',host, "{};{};{}".format(cmd[0],cmd[1],cmd[2])], stdin=PIPE, stdout = output.append('a'))
mypassword = conn.communicate('password')
print (output)
length = range(len(output))
print length
for i in output:
open("$i",'w')
and
cmd = ['ls', 'pwd', 'uname']
conn = Popen(['ssh',host, "{};{};{}".format(cmd[0],cmd[1],cmd[2])], stdin=PIPE, stdout = output())
mypassword = conn.communicate('password')
def output():
for i in cmd:
open(i,'w')
return
Not sure what is the best way of doing it. Should I save it in an array and then save each item in a separate file or should I call a function that will do it?
NOTE that the commands I want to run do not have small output like given in examples here (uname, pwd); it is big as tcpdump, lsof etc.
A single ssh session runs a single command e.g., /bin/bash on the remote host -- you can pass input to that command to emulate running multiple commands in a single ssh session.
Your code won't run even a single command. There are multiple issues in it:
ssh may read the password directly from the terminal (not its stdin stream). conn.communicate('password') in your code writes to ssh's stdin therefore ssh won't get the password.
There are multiple ways to authenticate via ssh e.g., use ssh keys (passwordless login).
stdout = output.append('a') doesn't redirect ssh's stdout because .append list method returns None.
It won't help you to save output of several commands to different files. You could redirect the output to remote files and copy them back later: ls >ls.out; uname -a >uname.out; pwd >pwd.out.
A (hacky) alternative is to use inside stream markers (echo <GUID>) to differentiate the output from different commands. If the output can be unlimited; learn how to read subprocess' output incrementally (without calling .communicate() method).
for i in cmd: open(i,'w') is pointless. It opens (and immediately closes on CPython) multiple files without using them.
To avoid such basic mistakes, write several Python scripts that operate on local files.
SYS_STATS={"Number of CPU Cores":"cat /proc/cpuinfo|grep -c 'processor'\n",
"CPU MHz":"cat /proc/cpuinfo|grep 'cpu MHz'|head -1|awk -F':' '{print $2}'\n",
"Memory Total":"cat /proc/meminfo|grep 'MemTotal'|awk -F':' '{print $2}'|sed 's/ //g'|grep -o '[0-9]*'\n",
"Swap Total":"cat /proc/meminfo|grep 'SwapTotal'|awk -F':' '{print $2}'|sed 's/ //g'|grep -o '[0-9]*'\n"}
def get_system_details(self,ipaddress,user,key):
_OutPut={}
values=[]
sshProcess = subprocess.Popen(['ssh','-T','-o StrictHostKeyChecking=no','-i','%s' % key,'%s#%s'%(user,ipaddress),"sudo","su"],
stdin=subprocess.PIPE, stdout = subprocess.PIPE, universal_newlines=True,bufsize=0)
for element in self.SYS_STATS1:
sshProcess.stdin.write("echo END\n")
sshProcess.stdin.write(element)
sshProcess.stdin.close()
for element in sshProcess.stdout:
if element.rstrip('\n')!="END":
values.append(element.rstrip('\n'))
mapObj={k: v for k, v in zip(self.SYS_STATS_KEYS, values)}
return mapObj
I'm making a custom shell in Python for a very limited user on a server, who is logged in via ssh with a public key authentication. They need to be able to run ls, find -type d, and cat in specific directories with certain limitations. This works fine if you run something like ssh user#server -i keyfile, because you see the interactive prompt, and can run those commands. However, something like ssh user#server -i keyfile "ls /var/log" doesn't. ssh simply hangs, with no response. By using the -v switch I've found that the connection is succeeding, so the problem is in my shell. I'm also fairly certain that the script isn't even being started, since print sys.argv at the beginning of the program does nothing. Here's the code:
#!/usr/bin/env python
import subprocess
import re
import os
with open(os.devnull, 'w') as devnull:
proc = lambda x: subprocess.Popen(x, stdout=subprocess.PIPE, stderr=devnull)
while True:
try:
s = raw_input('> ')
except:
break
try:
cmd = re.split(r'\s+', s)
if len(cmd) != 2:
print 'Not permitted.'
continue
if cmd[0].lower() == 'l':
# Snip: verify directory
cmd = proc(['ls', cmd[1]])
print cmd.stdout.read()
elif cmd[0].lower() == 'r':
# Snip: verify directory
cmd = proc(['cat', cmd[1]])
print cmd.stdout.read()
elif cmd[0].lower() == 'll':
# Snip: verify directory
cmd = proc(['find', cmd[1], '-type', 'd'])
print cmd.stdout.read()
else:
print 'Not permitted.'
except OSError:
print 'Unknown error.'
And here's the relevant line from ~/.ssh/authorized_keys:
command="/path/to/shell $SSH_ORIGINAL_COMMAND" ssh-rsa [base-64-encoded-key] user#host
How can I make the shell script when the command is passed on the command line so it can be used in scripts without starting an interactive shell?
The problem with ssh not responding is related to the fact that ssh user#host cmd does not open a terminal for the command being run. Try calling ssh user#host -t cmd.
However, even if you pass the -t option, you'd still have another problem with your script: it only works interactively and totally ignores the $SSH_ORIGINAL_PROGRAM being passed. A naive solution would be to check sys.argv and if its bigger than 1 you don't loop forever, and instead only execute whatever command you have in it.
Does anyone know how to make environment variables registered for
exec_command calls when using SSHClient?
I'm using a basic script that instantiates the SSHClient class, connects to another computer using the connect method, then sends out commands using the exec_command method. However, none of the environment variables seem to be registered when I try to issue commands. I can do basic things like 'ls' and see the stdout, but when trying to run installed programs, the fact that the environment variables are missing makes it impossible to run them. Using ssh in the command line to do the same thing works, as the environment variables for the user are set.
#!/usr/bin/python
import paramiko
ssh.connect('mymachine',username='myname',password='pass')
stdin,stdout,stderr=ssh.exec_command('cd /myfolder/path')
stdin,stdout,stderr=ssh.exec_command('ls')
....
....
ssh.close()
Note: I can't change my directory in paramiko. I appended the cd command in the followup command in a single ssh.exec_command('cd /dddd/ddd;ls'). I have given ls as an example but my actual followup command is different.
Since release 2.1.0 2016-12-09 , you can add an environment variable dictionary to the exec_command:
import paramiko
paramiko.util.log_to_file("paramiko.log")
ssh = paramiko.SSHClient()
k = paramiko.RSAKey.from_private_key_file("<private_key_file>")
ssh.connect(<hostname>,username=<username>,pkey=k)
env_dict={"LC_TELEPHONE":"ET_HOME","LC_MEASUREMENT":"MILES_APART"}
stdin , stdout, stderr = ssh.exec_command('echo $LC_TELEPHONE; echo "..."; echo $LC_MEASUREMENT',environment=env_dict)
print stdout.read()
output:
ET_HOME
...
MILES_APART
But why did I choose LC_TELEPHONE and LC_MEASUREMENT? Because those are two of the few environments that the target host's ssh config allows me to set:
grep AcceptEnv /etc/ssh/sshd_config
output:
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL
In other words, this doesn't work:
env_dict={"HELLO":"WORLD","GOODBYE":"CRUEL_WORLD"}
stdin , stdout, stderr = ssh.exec_command("echo $HELLO; echo '...'; echo $GOODBYE")
print stdout.read()
output:
...
As the documentation warns, the environment variables are silently rejected
http://docs.paramiko.org/en/2.1/api/client.html
http://docs.paramiko.org/en/2.1/api/channel.html#paramiko.channel.Channel.set_environment_variable
If you cannot control the target server's sshd config, putting the environment variables into a file and sourcing it works:
stdin , stdout, stderr = ssh.exec_command("cat .set_env;source .set_env; echo $HELLO; echo '...'; echo $GOODBYE")
print stdout.read()
output:
# begin .set_env
HELLO="WORLD"
GOODBYE="CRUEL_WORLD"
# end .set_env
WORLD
...
CRUEL_WORLD
#!/usr/bin/python
import paramiko
client = paramiko.SSHClient()
client.load_system_host_keys()
client.set_missing_host_key_policy(paramiko.WarningPolicy)
client.connect(myhostname, theport, myuser, thepass)
stdin,stdout,stderr = client.exec_command('cd /tmp;pwd;ls -al')
#returns your output
print stdout.read()
which all works fine for me. If you have special environment Variables you might
have to set them on the remote command prompt. Maybe it helps if you write the
variables into a myENV file and then call
stdin,stdout,stderr = client.exec_command('source ./myEnv')
Did you tried something like that?
You can do: client.exec_command(..., get_pty=True).
This will make paramiko allocate a pseudo terminal, similar to ssh.
I found this problem too. And besides the above approaches, I also fixed the problem by using the following approach:
e.g.,
...
bin_paths = '/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin'
path_prefix = 'PATH=$PATH:%s && ' % bin_paths
command = path_prefix + command
ssh.exec_command(command=command)