We Keep Coding

How to keep ssh connection open and doing multiple requests and outputs within python

Because this question seems to aim somewhere else I am going to point my problem here:
In my python script I am using multiple requests to a remote server using ssh:
def ssh(command):
command = 'ssh SERVER "command"'
output = subprocess.check_output(
command,
stderr=subprocess.STDOUT,
shell=True,
universal_newlines=True
)
return output
here I will get the content of file1 as output.
I have now multiple methods which use this function:
def show_one():
ssh('cat file1')
def show_two():
ssh('cat file2')
def run():
one = show_one()
print(one)
two = show_two()
print(two)
Executing run() will open and close the ssh connection for each show_* method which makes it pretty slow.
Solutions:
I can put:
Host SERVER
ControlMaster auto
ControlPersist yes
ControlPath ~/.ssh/socket-%r#%h:%p
into my .ssh/config but I would like to solve this within python.
There is the ssh flag -T to keep a connection open, and in the before mentioned Question one answer was to use this with Popen() and p.communicate() but it is not possible to get the output between the communicates because it throws an error ValueError: Cannot send input after starting communication
I could somehow change my functions to execute a single ssh command like echo "--show1--"; cat file1; echo "--show2--"; cat file2 but this looks hacky to me and I hope there is a better method to just keep the ssh connection open and use it like normal.
What I would like to have: For example a pythonic/bashic to do the same as I can configure in the .ssh/config (see 1.) to declare a specific socket for the connection and explicitly open, use, close it

Try to create ssh object from class and pass it to the functions:
import paramiko
from pythonping import ping
from scp import SCPClient
class SSH():
def __init__(self, ip='192.168.1.1', username='user', password='pass',connect=True,Timeout=10):
self.ip = ip
self.username = username
self.password = password
self.Timeout=Timeout
self.ssh = paramiko.SSHClient()
self.ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
if connect:
self.OpenConnection()
self.scp = SCPClient(self.ssh.get_transport())
def OpenConnection(self):
try:
skip_ping = False
ping_res=False
log.info('Sending ping to host (timeout=3,count=3) :'+self.ip)
try:
PingRes = ping(target=self.ip,timeout=3,count=3, verbose=True)
log.info('Ping to host result :' + str(PingRes.success()))
ping_res=PingRes.success()
except:
skip_ping=True
if ping_res or skip_ping:
log.info('Starting to open connection....')
self.ssh.connect(hostname=self.ip, username=self.username, password=self.password, timeout=self.Timeout, auth_timeout=self.Timeout,banner_timeout=self.Timeout)
self.scp = SCPClient(self.ssh.get_transport())
log.info('Connection open')
return True
else:
log.error('ssh OpenConnection failed: No Ping to host')
return False
myssh = SSH(ip='192.168.1.1',password='mypass',username='myusername')
the ping result is wrapped in try catch because sometimes my machine return an error you can remove it and just verify a ping to the host.
The self.scp is for file transfer.

How to skip lines when printing output from Paramiko SSH

So I built a program that prints out the login logs of my ubuntu server using tail -f.
The program uses Paramiko to connect via ssh and runs the command to tail the logs.
The program works but it prints out the motd from the server which is unnecessary.
I've tried splicing using itertools.
Tried using next().
Still doesn't work.
Here's my code:
import yaml, paramiko, getpass, traceback, time, itertools
from paramiko_expect import SSHClientInteraction
with open("config.yaml", "r") as yamlfile:
cfg = yaml.load(yamlfile, Loader=yaml.FullLoader)
def main():
command = "sudo tail -f /var/log/auth.log"
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
server_pw = getpass.getpass("Enter the password for your account %s on %s:" % (cfg['ssh_config']['username'], cfg['ssh_config']['host']))
sudo_pw = getpass.getpass("Enter the sudo password for %s on %s: " % (cfg['ssh_config']['username'], cfg['ssh_config']['host']))
ssh.connect(hostname = cfg['ssh_config']['host'], username = cfg['ssh_config']['username'], port = cfg['ssh_config']['port'], password = server_pw)
interact = SSHClientInteraction(ssh, timeout=10, display=False)
interact.send(command)
interact.send(sudo_pw + "\n")
with open(interact.tail(line_prefix=cfg['ssh_config']['servername']+': ')) as tail:
for line in itertools.islice(tail, 17, None):
print(line)
except KeyboardInterrupt:
print('Ctrl+C interruption detected, stopping tail')
except Exception:
traceback.print_exc()
finally:
try:
ssh.close()
except:
pass
if __name__ == '__main__':
main()

You get MOTD because you are opening an interactive shell session. I do not think you need that, quite on the contrary.
Use SSHClient.exec_command instead:
stdin, stdout, stderr = ssh.exec_command(command, get_pty=True)
stdin.write(sudo_pw + "\n")
stdin.flush()
for line in iter(stdout.readline, ""):
print(line, end="")
Related questions:
Get output from a Paramiko SSH exec_command continuously
Pass input/variables to command/script over SSH using Python Paramiko
What is the difference between exec_command and send with invoke_shell() on Paramiko?
Obligatory warning: Do not use AutoAddPolicy – You are losing a protection against MITM attacks by doing so. For a correct solution, see Paramiko "Unknown Server".

Find out if asked for a password when looping over hosts using subprocess.Popen

I am looping over a number of hosts (about 400) to get some information from the systems. most of them have ssh keys so no problem there. Only a handful don't have ssh keys and come back with a password. Now I want to detect when I'm asked for a password, kill the ssh process and continue to the next host.
here's the code for the ssh part
def get_os_info(hostname, command, user=None):
remote = "%s" % hostname if user is None else "%s#%s" %(user, hostname)
ssh = subprocess.Popen(["ssh", remote, command],
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
result = ssh.stdout.readlines()
if result == []:
error = ssh.stderr.readlines()
print >>sys.stderr, "ERROR: %s" % error
return None
else:
return result
I am not able to use paramiko.

Instead of using trying to detect for a password, instead avoid prompting for it, look at Batch Mode (with Open SSH):
ssh -oBatchMode=yes
Which is designed to prevent prompting for passwords when calling from scripts. Now you just need to detect if connected or not.
(although, as your comment at the bottom suggest, using paramiko is probably technically better solution)

paramiko exec_command hangs after a few commands

im connecting a linux server with sshclient .And then, im connecting cisco routers via telnet on this server. I'm connecting server and execute telnet command perfectly but in second or third telnet command code get stucked and doesnt throw error.Here is part of my code:
def __init__(self):
self.pre_client=paramiko.SSHClient()
self.pre_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
sellf.pre_client.connect("server",username="user",password="password")
self.client=self.pre_client.invoke_shell()
def connect(self,ip):
o=self.client.recv(1024)
print o
self.client.exec_command("telnet %s\n"%(ip))
while True:
o=self.client.recv(1024)
print o
#EXECUTE COMMAND ON ROUTER
self.client.exec_command("exit\n")
if 'exit' in o:
break
Why it get stuck on this command? How can i handle it?

I think I'd need to see more code to find what's wrong. If you don't have a specific reason for needing to use a Paramiko Channel, life will probably be a lot easier for you if you just use a Paramiko Client. This is a rough snippet from an old script of mine:
def ssh_connect(usr, pswds, host):
# Open Connection - auto add policy
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
# Retry with multiple passwords
for pswd in pswds:
try:
ssh.connect(host, username=usr, password=pswd)
return ssh
except:
pass
else:
print("Could not login to: " + host)
return None
def send_command(conn, command):
try:
stdin, stdout, stderr = conn.exec_command(command)
if stdout:
for str in stdout:
sys.stdout.write("\t%s" % str)
return True
if stderr:
for str in stderr:
sys.stderr.write("\t%s" % str)
return False
else:
print("\n")
return True
except paramiko.ssh_exception.SSHException as e:
print(e.message)
return False
And of course, call them with:
conn = ssh_connect(login, passwords, host)
send_command(conn, command)

Perform commands over ssh with Python

I'm writing a script to automate some command line commands in Python. At the moment, I'm doing calls like this:
cmd = "some unix command"
retcode = subprocess.call(cmd,shell=True)
However, I need to run some commands on a remote machine. Manually, I would log in using ssh and then run the commands. How would I automate this in Python? I need to log in with a (known) password to the remote machine, so I can't just use cmd = ssh user#remotehost, I'm wondering if there's a module I should be using?

I will refer you to paramiko
see this question
ssh = paramiko.SSHClient()
ssh.connect(server, username=username, password=password)
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(cmd_to_execute)
If you are using ssh keys, do:
k = paramiko.RSAKey.from_private_key_file(keyfilename)
# OR k = paramiko.DSSKey.from_private_key_file(keyfilename)
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname=host, username=user, pkey=k)

Keep it simple. No libraries required.
import subprocess
# Python 2
subprocess.Popen("ssh {user}#{host} {cmd}".format(user=user, host=host, cmd='ls -l'), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
# Python 3
subprocess.Popen(f"ssh {user}#{host} {cmd}", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()

Or you can just use commands.getstatusoutput:
commands.getstatusoutput("ssh machine 1 'your script'")
I used it extensively and it works great.
In Python 2.6+, use subprocess.check_output.

I found paramiko to be a bit too low-level, and Fabric not especially well-suited to being used as a library, so I put together my own library called spur that uses paramiko to implement a slightly nicer interface:
import spur
shell = spur.SshShell(hostname="localhost", username="bob", password="password1")
result = shell.run(["echo", "-n", "hello"])
print result.output # prints hello
If you need to run inside a shell:
shell.run(["sh", "-c", "echo -n hello"])

All have already stated (recommended) using paramiko and I am just sharing a python code (API one may say) that will allow you to execute multiple commands in one go.
to execute commands on different node use : Commands().run_cmd(host_ip, list_of_commands)
You will see one TODO, which I have kept to stop the execution if any of the commands fails to execute, I don't know how to do it. please share your knowledge
#!/usr/bin/python
import os
import sys
import select
import paramiko
import time
class Commands:
def __init__(self, retry_time=0):
self.retry_time = retry_time
pass
def run_cmd(self, host_ip, cmd_list):
i = 0
while True:
# print("Trying to connect to %s (%i/%i)" % (self.host, i, self.retry_time))
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host_ip)
break
except paramiko.AuthenticationException:
print("Authentication failed when connecting to %s" % host_ip)
sys.exit(1)
except:
print("Could not SSH to %s, waiting for it to start" % host_ip)
i += 1
time.sleep(2)
# If we could not connect within time limit
if i >= self.retry_time:
print("Could not connect to %s. Giving up" % host_ip)
sys.exit(1)
# After connection is successful
# Send the command
for command in cmd_list:
# print command
print "> " + command
# execute commands
stdin, stdout, stderr = ssh.exec_command(command)
# TODO() : if an error is thrown, stop further rules and revert back changes
# Wait for the command to terminate
while not stdout.channel.exit_status_ready():
# Only print data if there is data to read in the channel
if stdout.channel.recv_ready():
rl, wl, xl = select.select([ stdout.channel ], [ ], [ ], 0.0)
if len(rl) > 0:
tmp = stdout.channel.recv(1024)
output = tmp.decode()
print output
# Close SSH connection
ssh.close()
return
def main(args=None):
if args is None:
print "arguments expected"
else:
# args = {'<ip_address>', <list_of_commands>}
mytest = Commands()
mytest.run_cmd(host_ip=args[0], cmd_list=args[1])
return
if __name__ == "__main__":
main(sys.argv[1:])

paramiko finally worked for me after adding additional line, which is really important one (line 3):
import paramiko
p = paramiko.SSHClient()
p.set_missing_host_key_policy(paramiko.AutoAddPolicy()) # This script doesn't work for me unless this line is added!
p.connect("server", port=22, username="username", password="password")
stdin, stdout, stderr = p.exec_command("your command")
opt = stdout.readlines()
opt = "".join(opt)
print(opt)
Make sure that paramiko package is installed.
Original source of the solution: Source

The accepted answer didn't work for me, here's what I used instead:
import paramiko
import os
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
# ssh.load_system_host_keys()
ssh.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
ssh.connect("d.d.d.d", username="user", password="pass", port=22222)
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command("ls -alrt")
exit_code = ssh_stdout.channel.recv_exit_status() # handles async exit error
for line in ssh_stdout:
print(line.strip())
total 44
-rw-r--r--. 1 root root 129 Dec 28 2013 .tcshrc
-rw-r--r--. 1 root root 100 Dec 28 2013 .cshrc
-rw-r--r--. 1 root root 176 Dec 28 2013 .bashrc
...
Alternatively, you can use sshpass:
import subprocess
cmd = """ sshpass -p "myPas$" ssh user#d.d.d.d -p 22222 'my command; exit' """
print( subprocess.getoutput(cmd) )
References:
https://github.com/onyxfish/relay/issues/11
https://stackoverflow.com/a/61016663/797495
Notes:
Just make sure to connect manually at least one time to the remote system via ssh (ssh root#ip) and accept the public key, this is many times the reason from not being able connect using paramiko or other automated ssh scripts.

I have used paramiko a bunch (nice) and pxssh (also nice). I would recommend either. They work a little differently but have a relatively large overlap in usage.

First: I'm surprised that no one has mentioned fabric yet.
Second: For exactly those requirements you describe I've implemented an own python module named jk_simpleexec. It's purpose: Making running commands easy.
Let me explain a little bit about it for you.
The 'executing a command locally' problem
My python module jk_simpleexec provides a function named runCmd(..) that can execute a shell (!) command locally or remotely. This is very simple. Here is an example for local execution of a command:
import jk_simpleexec
cmdResult = jk_simpleexec.runCmd(None, "cd / ; ls -la")
NOTE: Be aware that the returned data is trimmed automatically by default to remove excessive empty lines from STDOUT and STDERR. (Of course this behavior can be deactivated, but for the purpose you've in mind exactly that behavior is what you will want.)
The 'processing the result' problem
What you will receive is an object that contains the return code, STDOUT and STDERR. Therefore it's very easy to process the result.
And this is what you want to do as the command you execute might exist and is launched but might fail in doing what it is intended to do. In the most simple case where you're not interested in STDOUT and STDERR your code will likely look something like this:
cmdResult.raiseExceptionOnError("Something went wrong!", bDumpStatusOnError=True)
For debugging purposes you want to output the result to STDOUT at some time, so for this you can do just this:
cmdResult.dump()
If you would want to process STDOUT it's simple as well. Example:
for line in cmdResult.stdOutLines:
print(line)
The 'executing a command remotely' problem
Now of course we might want to execute this command remotely on another system. For this we can use the same function runCmd(..) in exactly the same way but we need to specify a fabric connection object first. This can be done like this:
from fabric import Connection
REMOTE_HOST = "myhost"
REMOTE_PORT = 22
REMOTE_LOGIN = "mylogin"
REMOTE_PASSWORD = "mypwd"
c = Connection(host=REMOTE_HOST, user=REMOTE_LOGIN, port=REMOTE_PORT, connect_kwargs={"password": REMOTE_PASSWORD})
cmdResult = jk_simpleexec.runCmd(c, "cd / ; ls -la")
# ... process the result stored in cmdResult ...
c.close()
Everything remains exactly the same, but this time we run this command on another host. This is intended: I wanted to have a uniform API where there are no modifications required in the software if you at some time decide to move from the local host to another host.
The password input problem
Now of course there is the password problem. This has been mentioned above by some users: We might want to ask the user executing this python code for a password.
For this problem I have created an own module quite some time ago. jk_pwdinput. The difference to regular password input is that jk_pwdinput will output some stars instead of just printing nothing. So for every password character you type you will see a star. This way it's more easy for you to enter a password.
Here is the code:
import jk_pwdinput
# ... define other 'constants' such as REMOTE_LOGIN, REMOTE_HOST ...
REMOTE_PASSWORD = jk_pwdinput.readpwd("Password for " + REMOTE_LOGIN + "#" + REMOTE_HOST + ": ")
(For completeness: If readpwd(..) returned None the user canceled the password input with Ctrl+C. In a real world scenario you might want to act on this appropriately.)
Full example
Here is a full example:
import jk_simpleexec
import jk_pwdinput
from fabric import Connection
REMOTE_HOST = "myhost"
REMOTE_PORT = 22
REMOTE_LOGIN = "mylogin"
REMOTE_PASSWORD = jk_pwdinput.readpwd("Password for " + REMOTE_LOGIN + "#" + REMOTE_HOST + ": ")
c = Connection(host=REMOTE_HOST, user=REMOTE_LOGIN, port=REMOTE_PORT, connect_kwargs={"password": REMOTE_PASSWORD})
cmdResult = jk_simpleexec.runCmd(
c = c,
command = "cd / ; ls -la"
)
cmdResult.raiseExceptionOnError("Something went wrong!", bDumpStatusOnError=True)
c.close()
Final notes
So we have the full set:
Executing a command,
executing that command remotely via the same API,
creating the connection in an easy and secure way with password input.
The code above solves the problem quite well for me (and hopefully for you as well). And everything is open source: Fabric is BSD-2-Clause, and my own modules are provided under Apache-2.
Modules used:
fabric : http://www.fabfile.org/
jk_pwdinput : https://github.com/jkpubsrc/python-module-jk-pwdinput
jk_simplexec : https://github.com/jkpubsrc/python-module-jk-simpleexec
Happy coding! ;-)

Works Perfectly...
import paramiko
import time
ssh = paramiko.SSHClient()
#ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('10.106.104.24', port=22, username='admin', password='')
time.sleep(5)
print('connected')
stdin, stdout, stderr = ssh.exec_command(" ")
def execute():
stdin.write('xcommand SystemUnit Boot Action: Restart\n')
print('success')
execute()

You can use any of these commands, this will help you to give a password also.
cmd = subprocess.run(["sshpass -p 'password' ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root#domain.com ps | grep minicom"], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
print(cmd.stdout)
OR
cmd = subprocess.getoutput("sshpass -p 'password' ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root#domain.com ps | grep minicom")
print(cmd)

Have a look at spurplus, a wrapper we developed around spur that provides type annotations and some minor gimmicks (reconnecting SFTP, md5 etc.): https://pypi.org/project/spurplus/

Asking User to enter the command as per the device they are logging in.
The below code is validated by PEP8online.com.
import paramiko
import xlrd
import time
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
loc = ('/Users/harshgow/Documents/PYTHON_WORK/labcred.xlsx')
wo = xlrd.open_workbook(loc)
sheet = wo.sheet_by_index(0)
Host = sheet.cell_value(0, 1)
Port = int(sheet.cell_value(3, 1))
User = sheet.cell_value(1, 1)
Pass = sheet.cell_value(2, 1)
def details(Host, Port, User, Pass):
time.sleep(2)
ssh.connect(Host, Port, User, Pass)
print('connected to ip ', Host)
stdin, stdout, stderr = ssh.exec_command("")
x = input('Enter the command:')
stdin.write(x)
stdin.write('\n')
print('success')
details(Host, Port, User, Pass)

#Reading the Host,username,password,port from excel file
import paramiko
import xlrd
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
loc = ('/Users/harshgow/Documents/PYTHON_WORK/labcred.xlsx')
wo = xlrd.open_workbook(loc)
sheet = wo.sheet_by_index(0)
Host = sheet.cell_value(0,1)
Port = int(sheet.cell_value(3,1))
User = sheet.cell_value(1,1)
Pass = sheet.cell_value(2,1)
def details(Host,Port,User,Pass):
ssh.connect(Host, Port, User, Pass)
print('connected to ip ',Host)
stdin, stdout, stderr = ssh.exec_command("")
stdin.write('xcommand SystemUnit Boot Action: Restart\n')
print('success')
details(Host,Port,User,Pass)

The most modern approach is probably to use fabric. This module allows you to set up an SSH connection and then run commands and get their results over the connection object.
Here's a simple example:
from fabric import Connection
with Connection("your_hostname") as connection:
result = connection.run("uname -s", hide=True)
msg = "Ran {0.command!r} on {0.connection.host}, got stdout:\n{0.stdout}"
print(msg.format(result))

I wrote a simple class to run commands on remote over native ssh, using the subprocess module:
Usage
from ssh_utils import SshClient
client = SshClient(user='username', remote='remote_host', key='path/to/key.pem')
# run a list of commands
client.cmd(['mkdir ~/testdir', 'ls -la', 'echo done!'])
# copy files/dirs
client.scp('my_file.txt', '~/testdir')
Class source code
https://gist.github.com/mamaj/a7b378a5c969e3e32a9e4f9bceb0c5eb
import subprocess
from pathlib import Path
from typing import Union
class SshClient():
""" Perform commands and copy files on ssh using subprocess
and native ssh client (OpenSSH).
"""
def __init__(self,
user: str,
remote: str,
key_path: Union[str, Path]) -> None:
"""
Args:
user (str): username for the remote
remote (str): remote host IP/DNS
key_path (str or pathlib.Path): path to .pem file
"""
self.user = user
self.remote = remote
self.key_path = str(key_path)
def cmd(self,
cmds: list[str],
strict_host_key_checking=False) -> None:
"""runs commands consecutively, ensuring success of each
after calling the next command.
Args:
cmds (list[str]): list of commands to run.
strict_host_key_checking (bool, optional): Defaults to True.
"""
strict_host_key_checking = 'yes' if strict_host_key_checking \
else 'no'
cmd = ' && '.join(cmds)
subprocess.run(
[
'ssh',
'-i', self.key_path,
'-o', f'StrictHostKeyChecking={strict_host_key_checking}',
'-o', 'UserKnownHostsFile=/dev/null',
f'{self.user}#{self.remote}',
cmd
]
)
def scp(self, source: Union[str, Path], destination: Union[str, Path]):
"""Copies `srouce` file to remote `destination` using the
native `scp` command.
Args:
source (Union[str, Path]): Source file path.
destination (Union[str, Path]): Destination path on remote.
"""
subprocess.run(
[
'scp',
'-i', self.key_path,
str(source),
f'{self.user}#{self.remote}:{str(destination)}',
]
)

Below example, incase if you want user inputs for hostname,username,password and port no.
import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
def details():
Host = input("Enter the Hostname: ")
Port = input("Enter the Port: ")
User = input("Enter the Username: ")
Pass = input("Enter the Password: ")
ssh.connect(Host, Port, User, Pass, timeout=2)
print('connected')
stdin, stdout, stderr = ssh.exec_command("")
stdin.write('xcommand SystemUnit Boot Action: Restart\n')
print('success')
details()