We Keep Coding

Nornir Netmiko connection to AWS EC2 Cisco IOS-XE router fails - can't connect because of SSH Key?

I try to connect via Nornir / Netmiko to an Cisco CSR1k Router. In AWS keys are used and the EC2-USER expect a key authentication.
This is my simple test Nornir Script:
from nornir_netmiko import netmiko_send_config, netmiko_send_command
from nornir import InitNornir
from nornir_utils.plugins.functions import print_result
from nornir.core.filter import F
nr = InitNornir(config_file="config.yaml")
cisco = nr.filter(F(platform="cisco_xe"))
result = cisco.run(
task=netmiko_send_command,
command_string="show arp"
)
print_result(result)
And this is my Inventroy:
---
Europe-Cisco-Site1:
hostname: "18.158.60.42"
username: "ec2-user"
password: ""
platform: "cisco_xe"
groups:
- cisco
I would expect that Nornir will handle the authentication automatically and use the keys. It seams this is not happening.
When I try to connect via ssh from the same host to the AWS EC2 Cisco CSR1k everthing works.
Question:
Do I need to give some extra options to use the SSH key?
I found something like this but it does not work:
connection_options:
paramiko:
port: 22
hostname:
username: "ec2-user"
password:
platform: "cisco_xe"
extras:
alt_host_keys: True
alt_key_file: "/home/coder/.ssh/id_rsa"
use_keys: True
And this is the error what I get:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/coder/.local/lib/python3.8/site-packages/nornir/core/task.py", line 99, in start
r = self.task(self, **self.params)
File "/home/coder/.local/lib/python3.8/site-packages/nornir_netmiko/tasks/netmiko_send_command.py", line 26, in netmiko_send_command
net_connect = task.host.get_connection(CONNECTION_NAME, task.nornir.config)
File "/home/coder/.local/lib/python3.8/site-packages/nornir/core/inventory.py", line 494, in get_connection
self.open_connection(
File "/home/coder/.local/lib/python3.8/site-packages/nornir/core/inventory.py", line 546, in open_connection
conn_obj.open(
File "/home/coder/.local/lib/python3.8/site-packages/nornir_netmiko/connections/netmiko.py", line 59, in open
connection = ConnectHandler(**parameters)
File "/home/coder/.local/lib/python3.8/site-packages/netmiko/ssh_dispatcher.py", line 326, in ConnectHandler
return ConnectionClass(*args, **kwargs)
File "/home/coder/.local/lib/python3.8/site-packages/netmiko/cisco/cisco_ios.py", line 17, in __init__
return super().__init__(*args, **kwargs)
File "/home/coder/.local/lib/python3.8/site-packages/netmiko/base_connection.py", line 350, in __init__
self._open()
File "/home/coder/.local/lib/python3.8/site-packages/netmiko/base_connection.py", line 355, in _open
self.establish_connection()
File "/home/coder/.local/lib/python3.8/site-packages/netmiko/base_connection.py", line 972, in establish_connection
raise NetmikoAuthenticationException(msg)
netmiko.ssh_exception.NetmikoAuthenticationException: Authentication to device failed.
Common causes of this problem are:
1. Invalid username and password
2. Incorrect SSH-key file
3. Connecting to the wrong device
Device settings: cisco_xe 3.121.222.37:22
What I'm missing here?

Found the solution, this was missing in the inventory file:
connection_options:
netmiko:
extras:
use_keys: true
key_file: "/home/coder/.ssh/id_rsa"

Airflow Can't Connect to SFTP with SSH Private Key

I am trying to use the SFTP Operator/Hook with a SSH private key instead of a password.
My connection has the following extra arguments: {"key_file": "/home/airflow/key_files/some_key", "no_host_key_check": "true"}. My SSH Private key looks like the following:
-----BEGIN OPENSSH PRIVATE KEY-----
...
-----END OPENSSH PRIVATE KEY-----
Now when I try to create a hook I get the following error:
>>> from airflow.contrib.hooks.sftp_hook import SFTPHook
>>> s = SFTPHook(ftp_conn_id='my_sftp_connection')
[2020-07-08 13:40:25,449] {base_hook.py:84} INFO - Using connection to: id: my_sftp_connection. Host: my_sftp.com, Port: 22, Schema: None, Login: user, Password: None, extra: {'key_file': '/home/airflow/key_files/some_key', 'no_host_key_check': 'true'}
[2020-07-08 13:40:25,452] {base_hook.py:84} INFO - Using connection to: id: my_sftp_connection. Host: my_sftp.com, Port: 22, Schema: None, Login: user, Password: None, extra: {'key_file': '/home/airflow/key_files/some_key', 'no_host_key_check': 'true'}
>>> s.list_directory('/')
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/pysftp/__init__.py", line 167, in _set_authentication
private_key_file, private_key_pass)
File "/home/airflow/.local/lib/python3.6/site-packages/paramiko/pkey.py", line 206, in from_private_key_file
key = cls(filename=filename, password=password)
File "/home/airflow/.local/lib/python3.6/site-packages/paramiko/rsakey.py", line 55, in __init__
self._from_private_key_file(filename, password)
File "/home/airflow/.local/lib/python3.6/site-packages/paramiko/rsakey.py", line 175, in _from_private_key_file
data = self._read_private_key_file("RSA", filename, password)
File "/home/airflow/.local/lib/python3.6/site-packages/paramiko/pkey.py", line 279, in _read_private_key_file
data = self._read_private_key(tag, f, password)
File "/home/airflow/.local/lib/python3.6/site-packages/paramiko/pkey.py", line 289, in _read_private_key
raise SSHException("not a valid " + tag + " private key file")
paramiko.ssh_exception.SSHException: not a valid RSA private key file
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/hooks/sftp_hook.py", line 151, in list_directory
conn = self.get_conn()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/contrib/hooks/sftp_hook.py", line 114, in get_conn
self.conn = pysftp.Connection(**conn_params)
File "/home/airflow/.local/lib/python3.6/site-packages/pysftp/__init__.py", line 142, in __init__
self._set_authentication(password, private_key, private_key_pass)
File "/home/airflow/.local/lib/python3.6/site-packages/pysftp/__init__.py", line 171, in _set_authentication
private_key_file, private_key_pass)
File "/home/airflow/.local/lib/python3.6/site-packages/paramiko/pkey.py", line 206, in from_private_key_file
key = cls(filename=filename, password=password)
File "/home/airflow/.local/lib/python3.6/site-packages/paramiko/dsskey.py", line 65, in __init__
self._from_private_key_file(filename, password)
File "/home/airflow/.local/lib/python3.6/site-packages/paramiko/dsskey.py", line 224, in _from_private_key_file
data = self._read_private_key_file("DSA", filename, password)
File "/home/airflow/.local/lib/python3.6/site-packages/paramiko/pkey.py", line 279, in _read_private_key_file
data = self._read_private_key(tag, f, password)
File "/home/airflow/.local/lib/python3.6/site-packages/paramiko/pkey.py", line 289, in _read_private_key
raise SSHException("not a valid " + tag + " private key file")
paramiko.ssh_exception.SSHException: not a valid DSA private key file
What am I doing wrong here? That same SSH key works with WINSCP and the SFTP command in bash.

Think I figured it out. I upgraded that paramiko library and now it appears to be working. Will leave this as unanswered in case there was something else I was doing wrong.

I have the exact same problem, except that I'm using Composer, which is Airflow-based managed service in Google Cloud and I do not use the SFTP contrib operator.
Instead I use the apache-airflow-backport-providers-sftp which is backported from Airflow 2.0 and is compatible with my Airflow environment, which is 1.10.14.
It works fine with a user/password, but fails with an RSA key.
The contrib operator and the provider package are equivalent, and depend on the same Python modules: paramiko, pysftp, and sshtunnel.
My pip freeze:
paramiko==2.7.2 (latest release)
pysftp==0.2.9 (latest release)
sshtunnel==0.1.5 (latest release is 0.4.0)
In my understanding, and following your description, the problem is not in the top module (contrib or provider). You said that the problem is in paramiko but it works fine for me with the SSH operator and a RSA key. So, maybe it is in pysftp...
I am lost here.

How to fetch sizes of all SFTP files in a directory through Paramiko

import paramiko
from socket import error as socket_error
import os
server =['10.10.0.1','10.10.0.2']
path='/home/test/'
for hostname in server:
try:
ssh_remote =paramiko.SSHClient()
ssh_remote.set_missing_host_key_policy(paramiko.AutoAddPolicy())
privatekeyfile = os.path.expanduser('~/.ssh/id')
mykey = paramiko.RSAKey.from_private_key_file(privatekeyfile, password='test123')
ssh_remote.connect(hostname, username = 'test1', pkey = mykey)
sftp=ssh_remote.open_sftp()
for i in sftp.listdir(path):
info = sftp.stat(i)
print info.st_size
except paramiko.SSHException as sshException:
print "Unable to establish SSH connection:{0}".format(hostname)
except socket_error as socket_err:
print "Unable to connect connection refused"
This is my code. I tried to get file size of remote server files. But below error was throwing. Can some please guide on this?
Error
Traceback (most recent call last):
File "<stdin>", line 15, in <module>
File "/usr/lib/python2.6/site-packages/paramiko/sftp_client.py", line 337, in stat
t, msg = self._request(CMD_STAT, path)
File "/usr/lib/python2.6/site-packages/paramiko/sftp_client.py", line 624, in _request
return self._read_response(num)
File "/usr/lib/python2.6/site-packages/paramiko/sftp_client.py", line 671, in _read_response
self._convert_status(msg)
File "/usr/lib/python2.6/site-packages/paramiko/sftp_client.py", line 697, in _convert_status
raise IOError(errno.ENOENT, text)
IOError: [Errno 2] No such file

SFTPClient.listdir returns file names only, not a full path. So to use the filename in another API, you have to add a path:
for i in sftp.listdir(path):
info = sftp.stat(path + "/" + i)
print info.st_size
Though that's inefficient. Paramiko knows the size already, you are just throwing the information away by using SFTPClient.listdir instead of SFTPClient.listdir_attr (listdir calls listdir_attr internally).
for i in sftp.listdir_attr(path):
print i.st_size

How use Connection in Fabric 2?

I'm trying to connect to host and run command with module Fabric 2 and have this error:
Traceback (most recent call last):
File "Utilities/fabfile.py", line 4, in <module>
res.run('uname -s')
File "<decorator-gen-3>", line 2, in run
File "/usr/local/lib/python2.7/dist-packages/fabric/connection.py", line 29, in opens
self.open()
File "/usr/local/lib/python2.7/dist-packages/fabric/connection.py", line 501, in open
self.client.connect(**kwargs)
File "/home/trishnevskaya/.local/lib/python2.7/site-packages/paramiko/client.py", line 424, in connect
passphrase,
File "/home/username/.local/lib/python2.7/site-packages/paramiko/client.py", line 715, in _auth
raise SSHException('No authentication methods available')
paramiko.ssh_exception.SSHException: No authentication methods available
Simple code from docs (http://docs.fabfile.org/en/latest/getting-started.html):
from fabric import Connection
res = Connection('<host-ip>')
res.run('uname -s')
Accoding to docs, I don't need in special configs, but it's doesn't work...
fabric 2.1.3
python 2.7.14

Following works for me.
connect_kwargs = {"key_filename":['PATH/KEY.pem']}
with Connection(host="EC2", user="ubuntu", connect_kwargs=connect_kwargs) as c:
c.run("mkdir abds")

I run into the same issue. Rather than passing a SSH keyfile, as suggested previously, another trivial way to sort it out might be to pass a password (that would be fine just over the test/development stage).
import getpass
from fabric import Connection, Config
sudo_pass = getpass.getpass("What's your user password?\n")
config = Config(overrides={'user': '<host-user>', 'connect_kwargs': {'password': sudo_pass}})
c = Connection('<host-ip>', config=config)
c.run('uname -s')

Connecting to Gmail from Python

I've been trying to connect to my Gmail account using python. imap is enabled.
import imaplib
imap_server = imaplib.IMAP4_SSL("imap.gmail.com",993)
# also tried imap_server = imaplib.IMAP4_SSL("imap.gmail.com"), doesnt work.
Traceback is :
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
imap_server = imaplib.IMAP4_SSL("imap.gmail.com",993)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/imaplib.py", line 1202, in __init__
IMAP4.__init__(self, host, port)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/imaplib.py", line 172, in __init__
self.open(host, port)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/imaplib.py", line 1217, in open
IMAP4.open(self, host, port)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/imaplib.py", line 248, in open
self.sock = self._create_socket()
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/imaplib.py", line 1205, in _create_socket
sock = IMAP4._create_socket(self)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/imaplib.py", line 238, in _create_socket
return socket.create_connection((self.host, self.port))
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socket.py", line 435, in create_connection
raise err
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socket.py", line 426, in create_connection
sock.connect(sa)
OSError: [Errno 65] No route to host

What OSError: [Errno 65] No route to host means is what it say: you can't get to that machine from your machine.
You can test that from outside of Python by opening up a terminal/DOS prompt and typing this:
ping imap.gmail.com
It's possible that this is actually a name lookup error, and you're somehow getting a bad address for imap.gmail.com. So, just to be sure, check by IP address too:
ping 74.125.129.108
ping 74.125.129.109
If ping works, you can check whether your router is for some reason just blocking TCP access to the host, e.g., with:
telnet imap.gmail.com
If it's working, this should either hang for a long time, or give you a connection-refused error; if it gives you a no-route-to-host error, it's the same problem you're seeing.
It's also possible that your router is specifically blocking port 993. You can test this too:
telnet imap.gmail.com 993
If it doesn't come back with something like "Connected to gmail-imap.l.google.com", same problem here too.
At any rate, once you've verified that this is a system or network configuration problem, not a programming problem, go ask for help with your system on the appropriate site.