We Keep Coding

Python 3.9 SFTP connection

I'm trying to connect to SFTP server that is hosted locally (that's a POC ...). I'm using python 3.9 and the library paramiko.
The code I use is the following :
import paramiko
import sys
def main() -> None:
hostkeys = paramiko.hostkeys.HostKeys("known_hosts.txt")
ip = "127.0.0.1"
username = "my_username"
password = "my_password"
# As per the paramiko doc. The host fingerprint is stored using the ed25519 algorithm
hostFingerprint = hostkeys.lookup(ip)["ssh-ed25519"]
try:
tp = paramiko.Transport(ip, 22)
tp.connect(username=username, password=password, hostkey=hostFingerprint)
# Upload operations happening here...
tp.close()
except Exception as e:
print(e)
finally:
return None
The file known_hsts.txt has been generated via Powershell cmd : ssh-keyscan 127.0.0.1 > C:\Users\my_username\PycharmProjects\POC\known_hosts.txt.
When i run the script i don't understand why hostkeys.keys() == []
When i look at the file known_hosts.txt I see the different keys : ssh-rsa, ssh-ed25519, etc.

Downloading files using wildcard from SFTP server using Python Paramiko

I've a working code which works good when I mention the file name exactly how it is in folder. But these files have date and time added to its name. How can I make the file path for the same to read? Below is my code.
import paramiko
import os
paramiko.util.log_to_file('logfile.log')
host = "ftp.servername.com"
port = 22
transport = paramiko.Transport((host, port))
password = "mypass"
username = "myuser"
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
filepath = '/Home/user/Automation_2021-07-14_170139.csv'
localpath = 'file/dkc.csv'
sftp.get(filepath, localpath)
sftp.close()
transport.close()
How can I pass the * in the filepath? I wanted to make it like below.
filepath = '/Home/user/Automation_*.csv'
localpath = 'file/dkc.csv'
sftp.get(filepath, localpath)

You cannot pass a wildcard directly to Paramiko's SFTPClient.get.
You have to first find out the exact name and then use it with the get.
See List files on SFTP server matching wildcard in Python using Paramiko

Below code helped me to fix the issue.
latest = 0
latestfile = None
for fileattr in sftp.listdir_attr():
if fileattr.filename.startswith('Automation_DKC') and fileattr.st_mtime > latest:
latest = fileattr.st_mtime
latestfile = fileattr.filename

Deciphering .py Code

I am having a very specific problem. I use Putty for database management for my small business. We recently had an update and our normal command path to update our records is no longer in use.
We run Putty on all computers in store. Putty is used on a virtual machine with oracle. We have a main computer for the system in which the update occurred.
We normally input ~/Desktop/getdata.sh into putty using root user and it complies an updated list, creates a text file that we use. Unfortunately, the person who created this script no longer works with us.
I am trying to find a way to re execute this file again.
After the update, when we type in ~/Desktop/getdata.sh (after logging in as root) into Putty we get 'directory cannot be found.' I've searched everyday to find this file. However, I did find a getdata.py file and a getdata.bat files.
I can show both scripts if needed, I can update the question.
When I tried to run getdata.py I get
[root#RT_Store-01 /]# ~/Desktop/getdata.py
import: unable to open X server `'.
import: unable to open X server `'.
import: unable to open X server `'.
import: unable to open X server `'.
: command not foundta.py: line 5:
/root/Desktop/getdata.py: line 6: from: command not found
/root/Desktop/getdata.py: line 7: from: command not found
: command not foundta.py: line 8:
/root/Desktop/getdata.py: line 9: syntax error near unexpected token `('
/root/Desktop/getdata.py: line 9: `dir_path = os.path.dirname(os.path.realpath(_'file__))
Do I need to convert my files to .sh? How would I do that? Is this a bigger problem?
The script for getdata.py is
import os
import tempfile
import paramiko
import time
from pywinauto import Application
from subprocess import Popen, PIPE, STDOUT
dir_path = os.path.dirname(os.path.realpath(__file__))
class Connection(object):
"""Connects and logs into the specified hostname.
Arguments that are not given are guessed from the environment."""
def __init__(self,
host,
username=None,
private_key=None,
password=None,
port=22,
):
self._sftp_live = False
self._sftp = None
if not username:
username = os.environ['LOGNAME']
# Log to a temporary file.
templog = tempfile.mkstemp('.txt', 'ssh-')[1]
paramiko.util.log_to_file(templog)
# Begin the SSH transport.
self._transport = paramiko.Transport((host, port))
self._tranport_live = True
# Authenticate the transport.
if password:
# Using Password.
self._transport.connect(username=username, password=password)
else:
# Use Private Key.
if not private_key:
# Try to use default key.
if os.path.exists(os.path.expanduser('~/.ssh/id_rsa')):
private_key = '~/.ssh/id_rsa'
elif os.path.exists(os.path.expanduser('~/.ssh/id_dsa')):
private_key = '~/.ssh/id_dsa'
else:
raise TypeError(
"You have not specified a password or key.")
private_key_file = os.path.expanduser(private_key)
rsa_key = paramiko.RSAKey.from_private_key_file(private_key_file)
self._transport.connect(username=username, pkey=rsa_key)
def _sftp_connect(self):
"""Establish the SFTP connection."""
if not self._sftp_live:
self._sftp = paramiko.SFTPClient.from_transport(self._transport)
self._sftp_live = True
def get(self, remotepath, localpath=None):
"""Copies a file between the remote host and the local host."""
if not localpath:
localpath = os.path.split(remotepath)[1]
self._sftp_connect()
self._sftp.get(remotepath, localpath)
def put(self, localpath, remotepath=None):
"""Copies a file between the local host and the remote host."""
if not remotepath:
remotepath = os.path.split(localpath)[1]
self._sftp_connect()
self._sftp.put(localpath, remotepath)
def execute(self, command):
"""Execute the given commands on a remote machine."""
channel = self._transport.open_session()
channel.exec_command(command)
output = channel.makefile('rb', -1).readlines()
if output:
return output
else:
return channel.makefile_stderr('rb', -1).readlines()
def update(self):
"""Execute the given commands on a remote machine."""
channel = self._transport.invoke_shell(term='xterm')
channel.exec_command('~/Desktop/update.sh')
output = channel.makefile('rb', -1).readlines()
if output:
return output
else:
return channel.makefile_stderr('rb', -1).readlines()
def close(self):
"""Closes the connection and cleans up."""
# Close SFTP Connection.
if self._sftp_live:
self._sftp.close()
self._sftp_live = False
# Close the SSH Transport.
if self._tranport_live:
self._transport.close()
self._tranport_live = False
def __del__(self):
"""Attempt to clean up if not explicitly closed."""
self.close()
def getData():
"""Create, get, and put delim file when called directly."""
app = Application().start(r"c:\putty.exe trak#10.1.10.70 -pw trak")
app.window_(
title_re=".*trak.*").TypeKeys("/home/trak/Desktop/getdata.sh && exit{ENTER}")
app.window_(title_re=".*trak.*").WaitNot("exists", timeout=120)
trakfile = dir_path + '/storage/trakdelim.txt'
shell = Connection('10.1.10.70', "trak", password="trak")
shell.get('/trak/data/trakdelim.txt', trakfile)
shell.close()
if __name__ == "__main__":
getData()
I appreciate anyone who can help and I can clarify when needed!

A bit of googling showed me what this is:
Someone copied this code and wrote a simple script with it that takes a file (/trak/data/trakdelim.txt) on a remote computer with ip address 10.1.10.70 username trak and password trak and copies it to the storage/trakdelim.txt file. if this is not working for you now then take a tool that allows you to do this manually with such as winSCP and use that instead.
Good luck.

You need to execute it as a python script.
python ~/Desktop/getdata.py

python paramiko error in Django: 'str' object has no attribute 'get_name'

So I've heard that it's an incredibly bad idea to host media on the same server as your Django/Apache web framework. So to solve this problem with my iOS app and greatly improve server side performance in the long run would be to upload the images to the Django server and then right after that, transfer the newly uploaded image to a separate server dedicated to hosting user profile images.
So using Paramiko I can send files directly to the directory of my choosing, but this does not seem to be working in my Django view.
Here's an incredible simple python script using Paramiko that allows you to upload files to a remote server via SMTP:
import base64
import getpass
import os
import socket
import sys
import traceback
import paramiko
#from paramiko.py3compat import input
# setup logging
#paramiko.util.log_to_file('demo_sftp.log')
port = 22
hostname = '198.199.101.115'
password = 'XXXXXXXXX'
username = 'root'
hostkeytype = 'ecdsa-sha2-nistp256'
hostkey = 'XXXXXXXXX'
try:
host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
except IOError:
try:
# try ~/ssh/ too, because windows can't have a folder named ~/.ssh/
host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/ssh/known_hosts'))
except IOError:
print('*** Unable to open host keys file')
host_keys = {}
if hostname in host_keys:
hostkeytype = host_keys[hostname].keys()[0]
hostkey = host_keys[hostname][hostkeytype]
print('Using host key of type %s' % hostkeytype)
# now, connect and use paramiko Transport to negotiate SSH2 across the connection
t = paramiko.Transport((hostname, port))
t.connect(username=username, password=password, hostkey=hostkey)
sftp = paramiko.SFTPClient.from_transport(t)
# dirlist on remote host
dirlist = sftp.listdir('.')
print("Dirlist: %s" % dirlist)
# copy this demo onto the server
target_directory = '3COOL'
sftp.put('test_image.jpg', target_directory+'/test_image.jpg')
t.close()
So this script works perfectly fine when I run the script on my remote server which successfully transfers the jpeg image to another remote server I own. So I'd like to just cut and paste this script into my Django view which will transfer all freshly uploaded images to a different server, here's how this looks:
def profile_picture(request):
if request.POST:
form = UserProfileForm(request.POST, request.FILES)
obj = form.save(commit=False)
obj.user_id = request.user.id
obj.profile_picture = obj.profile_picture
check = UserProfile.objects.filter(user_id=request.user.id)
if check:
oldup = UserProfile.objects.get(user_id=request.user.id)
oldup.delete()
obj.save()
formNew = UserProfileForm()
args = {}
args.update(csrf(request))
args['uid'] = request.user.id
args['form'] = formNew
# CONVERT THE IMAGE TO A SMALLER SIZE
basewidth = 256
img = Image.open('var/www/bitcraft/static/'+str(obj.profile_picture))
wpercent = (basewidth/float(img.size[0]))
hsize = int((float(img.size[1])*float(wpercent)))
img = img.resize((basewidth,hsize), PIL.Image.ANTIALIAS)
img.save('var/www/bitcraft/static/'+str(obj.profile_picture))
import getpass
import os
import socket
import sys
import traceback
import paramiko
from paramiko.py3compat import input
# TRANSFER THIS IMAGE TO MEDIA HOSTING SERVER
port = 22
hostname = '198.199.101.115'
password = 'pcorysatqwrw'
username = 'root'
hostkeytype = 'ecdsa-sha2-nistp256'
hostkey = 'pcorysatqwrw'
try:
host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
except IOError:
try:
host_keys = paramiko.util.load_host_keys(os.path.expanduser('~/ssh/known_hosts'))
except IOError:
print('*** Unable to open host keys file')
host_keys = {}
if hostname in host_keys:
hostkeytype = host_keys[hostname].keys()[0]
hostkey = host_keys[hostname][hostkeytype]
print('Using host key of type %s' % hostkeytype)
t = paramiko.Transport((hostname, port))
t.connect(username=username, password=password, hostkey=hostkey)
sftp = paramiko.SFTPClient.from_transport(t)
dirlist = sftp.listdir('.')
print("Dirlist: %s" % dirlist)
target_directory = '3COOL'
sftp.put('var/www/bitcraft/static/'+str(obj.profile_picture), target_directory+str(obj.profile_picture))
t.close()
return render_to_response('profile.html', args, RequestContext(request))
else:
formNew = UserProfileForm()
args = {}
args.update(csrf(request))
args['uid'] = request.user.id
args['form'] = formNew
return render_to_response('profile.html', args, RequestContext(request))
Ultimately though, adding the simple Python script has not worked due to a strange error:
AttributeError at /upload_profile/ 'str' object has no attribute 'get_name'
Exception Location: /usr/local/lib/python2.7/dist-packages/paramiko-1.14.0-py2.7.egg/paramiko/transport.py in connect, line 873
/srv/www/django/chatfeed/views.py in profile_picture
t.connect(username=username, password=password, hostkey=hostkey)
I don't understand what's causing this... is this related to directory permission denied??

The problem is that you're passing a string for hostkey. As the docs show, that's supposed to be a PKey—that is, an object that wraps a private key.
And if you click on PKey, you'll see that a PKey has a get_name method. A string obviously doesn't. Hence the error.
You have some code that's supposed to use Paramiko to load a hostkey out of ~/.ssh/known_hosts at the top of your script. But if that hostname in host_keys is false, it won't do anything, and you'll end up with the default values you stuck at the top of the script:
hostkeytype = 'ecdsa-sha2-nistp256'
hostkey = 'XXXXXXXXX'
I don't know what you have in place of that 'XXXXXXXXX' in your real code, but presumably it's a string, and therefore not a valid hostkey.
I don't understand what's causing this... is this related to directory permission denied??
You'd have to tell us what you're talking about. Exactly what error or warning or whatever are you getting, and where? If the server is running as a user who doesn't have access to its own ~/.ssh, so you're seeing a "directory permission denied" warning of some kind and your host keys aren't getting loaded, then yes, this problem is indirectly related to that one—as in, if the host actually was in known_keys, and you solved the permissions problem, that would mask the error in your code, so you wouldn't notice it. But there are plenty of other things you could mean where the answer would be no, that has nothing to do with anything.

key = paramiko.RSAKey.from_private_key_file(filename='openssh key f',password='passphrase')
The keys file does not support .ppk format.

SFTP in Python? (platform independent)

I'm working on a simple tool that transfers files to a hard-coded location with the password also hard-coded. I'm a python novice, but thanks to ftplib, it was easy:
import ftplib
info= ('someuser', 'password') #hard-coded
def putfile(file, site, dir, user=(), verbose=True):
"""
upload a file by ftp to a site/directory
login hard-coded, binary transfer
"""
if verbose: print 'Uploading', file
local = open(file, 'rb')
remote = ftplib.FTP(site)
remote.login(*user)
remote.cwd(dir)
remote.storbinary('STOR ' + file, local, 1024)
remote.quit()
local.close()
if verbose: print 'Upload done.'
if __name__ == '__main__':
site = 'somewhere.com' #hard-coded
dir = './uploads/' #hard-coded
import sys, getpass
putfile(sys.argv[1], site, dir, user=info)
The problem is that I can't find any library that supports sFTP. What's the normal way to do something like this securely?
Edit: Thanks to the answers here, I've gotten it working with Paramiko and this was the syntax.
import paramiko
host = "THEHOST.com" #hard-coded
port = 22
transport = paramiko.Transport((host, port))
password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
transport.connect(username = username, password = password)
sftp = paramiko.SFTPClient.from_transport(transport)
import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
sftp.put(localpath, path)
sftp.close()
transport.close()
print 'Upload done.'
Thanks again!

Paramiko supports SFTP. I've used it, and I've used Twisted. Both have their place, but you might find it easier to start with Paramiko.

You should check out pysftp https://pypi.python.org/pypi/pysftp it depends on paramiko, but wraps most common use cases to just a few lines of code.
import pysftp
import sys
path = './THETARGETDIRECTORY/' + sys.argv[1] #hard-coded
localpath = sys.argv[1]
host = "THEHOST.com" #hard-coded
password = "THEPASSWORD" #hard-coded
username = "THEUSERNAME" #hard-coded
with pysftp.Connection(host, username=username, password=password) as sftp:
sftp.put(localpath, path)
print 'Upload done.'

Here is a sample using pysftp and a private key.
import pysftp
def upload_file(file_path):
private_key = "~/.ssh/your-key.pem" # can use password keyword in Connection instead
srv = pysftp.Connection(host="your-host", username="user-name", private_key=private_key)
srv.chdir('/var/web/public_files/media/uploads') # change directory on remote server
srv.put(file_path) # To download a file, replace put with get
srv.close() # Close connection
pysftp is an easy to use sftp module that utilizes paramiko and pycrypto. It provides a simple interface to sftp.. Other things that you can do with pysftp which are quite useful:
data = srv.listdir() # Get the directory and file listing in a list
srv.get(file_path) # Download a file from remote server
srv.execute('pwd') # Execute a command on the server
More commands and about PySFTP here.

If you want easy and simple, you might also want to look at Fabric. It's an automated deployment tool like Ruby's Capistrano, but simpler and of course for Python. It's build on top of Paramiko.
You might not want to do 'automated deployment' but Fabric would suit your use case perfectly none the less. To show you how simple Fabric is: the fab file and command for your script would look like this (not tested, but 99% sure it will work):
fab_putfile.py:
from fabric.api import *
env.hosts = ['THEHOST.com']
env.user = 'THEUSER'
env.password = 'THEPASSWORD'
def put_file(file):
put(file, './THETARGETDIRECTORY/') # it's copied into the target directory
Then run the file with the fab command:
fab -f fab_putfile.py put_file:file=./path/to/my/file
And you're done! :)

fsspec is a great option for this, it offers a filesystem like implementation of sftp.
from fsspec.implementations.sftp import SFTPFileSystem
fs = SFTPFileSystem(host=host, username=username, password=password)
# list a directory
fs.ls("/")
# open a file
with fs.open(file_name) as file:
content = file.read()
Also worth noting that fsspec uses paramiko in the implementation.

With RSA Key then refer here
Snippet:
import pysftp
import paramiko
from base64 import decodebytes
keydata = b"""AAAAB3NzaC1yc2EAAAADAQABAAABAQDl"""
key = paramiko.RSAKey(data=decodebytes(keydata))
cnopts = pysftp.CnOpts()
cnopts.hostkeys.add(host, 'ssh-rsa', key)
with pysftp.Connection(host=host, username=username, password=password, cnopts=cnopts) as sftp:
with sftp.cd(directory):
sftp.put(file_to_sent_to_ftp)

Twisted can help you with what you are doing, check out their documentation, there are plenty of examples. Also it is a mature product with a big developer/user community behind it.

There are a bunch of answers that mention pysftp, so in the event that you want a context manager wrapper around pysftp, here is a solution that is even less code that ends up looking like the following when used
path = "sftp://user:p#ssw0rd#test.com/path/to/file.txt"
# Read a file
with open_sftp(path) as f:
s = f.read()
print s
# Write to a file
with open_sftp(path, mode='w') as f:
f.write("Some content.")
The (fuller) example: http://www.prschmid.com/2016/09/simple-opensftp-context-manager-for.html
This context manager happens to have auto-retry logic baked in in the event you can't connect the first time around (which surprisingly happens more often than you'd expect in a production environment...)
The context manager gist for open_sftp: https://gist.github.com/prschmid/80a19c22012e42d4d6e791c1e4eb8515

Paramiko is so slow. Use subprocess and shell, here is an example:
remote_file_name = "filename"
remotedir = "/remote/dir"
localpath = "/local/file/dir"
ftp_cmd_p = """
#!/bin/sh
lftp -u username,password sftp://ip:port <<EOF
cd {remotedir}
lcd {localpath}
get {filename}
EOF
"""
subprocess.call(ftp_cmd_p.format(remotedir=remotedir,
localpath=localpath,
filename=remote_file_name
),
shell=True, stdout=sys.stdout, stderr=sys.stderr)

PyFilesystem with its sshfs is one option. It uses Paramiko under the hood and provides a nicer paltform independent interface on top.
import fs
sf = fs.open_fs("sftp://[user[:password]#]host[:port]/[directory]")
sf.makedir('my_dir')
or
from fs.sshfs import SSHFS
sf = SSHFS(...

Here's a generic function that will download any given sftp url to a specified path
from urllib.parse import urlparse
import paramiko
url = 'sftp://username:password#hostname/filepath.txt'
def sftp_download(url, dest):
url = urlparse(url)
with paramiko.Transport((url.hostname, 22)) as transport:
transport.connect(None,url.username,url.password)
with paramiko.SFTPClient.from_transport(transport) as sftp:
sftp.get(url.path, dest)
Call it with
sftp_download(url, "/tmp/filepath.txt")