Send matching/wildcard file to another server using sftp.put in Python - python

So i am trying to send a certain file from my local server to another server. I am able to send the file if i know the file with exact name.
Bur what i actually want to do is pick up a file having a matching name and send the same file over.
For example my filename is filename_: test_file_20190918
i want to pick up all the file matching test_file_*
Here's what i am trying to do, but it doesn't seem to be working
import paramiko
ssh=paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname='host_name',username='username',password='pwd',port=22)
sftp_client=ssh.open_sftp()
sftp_client.put("/home/mylocation/test_file_*",'/incoming/test_file_send*')
sftp_client.close()
ssh.close()

Paramiko does not support wildcards.
You have to find the right file yourself, before calling SFTPClient.put.
See Get a filtered list of files in a directory.
Obligatory warning: Do not use AutoAddPolicy, unless you do not care about security. You are losing a protection against MITM attacks this way.
For a correct solution, see Paramiko "Unknown Server".

Related

Make a single SSH connection via paramiko and send a command to check the hostname and a 2nd command to see if there is anything in the www directory

Your manager just asked you to find out what is the hostname (/etc/hostname) and if there are any files in the www directory. Your manager wants you to check this on two Ubuntu-based machines. However, they want you to have this written in a way that will scale to all 100 servers on the network. Achieve the following items:
Make a Flowchart.
Create Pseudocode.
Make a single SSH connection via paramiko and send a command to check the hostname and a second command to see if there is anything in the www directory.
Input the IP addresses from a file (text or CSV).
Save the output of both commands, make sure to note what IP the data came from, to a file (text or CSV).
As mirlinda said, this is a set of requirement rather than a question. I just have a plan for your information.
You can write all ip address to text file with one ip per line.
Use ssh-keygen -t rsa -C "your mail" command to create ssh rsa key in your 100 servers.
Run ssh-copy-id -i ~/.ssh/id_rsa.pub root#[ip] to make your administrator machine can access which server you want to access without password.
Write a python script to read one ip per time from the text file saved your all server ip by readline interface.
And use check_output interface to run hostname command to get server host name and ls [full path of www] to get contents of www directory.
You just can save return contents of check_output interface to a file by redirection symbol '>'.
Loop operations 4-6 in your python script by using while or for statement until get all host name and contents of www of all 100 servers.

How to take data from SFTP server without downloading files in Python?

I have an SFTP server. I can take data by transferring/downloading files. Is there a way that I can do without downloading files?
My code is as below:
# Connection to the SFTP server
with pysftp.Connection(hostname, username, passowrd, port) as sftp:
with sftp.cd('directory'):
sftp.get('filename.txt')
This code downloads file to my local machine.
Yes and no. You can use the data from the remote (SFTP) server without storing the files to a local disk.
But you cannot use data locally without downloading them. That's impossible. You have to transfer the data to use them – at least to a memory of the local machine.
See A way to load big data on Python from SFTP server, not using my hard disk.
My answer there talks about Paramiko. But pysftp is a just a thin wrapper around Paramiko. Its Connection.open is directly mapped to underlying Paramiko's SFTPClient.open. So you can keep using pysftp:
with sftp.open('filename.txt', bufsize=32768) as f:
# use f as if you have opened a local file with open()
Though I'd recommend you not to: pysftp vs. Paramiko.

Paramiko SFTP file renaming: OSError('Extended request not supported.')

I can't find anything about this exception. I am trying to rename a remote file on a local (Windows) SFTP server with fsspec. Paramiko behind the scenes is doing a posix_rename(). What does the error mean?
fs.rename(old_file_path, new_file_path)
Paths look like /folder/file.ext.
I can rename files with other FTP clients on that same server.
Indeed, fsspec SFTPFileSystem.mv calls Paramiko SFTPClient.posix_rename. That's imo a bad choice. The SFTPClient.posix_rename internally uses a proprietary OpenSSH posix-rename#openssh.com extension, which is naturally not supported by most other SFTP servers (such as yours).
I do not know what is the best solution/workaround. You can probably add your own "file system" implementation based on SFTPFileSystem, reimplementing SFTPFileSystem.mv to call standard Paramiko SFTPClient.rename (which uses standard SFTP rename request).
Actually, I just found that the SFTPClient is exposed through the SFTPFileSystem and I can call rename() on it directly, which worked!
fs.ftp.rename("testfile.txt", "x")

Using ssh_config file with Paramiko

I need to connect to a server with SSH to download files. I have Ubuntu and I've set up SSH in the standard way: I have a ssh_config file in .ssh which defines a host entry (say host_key) for the server address (Hostname.com) and username, and I've set up an RSA key. So when I try to log into SSH from the command line or bash, I just need to use ssh host_key
I would like to do this in Python. The standard solutions seems to be to use Paramiko to set up the connection. I tried this:
from paramiko import SSHClient
from scp import SCPClient
ssh = SSHClient()
ssh.load_system_host_keys()
ssh.connect('host_key')
scp = SCPClient(ssh.get_transport())
# etc...
However, it always seems to hang and time out on ssh.connect('host_key'). Even when I try to include the username and password: ssh.connect('host_key', username='usrnm', password='pswd').
Are my host keys not loading properly? And would this take care of the RSA keys as well?
It only works if I use the whole Hostname.com with username and typed-out password. Which is maybe a bit insecure.
Since paramiko has a SSHConfig class, you can use it for your ~/.ssh/config.
However, it is slightly messy, I recommend you to use fabric instead of that.
Here is the code example:
from fabric.api import put
put('local path', 'remote path')
I do not think that it is common to use ssh_config file with Paramiko (or any other code/language). ssh_config is a configuration file for OpenSSH tools, not for SSH in general.
Usually, you specify your private key directly in your code as an argument of SSHClient.connect method:
How to access to a remote server using Paramiko with a public key-file
If you want to keep using ssh_config, Paramiko can parse it. Check parse_ssh_config and lookup_ssh_host_config functions. But I believe, you still have to look up the key file from the config and pass it explicitly to SSHClient.connect method.

How can I use sftp and mput with Python?

I'm trying to send over multiple files from one server to another using Python. I've found a few ssh2 libraries, but either I can't find documentation on them (e.g. ssh), or they don't seem to support mput.
Anyone know of any sftp libraries which support mput?
Paramiko is a library that handles SSH and related things, such as SFTP, but it only supports a regular put, no mput that I can see.
What exactly does mput do? My sftp client doesn't have that command...
Guessing from the name, I'm thinking "multiple puts" or something like that, to send multiple files in one go? If that is the case, I suggest just looping over your list of files and using put.
I was faced with a similar problem a long time ago and could not get a satisfactory mput-method to run in Python. So the paramiko library seemed to make the most sense to me.
To enable progress output or other actions in your Python application, it is advantageous to send the files individually in a for-loop. However, the overhead increases minimally with this variant.
A small Paramiko-example code :
import pysftp
import os
list_files_to_transfer = []
# Check if List is empty
if list_files_to_transfer:
# Advanced connection options beyond authentication
cnopts = pysftp.CnOpts()
# Compression for lower Transfer-Load
cnopts.compression = True
cnopts.hostkeys = None
# Establish a connection with the SFTP server
with pysftp.Connection(host=host, username=username, port=port,
private_key=os.path.abspath(path_private_key),
cnopts=cnopts) as sftp:
# Change to the specified remote directory
with sftp.cd(self.path_remote_sink_folder):
# browse the list of files
for file in list_files_to_transfer:
# Upload the file to the server
sftp.put(file)

Categories