Is anybody aware of a Python solution to sftp into the Bloomberg enterprise service. We're converting some bloomberg pulling from Mathworks to Python. Historically, I would have done this in Mathworks like this using the bdl function.
username = 'xxxxx';
password = 'xxxxxxxx';
hostname = 'dlsftp.bloomberg.com';
portnumber = 30206;
decrypt = 'nAcLeZ';
c = bdl(username,password,hostname,portnumber,decrypt)
Any thoughts? Thanks!
OK, for anybody that needs SOCKS proxy through Bloomberg, here is what I've done and it works:
After some research, it appears the decrypt is not necessary. This code works, just be sure to include the two files for a test: ('readme.txt','readme-test.txt') on your bloomberg server using RequestBuilder or WinSCP.
# coding: utf-8
import paramiko
import socket
import socks
proxy_details = {'host': "xxx",
'port': 1080,
'username': "xxx",
'password': "xxx"}
auth_credentials = {'host': "sftp.bloomberg.com",
'username': "dlxxxxxx",
'password': "xxxxxx"}
s = socks.socksocket()
s.set_proxy(
proxy_type=socks.SOCKS5,
addr=proxy_details['host'],
port=proxy_details['port'],
username=proxy_details['username'],
password=proxy_details['password']
)
#setup the SFTP client using the connected socket
s.connect((auth_credentials['host'],22))
transport = paramiko.Transport(s)
transport.connect(username=auth_credentials['username'],
password=auth_credentials['password'])
# start SFTP Client from SSH transport
sftp = paramiko.SFTPClient.from_transport(transport) #.get_transport() was appended
#will download readme.txt from the remote server, and save as the filename in the second argument
sftp.get('readme.txt','readme-test.txt')
#Test whether transport is authenticated
print (transport.is_authenticated())
# cleanup
sftp.close()
transport.close()
Related
i am getting really confused on how to connect to an ftp server while using a proxy in python. Some previous examples have confused me a lot as i am new to Python. Below are the variables i believe that will be needed in the code sample (the proxy is only a example for security reasons as is the username and password).
can someone please show me how to connect to the ftp server and download a file
the end goal is to use python to download files from this ftp server
import ftplib
proxy = {'http': "auth-proxy.xxxx.com:232", 'https': "auth-proxy.xxx.com:232"}
ftp_site = 'https://data.ftse.com'
username = 'xxxxxx'
password = 'yyyyyy'
#file path where the file is on the ftp server is below
filepath = 'https://data.ftse.com/filedownloadservlet?filename=eerf1808.csv&phyName=nXVi8I4q5Boy8gBpdpDg3jNBqfAZy0vfydRBHC3zs3qt6YadMA6xa9LM3QaIgx-g'
I read "python network programming" and got to learn about netmiko. I tried connecting to the Cisco network given in the book and it did not connect. I went online and read other articles on netmiko and tried their Examples using the router login details given in the book, but none of them worked. My internet connection is good (I checked it). Please what is wrong? Is it my location (I stay in Nigeria)?
The error given to me is as follows:
possible reasons why a connection cannot be established are as follows:
*Wrong hostname
*Wrong TCP/IP port
*Wrong password
*Blocked access to router
Please what is wrong, I need help. Or if you have any free router I can connect to just for the sake of learning I would like to know.
If you don't have direct ssh permission and you want to connect through a private server, you can also try paramiko instead of netmiko. Below I wrote an example of connecting via a server and entering the device by typing a command.
import paramiko
import time
from termcolor import colored
from timeit import default_timer as timer
import sys
global password
username = "ldap"
password = "pwd"
def vrf_implement_control(device_ip):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
remote_connection = ssh.connect(device_ip, port="2222", username=username, password=password, timeout=15)
remote_connection = ssh.invoke_shell()
# device_connection
remote_connection.send(" 192.168.0.10\n")
time.sleep(2)
print (colored ("connected_ip_address_"+ device_ip,"blue"))
except Exception as e:
print("erisim yok_\n")
I don't know which sample script you are trying. But as per your error, it looks like there is an issue with device connectivity, You can try to ping the device and check ssh is enabled or not.
Try to run the below script with your credentials, It should work.
from netmiko import ConnectHandler
cisco = {
'device_type': 'cisco_ios', #refer netmiko device type
'host': '10.10.1.1', #Replace with your device ip
'username': 'admin', #Router username
'password': 'cisco123', #password
}
net_connect = ConnectHandler(**cisco)
output = net_connect.send_command("show version")
print(output)
I am posting the question and the answer, I found, as well, incase it would help someone. The following were my minimum requirements:
1. Client machine is Windows 10 and remote server is Linux
2. Connect to remote server via SSH through HTTP Proxy
3. HTTP Proxy uses Basic Authentication
4. Run commands on remote server and display output
The purpose of the script was to login to the remote server, run a bash script (check.sh) present on the server and display the result. The Bash script simply runs a list of commands displaying the overall health of the server.
There have been numerous discussions, here, on how to implement HTTP Proxy or running remote commands using Paramiko. However, I could not find the combination of both.
from urllib.parse import urlparse
from http.client import HTTPConnection
import paramiko
from base64 import b64encode
# host details
host = "remote-server-IP"
port = 22
# proxy connection & socket definition
proxy_url = "http://uname001:passw0rd123#HTTP-proxy-server-IP:proxy-port"
url = urlparse(proxy_url)
http_con = HTTPConnection(url.hostname, url.port)
auth = b64encode(bytes(url.username + ':' + url.password,"utf-8")).decode("ascii")
headers = { 'Proxy-Authorization' : 'Basic %s' % auth }
http_con.set_tunnel(host, port, headers)
http_con.connect()
sock = http_con.sock
# ssh connection
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect(hostname=host, port=port, username='remote-server-uname', password='remote-server-pwd', sock=sock)
except paramiko.SSHException:
print("Connection Failed")
quit()
stdin,stdout,stderr = ssh.exec_command("./check")
for line in stdout.readlines():
print(line.strip())
ssh.close()
I would welcome any suggestions to the code as I am a network analyst and not a coder but keen to learn and improve.
I do not think that your proxy code is correct.
For a working proxy code, see How to ssh over http proxy in Python?, particularly the answer by #tintin.
As it seems that you need to authenticate to the proxy, after the CONNECT command, add a Proxy-Authorization header like:
Proxy-Authorization: Basic <credentials>
where the <credentials> is base-64 encoded string username:password.
cmd_connect = "CONNECT {}:{} HTTP/1.1\r\nProxy-Authorization: Basic <credentials>\r\n\r\n".format(*target)
I'm a little over my head when it comes to this SSH thing. Basically I am trying to access a friends server through and SSH tunnel using twisted conch. He has given me the following information:
MONGO_HOST = "ip address"
MONGO_DB = "server name"
MONGO_USER = "user name"
MONGO_PASS = "server password"
I was able to get this information to work using the python library motor.motor_asyncio (I need this be async compatible in order to use with other libraries) but for reasons that I can get into if necessary, will not work on the raspberry pi that I plan on running this program on.
Long story short, I was wondering if anyone could help me with some sample code to access my friends server using the information given above with twisted.conch.
I looked on the twisted.conch readthedocs, but the example needs more information than I can provide (I think) and is WAY over my head in terms of networking/SSH/etc.
Thanks in advance. I am willing to put in the work, but I need to know where to look.
Here is my relevant bit of code so far:
from motor.motor_asyncio import AsyncIOMotorClient
from sshtunnel import SSHTunnelForwarder
MONGO_HOST = "host address"
MONGO_DB = "server name"
MONGO_USER = "username"
MONGO_PASS = "password"
server = SSHTunnelForwarder(
MONGO_HOST,
ssh_username=MONGO_USER,
ssh_password=MONGO_PASS,
remote_bind_address=('address', gate),
local_bind_address=('address', gate)
)
server.start()
client = AsyncIOMotorClient('address', gate)
db = client.server_name
You can forward ports with Conch like this:
rom twisted.internet.defer import Deferred
from twisted.conch.scripts import conch
from twisted.conch.scripts.conch import ClientOptions, SSHConnection
from twisted.conch.client.direct import connect
from twisted.conch.client.default import SSHUserAuthClient, verifyHostKey
from twisted.internet.task import react
def main(reactor):
# authenticate as this user to SSH
user = "sshusername"
# the SSH server address
host = "127.0.0.1"
# the SSH server port number
port = 22
# a local port number to listen on and forward
localListenPort = 12345
# an address to forward to from the remote system
remoteForwardHost = "127.0.0.1"
# and the port number to forward to
remoteForwardPort = 22
conch.options = ClientOptions()
conch.options.parseOptions([
# don't ask the server for a shell
"--noshell",
# set up the local port forward
"--localforward={}:{}:{}".format(
localListenPort,
remoteForwardHost,
remoteForwardPort,
),
# specify the hostname for host key checking
host,
])
# don't stop when the last forwarded connection stops
conch.stopConnection = lambda: None
# do normal-ish authentication - agent, keys, passwords
userAuthObj = SSHUserAuthClient(user, conch.options, SSHConnection())
# create a Deferred that will tell `react` when to stop the reactor
runningIndefinitely = Deferred()
# establish the connection
connecting = connect(
host,
port,
conch.options,
verifyHostKey,
userAuthObj,
)
# only forward errors so the reactor will run forever unless the
# connection attempt fails. note this does not set up reconnection for a
# connection that succeeds and then fails later.
connecting.addErrback(runningIndefinitely.errback)
return runningIndefinitely
# run the reactor, call the main function with it, passing no other args
react(main, [])
Some of the APIs are weird because they are CLI focused. You don't have to do it this way but port forwarding is most easily accessible with these APIs rather than the APIs that are more focused on programmatic use.
How I can make SFTP transport through SSHClient on the remote server? I have a local host and two remote hosts. Remote hosts are backup server and web server. I need to find on backup server necessary backup file and put it on web server over SFTP. How can I make Paramiko's SFTP transport work with Paramiko's SSHClient?
paramiko.SFTPClient
Sample Usage:
import paramiko
paramiko.util.log_to_file("paramiko.log")
# Open a transport
host,port = "example.com",22
transport = paramiko.Transport((host,port))
# Auth
username,password = "bar","foo"
transport.connect(None,username,password)
# Go!
sftp = paramiko.SFTPClient.from_transport(transport)
# Download
filepath = "/etc/passwd"
localpath = "/home/remotepasswd"
sftp.get(filepath,localpath)
# Upload
filepath = "/home/foo.jpg"
localpath = "/home/pony.jpg"
sftp.put(localpath,filepath)
# Close
if sftp: sftp.close()
if transport: transport.close()
The accepted answer "works". But with its use of the low-level Transport class, it bypasses a host key verification, what is a security flaw, as it makes the code susceptible to Man-in-the-middle attacks.
Better is to use the right Paramiko SSH API, the SSHClient, which does verify the host key:
import paramiko
paramiko.util.log_to_file("paramiko.log")
ssh = paramiko.SSHClient()
ssh.connect(host, username='user', password='password')
# or
# key = paramiko.RSAKey.from_private_key_file('id_rsa')
# ssh.connect(host, username='user', pkey=key)
sftp = ssh.open_sftp()
sftp.get(remotepath, localpath)
# or
sftp.put(localpath, remotepath)
For details about verifying the host key, see:
Paramiko "Unknown Server"
If you have a SSHClient, you can also use open_sftp():
import paramiko
# lets say you have SSH client...
client = paramiko.SSHClient()
sftp = client.open_sftp()
# then you can use upload & download as shown above
...
In addition to the first answer which is great but depends on username/password, the following shows how to use an ssh key:
from paramiko import Transport, SFTPClient, RSAKey
key = RSAKey(filename='path_to_my_rsakey')
con = Transport('remote_host_name_or_ip', 22)
con.connect(None,username='my_username', pkey=key)
sftp = SFTPClient.from_transport(con)
sftp.listdir(path='.')
For those anyone need to integrate with an ssh/sftp server that requires a private key and want to perform host key verification for the known host by using a specific public key, here is a snippet code with paramiko:
import paramiko
sftp_hostname = "target.hostname.com"
sftp_username = "tartgetHostUsername"
sftp_private_key = "/path/to/private_key_file.pvt"
sftp_private_key_password = "private_key_file_passphrase_if_it_encrypted"
sftp_public_key = "/path/to/public_certified_file.pub"
sftp_port = 22
remote_path = "."
target_local_path = "/path/to/target/folder"
ssh = paramiko.SSHClient()
# Load target host public cert for host key verification
ssh.load_host_keys(sftp_public_key)
# Load encrypted private key and ssh connect
key = paramiko.RSAKey.from_private_key_file(sftp_private_key, sftp_private_key_password)
ssh.connect(host=sftp_hostname, port=sftp_port, username=sftp_username, pkey=key)
# Get the sftp connection
sftp_connection = ssh.open_sftp()
directory_list = sftp_connection.listdir(remote_path)
# ...
if sftp_connection: sftp_connection.close()
if ssh: ssh.close()
Notice that only certificates in classic Openssh format are supported, otherwise needs to be converted with the following commands (also for the latest Openssh formats):
$chmod 400 /path/to/private_key_file.pvt
$ssh-keygen -p -f /path/to/private_key_file.pvt -m pem -P <currentPassphrase> -N <newPassphrase>
In order to avoid man in the middle attack, it is important to do not use paramiko.AutoAddPolicy() and load the public host key programmatically as above or load it from ~/.ssh/known_hosts
The file must be in the format "<host_name> ssh-rsa AAAAB3NzaC1yc2EAAAA..."
In case you don't have the public key and you trust the target host (take care to mitm), you can download it using $ssh-keyscan target.hostname.com command.
The above code is the only way I found to avoid the following error during connection:
paramiko.ssh_exception.SSHException: Server 'x.y.z' not found in known_hosts
This error was prompted also with the following way to load the public certificates:
key = paramiko.RSAKey(data=decodebytes(sftp_public_key))
ssh_client.get_host_keys().add(sftp_hostname, 'ssh-rsa', key)
Also the following code was not able for me to load the certificate (tried also by encoding the certificate in base64):
ssh_client=paramiko.SSHClient()
rsa_key = paramiko.RSAKey.from_private_key_file(sftp_private_key, sftp_private_key_password)
rsa_key.load_certificate(sftp_public_key)
It always ends with:
File "/usr/local/lib/python3.9/site-packages/paramiko/pkey.py", line 720, in from_string
key_blob = decodebytes(b(fields[1]))
File "/usr/lib64/python3.9/base64.py", line 538, in decodebytes
return binascii.a2b_base64(s)
binascii.Error: Incorrect padding
The above code above worked for the SFTP integration with GoAnywhere.
I hope this is helpful, I've not found any working example and spent many hours in searches and tests.
The implementations using pysftp wrapper it is now to be considered as discontinued from 2016.