We Keep Coding

Python 3.9 SFTP connection

I'm trying to connect to SFTP server that is hosted locally (that's a POC ...). I'm using python 3.9 and the library paramiko.
The code I use is the following :
import paramiko
import sys
def main() -> None:
hostkeys = paramiko.hostkeys.HostKeys("known_hosts.txt")
ip = "127.0.0.1"
username = "my_username"
password = "my_password"
# As per the paramiko doc. The host fingerprint is stored using the ed25519 algorithm
hostFingerprint = hostkeys.lookup(ip)["ssh-ed25519"]
try:
tp = paramiko.Transport(ip, 22)
tp.connect(username=username, password=password, hostkey=hostFingerprint)
# Upload operations happening here...
tp.close()
except Exception as e:
print(e)
finally:
return None
The file known_hsts.txt has been generated via Powershell cmd : ssh-keyscan 127.0.0.1 > C:\Users\my_username\PycharmProjects\POC\known_hosts.txt.
When i run the script i don't understand why hostkeys.keys() == []
When i look at the file known_hosts.txt I see the different keys : ssh-rsa, ssh-ed25519, etc.

How create port forwarding using SSHtunnelForwarder?

I am trying to replicate: ssh -i [KEY] -L [FROM_PORT]:localhost:[TO_PORT] ubuntu#[REMOTE_SERVER_IP] in python and decided to use sshtunnel for it. The command given above works and I can connect to the remote Theia IDE, but I can't figure out how I need to configure SSHtunnelForwarder to achieve the same from within python. I am probably misunderstanding its documentation: https://github.com/pahaz/sshtunnel
EDIT: This code is faulty. Look for an answer below.
# I replaced the remote-ip with `123.45.67.89`.
# I replaced my ip with `987.65.43.21`.
with SSHTunnelForwarder(
ssh_address_or_host="123.45.67.89", # ip of remote server
ssh_pkey="PRIVATE_KEY", # I am unsure which one is needed tried both
ssh_private_key="PRIVATE_KEY", # I am unsure which one is needed tried both
ssh_username="ubuntu", # username of remote server
ssh_bind_address= ("127.0.0.1", 9191), # where to listen locally
remote_bind_address= ("127.0.0.1", 8181) # where to send info coming to 22
# ssh-port 22 (default port)
# localhost:9191 -> 123.45.67.89:22 -> localhost:8181
# which is basically 987.65.43.21:9191 -> 123.45.67.89:22 -> 123.45.67.89:8181
) as server:
server.start()
But when I try to connect to http://localhost:9191/ I receive unable to connect; so the tunneling is probably broken. I added plenty of comments to the code to make it easier to spot my misunderstanding.

When I run your code then I see error
ValueError: You can't use both 'ssh_private_key' and 'ssh_pkey'. Please only use one of them.
You have to use one of them and example in your link show what it should be.
ssh_pkey="/var/ssh/rsa_key",
or
ssh_private_key_password="secret",
It needs to generate file rsa_key using program ssh-keygen which should be installed with ssh
(on Linux I would keep this file in user's folder ~/.ssh)
Next I see error
ValueError: Unknown arguments: {'ssh_bind_address': ('127.0.0.1', 9191)}
it has to be local_bind_address instead of ssh_bind_address.
Even example in your link uses local_bind_address.
You can see all arguments in commends in source code
or using help(SSHTunnelForwarder) in code.
This code works for me:
I can use pair username, password
ssh_username = "pi",
ssh_password = "raspberry",
or pair username and standard file id_rsa with my keys
ssh_username = "pi",
ssh_pkey='/home/furas/.ssh/id_rsa',
Code:
from sshtunnel import SSHTunnelForwarder
import time
#import sshtunnel
#print('author :', sshtunnel.__author__)
#print('version:', sshtunnel.__version__)
#help(SSHTunnelForwarder)
with SSHTunnelForwarder(
ssh_address_or_host = "192.168.1.29", # Raspberry Pi in my network
ssh_username = "pi",
ssh_password = "raspberry",
#ssh_username = "pi",
#ssh_pkey='/home/furas/.ssh/id_rsa',
local_bind_address = ("127.0.0.1", 9191),
remote_bind_address = ("127.0.0.1", 8181)
) as server:
print('Starting ...')
server.start()
print('Keep running ...')
while True:
time.sleep(1)
#print('Stopping ...')
#server.stop()

SSH Tunnel using Google Cloud Function

Is there a way to do something like the following in GCF?
import pymysql
import paramiko
from sshtunnel import SSHTunnelForwarder
def hello_world(request, data=''):
with SSHTunnelForwarder(ssh_address_or_host='56.48.28.32', ssh_username='ubuntu', ssh_pkey='id_rsa', remote_bind_address=('aval.myixal.net', 3306)) as tunnel:
conn = pymysql.connect(host='127.0.0.1', user='xian', passwd='bwQi', db='avails', port=tunnel.local_bind_port)
cursor = conn.cursor()
cursor.execute('SELECT * FROM global_event LIMIT 10')
data = str(cursor.fetchone())
conn.close()
return data
I have included the id_rsa key and the requirements. This will work locally, but I'm guessing the whole 'port-forwarding' thing might be a non-starter on GCF. Is there a way to do something like the above?

Totally possible. Quick test i did:
import io
import pymysql
import paramiko
from sshtunnel import SSHTunnelForwarder
def hello_world(request):
k = '''-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAB
...
zLFzhIqql0Av4K4XKQAAABVqYWJic29uQHRlc3QtaW5zdGFuY2UBAgME
-----END OPENSSH PRIVATE KEY-----'''
key = paramiko.RSAKey.from_private_key(io.StringIO(k))
with SSHTunnelForwarder(ssh_address_or_host='<ssh-ip-address>', ssh_username='<ssh-user-name>', ssh_pkey=key, remote_bind_address=('127.0.0.1', 3306)) as tunnel:
conn = pymysql.connect(host='127.0.0.1', user='<mysql-user>', passwd='<mysql-pass>', port=tunnel.local_bind_port)
cursor = conn.cursor()
cursor.execute('SELECT NOW();')
data = str(cursor.fetchone())
conn.close()
return data
result:
gcurl https://us-central1-my-project.cloudfunctions.net/function-1
(datetime.datetime(2021, 5, 13, 5, 4, 16),)
make sure not to hardcode your values like I did, instead use env variables.

Issues with connecting to Postgres DB via SSH in Python

I've been given a postgres DB in my uni project in which I have to SSH into the network from which I can then access the DB. I've set up the connection in DBeaver using the SSH tab and it works perfectly fine. However, using Python, I can connect to the SSH just fine, but cannot connect to the DB itself. I've checked with another DB that doesn't require SSH and that works just fine. Here is my code. Note: I've already tried using SSHTunnel, too, to no avail. Also, ignore my quick hack to anonymize my SSH login data, as I didn't find how to use a proper config file with paramiko late at night yesterday...
import os
from psycopg2 import connect, Error
from paramiko import SSHClient
from config import config
with open("ssh_config.txt", "r") as f:
lines = f.readlines()
hostname = lines[0].strip()
username = lines[1].strip()
password = lines[2].strip()
ssh = SSHClient()
ssh.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
ssh.connect(hostname=hostname, username=username, password=password)
print("SSH connected.")
try:
params = config()
conn = connect(**params)
cursor = conn.cursor()
print("DB connected.")
# Print PostgreSQL connection properties.
print(conn.get_dsn_parameters(), "\n")
# Print PostgreSQL version.
cursor.execute("SELECT version();")
record = cursor.fetchone()
print("You are connected to - ", record, "\n")
except (Exception, Error) as error:
print("Error while connecting to PostgreSQL", error)
Any help would be appreciated. Thanks!

I've figured it out myself. Here is the updated code. Basically, I had to forward the remote address to localhost and then connect to localhost instead of the DB address.
from psycopg2 import connect, Error
from sshtunnel import SSHTunnelForwarder
from config import config
with open("ssh_config.txt", "r") as f:
lines = f.readlines()
hostname = lines[0].strip()
username = lines[1].strip()
password = lines[2].strip()
remote_bind_address = lines[3].strip()
try:
with SSHTunnelForwarder(
(hostname, 22),
ssh_username=username,
ssh_password=password,
remote_bind_address=(remote_bind_address, 5432),
local_bind_address=("localhost", 8080)) \
as tunnel:
tunnel.start()
print("SSH connected.")
params = config()
conn = connect(**params)
cursor = conn.cursor()
print("DB connected.")
# Print PostgreSQL connection properties.
print(conn.get_dsn_parameters(), "\n")
# Print PostgreSQL version.
cursor.execute("SELECT version();")
record = cursor.fetchone()
print("You are connected to - ", record, "\n")
cursor.close()
conn.close()
tunnel.close()
print("DB disconnected.")
except (Exception, Error) as error:
print("Error while connecting to DB", error)

How to connect to main Flask APP from a remote server

import psutil
STATS_URL = ' http://172.20.10.2:8080/report'
SERVER_NAME="test_local_server"
def get_health():
print('generating health report')
cpu_percent = psutil.cpu_percent(interval=2.0)
ctime = psutil.cpu_times()
disk_usage = psutil.disk_usage("/")
net_io_counters = psutil.net_io_counters()
virtual_memory = psutil.virtual_memory()
# The keys in this dict should match the db cols
report = dict (
USER_NAME="USER",
SERVER_NAME="Test_Server",
cpupercent = cpu_percent,
cpu_total = ctime.user + ctime.system,
free_Percnt=((disk_usage.free/disk_usage.used)*100),
bytes_sent = net_io_counters.bytes_sent,
bytes_received = net_io_counters.bytes_recv,
packets_sent = net_io_counters.packets_sent,
packets_received = net_io_counters.packets_recv,
memory_Free = virtual_memory.free,
)
return report
if __name__=='__main__':
print(f'starting health report stream for server :\t{SERVER_NAME}')
while True:
report = get_health()
r = requests.post(STATS_URL, json=report)
time.sleep(20)
I need to send the values of this script which is on an AWS remote server back to my Flask APP which is my main server on my windows laptop.
#app.route("/report",methods=['POST'])
def report():
if request.method=='POST':
time_epoch=time.time()
incoming_report = request.get_json()
print("Generating Health report")
username=incoming_report["USER_NAME"]
server_name=incoming_report["SERVER_NAME"]
disk_free=incoming_report["free_Percnt"]
bytes_sent=incoming_report["bytes_sent"]
bytes_received=incoming_report["bytes_received"]
packets_sent=incoming_report["packets_sent"]
packets_received=incoming_report["packets_received"]
memory_free=incoming_report["memory_Free"]
cpu_percent=incoming_report["cpupercent"]
cpu_total=incoming_report["cpu_total"]
conn=sqlite3.connect("Health.db")
conn.execute(f"create table if not exists {username}_{server_name}
(HEALTH_ID integer primary key AUTOINCREMENT,Time_Epoch integer,Disk_Free varchar(80),Bytes_Sent varchar(80),Bytes_Received varchar(80),Packets_Sent varchar(80),Packets_Received varchar(80),Memory_Free varchar(80),Cpu_Usage_Percent varchar(80),Cpu_Time varchar(80));")
conn.execute(f'insert into {username}_{server_name} (Time_Epoch,Disk_Free,Bytes_Sent,Bytes_Received,Packets_Sent,Packets_Received,Memory_Free,Cpu_Usage_Percent,Cpu_Time) values {time_epoch,disk_free,bytes_sent,bytes_received,packets_sent,packets_received,memory_free,cpu_percent,cpu_total}')
conn.commit()
return {'message': 'success'}
if __name__ ==("__main__"):
app.run(host='0.0.0.0',port=8080)
So i changed the host to '0.0.0.0' cause when i checked online thats what i found to make the flask app public but the connection from script says connection timed out.Can someone please help me send the values from the python script back to my flask app?

You need to forward port 8080 on your router. Right now your home router is also a firewall and doesn't permit anything initiating a connection with a LAN machine (your PC). It needs to be told "if anyone tries to connect to our home IP at port 8080, connect it to my PC at the internal IP x.x.x.x"