We Keep Coding

Issues with connecting to Postgres DB via SSH in Python

I've been given a postgres DB in my uni project in which I have to SSH into the network from which I can then access the DB. I've set up the connection in DBeaver using the SSH tab and it works perfectly fine. However, using Python, I can connect to the SSH just fine, but cannot connect to the DB itself. I've checked with another DB that doesn't require SSH and that works just fine. Here is my code. Note: I've already tried using SSHTunnel, too, to no avail. Also, ignore my quick hack to anonymize my SSH login data, as I didn't find how to use a proper config file with paramiko late at night yesterday...
import os
from psycopg2 import connect, Error
from paramiko import SSHClient
from config import config
with open("ssh_config.txt", "r") as f:
lines = f.readlines()
hostname = lines[0].strip()
username = lines[1].strip()
password = lines[2].strip()
ssh = SSHClient()
ssh.load_host_keys(os.path.expanduser('~/.ssh/known_hosts'))
ssh.connect(hostname=hostname, username=username, password=password)
print("SSH connected.")
try:
params = config()
conn = connect(**params)
cursor = conn.cursor()
print("DB connected.")
# Print PostgreSQL connection properties.
print(conn.get_dsn_parameters(), "\n")
# Print PostgreSQL version.
cursor.execute("SELECT version();")
record = cursor.fetchone()
print("You are connected to - ", record, "\n")
except (Exception, Error) as error:
print("Error while connecting to PostgreSQL", error)
Any help would be appreciated. Thanks!

I've figured it out myself. Here is the updated code. Basically, I had to forward the remote address to localhost and then connect to localhost instead of the DB address.
from psycopg2 import connect, Error
from sshtunnel import SSHTunnelForwarder
from config import config
with open("ssh_config.txt", "r") as f:
lines = f.readlines()
hostname = lines[0].strip()
username = lines[1].strip()
password = lines[2].strip()
remote_bind_address = lines[3].strip()
try:
with SSHTunnelForwarder(
(hostname, 22),
ssh_username=username,
ssh_password=password,
remote_bind_address=(remote_bind_address, 5432),
local_bind_address=("localhost", 8080)) \
as tunnel:
tunnel.start()
print("SSH connected.")
params = config()
conn = connect(**params)
cursor = conn.cursor()
print("DB connected.")
# Print PostgreSQL connection properties.
print(conn.get_dsn_parameters(), "\n")
# Print PostgreSQL version.
cursor.execute("SELECT version();")
record = cursor.fetchone()
print("You are connected to - ", record, "\n")
cursor.close()
conn.close()
tunnel.close()
print("DB disconnected.")
except (Exception, Error) as error:
print("Error while connecting to DB", error)

How would one connect an SQL db securely to an external client?

I'm attempting to connect a database, located on a web server, to a robot but I do not know how to connect the database to the robot. I would like the robot to run SELECT and UPDATE queries from the robot. The other issue is that I do not intend on using C-languages or Java; I plan on using python in the main control system.
I do know:
PHP
VBScript
Batch
Python
If anyone knows how to connect the DB to a bot it would be a great help.

So basically how to connect to an SQL DB in python? I'm working on a virtual bot right now doing the same thing. Look into the module , SQL-connector! http://www.mysqltutorial.org/python-connecting-mysql-databases/
You would start with creating a config.ini with your credentials
[mysql]
host = localhost
database = python_mysql
user = root
password =
Read Config.ini and return a dictionary
from configparser import ConfigParser
def read_db_config(filename='config.ini', section='mysql'):
""" Read database configuration file and return a dictionary object
:param filename: name of the configuration file
:param section: section of database configuration
:return: a dictionary of database parameters
"""
# create parser and read ini configuration file
parser = ConfigParser()
parser.read(filename)
# get section, default to mysql
db = {}
if parser.has_section(section):
items = parser.items(section)
for item in items:
db[item[0]] = item[1]
else:
raise Exception('{0} not found in the {1} file'.format(section, filename))
return db
and connect to MYSQL database
from mysql.connector import MySQLConnection, Error
from python_mysql_dbconfig import read_db_config
def connect():
""" Connect to MySQL database """
db_config = read_db_config()
try:
print('Connecting to MySQL database...')
conn = MySQLConnection(**db_config)
if conn.is_connected():
print('connection established.')
else:
print('connection failed.')
except Error as error:
print(error)
finally:
conn.close()
print('Connection closed.')
if __name__ == '__main__':
connect()
and update statement would look like the following
def update_book(book_id, title):
# read database configuration
db_config = read_db_config()
# prepare query and data
query = """ UPDATE books
SET title = %s
WHERE id = %s """
data = (title, book_id)
try:
conn = MySQLConnection(**db_config)
# update book title
cursor = conn.cursor()
cursor.execute(query, data)
# accept the changes
conn.commit()
except Error as error:
print(error)
finally:
cursor.close()
conn.close()
if __name__ == '__main__':
update_book(37, 'The Giant on the Hill *** TEST ***')

AWS Lambda Python with MySQL

I'm trying to connect to mysql from my AWS Lambda script.I did pip install --allow-external mysql-connector-python mysql-connector-python -t <dir>
to install mysql-connector-python in local directory.
I zipped the file and uploaded it to AWS Lambda where my python files are being executed.
My scripts are executing correctly up to the point where I initialize a mysql connection.
I have this
log('about to set connection for db')
connection = mysql.connector.connect(user=DB_USER, password=DB_PASSWORD, host=DB_HOST, database=DB_DATABASE)
query = "SELECT * FROM customers WHERE customer_email LIKE '%s' LIMIT 1"
log('set connection for DB')
'about to set connection for db' is being logged but 'set connection for DB' is never logged and my program hits a timeout and stops executing.
What might I be doing wrong?
EDIT:
This is my class that I'm calling from lambda_function.py
import mysql.connector
import logging
from mysql.connector import errorcode
class MySql( object ):
USER =None
PASSWORD=None
HOST =None
DATABASE=None
logger = logging.getLogger()
logger.setLevel( logging.INFO )
def __init__(self, user, password, host, database):
global USER, PASSWORD, HOST, DATABASE
USER = user
PASSWORD = password
HOST = host
DATABASE = database
def getId( self, customer_email ):
email_exists = False
connection = mysql.connector.connect(user=USER, password=PASSWORD, host=HOST, database=DATABASE)
query = "SELECT * FROM customers WHERE customer_email LIKE '%s' LIMIT 1"
cursor = connection.cursor()
cursor.execute( query % customer_email )
data = cursor.fetchall()
id = None
for row in data :
id = row[1]
break
cursor.close()
connection.close()
return id
def insertCustomer( self, customer_email, id ):
log('about to set connection for db')
connection = mysql.connector.connect(user=USER, password=PASSWORD, host=HOST, database=DATABASE)
log('set connection for DB')
cursor = connection.cursor()
try:
cursor.execute("INSERT INTO customers VALUES (%s,%s)",( customer_email, id ))
connection.commit()
except:
connection.rollback()
connection.close()
def log( logStr):
logger.info( logStr )
def main():
user = 'xxx'
password = 'xxx'
host = ' xxx'
database = 'xxx'
mysql = MySql( user, password, host, database )
id = mysql.getId('testing')
if id == None:
mysql.insertCustomer('blah','blahblah')
print id
if __name__ == "__main__":
main()
When I execute the MySql.py locally my code works fine. My database gets updated but nothing happens when I run it from AWS.

Is it a MySQL instance on AWS (RDS) or on premise DB? If RDS, check the NACL inbound rules of the vpc associated with your DB instance. Inbound rules can allow/deny from specific IP sources

when you did a zip file create. Did you do a pip to target directory.
I have enclosed the syntax below.This copies the files to your target directory to zip.
That may be the reason you are able to execute it locally but not in a lambda.
This is the syntax
pip install module-name -t /path/to/PythonExampleDir

Connect by DSN fails with "invalid username or password", but DSN does have correct username and password

I'm attempting to connect to a vertica database from OS/X 10.10.2 using the following small script:
#!/usr/bin/python
import sys
import pyodbc
def usage(retval):
if retval == 0:
file_ = sys.stdout
else:
file_ = sys.stderr
file_.write('Usage: {} --query sql_query --dsn data_source_name\n'.format(sys.argv[0]))
sys.exit(retval)
def main():
query = None
dsn = None
while sys.argv[1:]:
if sys.argv[1] == '--query':
query = sys.argv[2]
del sys.argv[1]
elif sys.argv[1] == '--dsn':
dsn = sys.argv[2]
del sys.argv[1]
elif sys.argv[1] in {'-h', '--help'}:
usage(0)
else:
sys.stderr.write('{}: Unrecognized option: {}\n'.format(sys.argv[0], sys.argv[1]))
usage(1)
del sys.argv[1]
if query is None:
sys.stderr.write('{}: --query is a required option\n'.format(sys.argv[0]))
usage(1)
if dsn is None:
sys.stderr.write('{}: --dsn is a required option\n'.format(sys.argv[0]))
usage(1)
connection_string = 'DSN={}'.format(dsn)
sys.stderr.write('connection_string is {}\n'.format(connection_string))
connection = pyodbc.connect(connection_string)
cursor = connection.cursor()
cursor.execute(query)
# Fetchall might be faster, but it also has memory limit issues
for row in cursor:
print(row)
main()
Invoking this script like so:
ODBCINI=/Library/ODBC/odbc.ini /simple-connect --query 'select * from GLITCH.METRIC' --dsn GLITCH_ADMIN
...fails with:
21L, 1L, u"FAILED: ('28000', '[28000] FATAL 3781: Invalid username or password\\n (3781) (SQLDriverConnect)')", u'unique_seconds', ...
...plus a little more text on the same line, and one additional line which I've deleted for protection.
I have another (much larger) program that connects fine using the same DSN, without prompting for a username or password.
The relevant DSN in /Library/ODBC/odbc.ini looks like (hostname, username and password changed for protection) :
Driver = Vertica
Description = Glitch Vertica Meta Data DSN
Database = Analytics
Servername = hostname
UID = username
PWD = password
PORT = 5433
SSLMode = require
I dtruss'd the script, and found that it was looking in /Library/ODBC/odbc.ini - or rather, it does with the ODBCINI environment variable set appropriately. Without that variable, it appeared to be ignoring my odbc.ini.
Does anyone know why it's failing to authenticate?
Thanks!

Psycopg2 access PostgreSQL database on remote host without manually opening ssh tunnel

My standard procedure for accessing a PostgreSQL database on a remote server is to first create an ssh tunnel as:
ssh username1#remote.somewhere.com -L 5432:localhost:5432 -p 222
and then run my query in python from another shell as:
conn = psycopg2.connect("host=localhost" + " dbname=" +
conf.dbname + " user=" + conf.user +
" password=" + conf.password)
cur = conn.cursor()
cur.execute(query)
This piece of python code works nicely once the tunnel is created. However, I would like psycopg2 to already open the SSH tunnel or reach "somehow" the remote database without need to redirect it on my localhost.
Is it possible to do this with psycopg2?
Is otherwise possible open the ssh tunnel in my python code?
if I use:
os.system("ssh username1#remote.somewhere.com -L 5432:localhost:5432 -p 222")
The shell will be redirected to the remote host blocking the execution of main thread.

You could also use sshtunnel, short and sweet:
from sshtunnel import SSHTunnelForwarder
PORT=5432
with SSHTunnelForwarder((REMOTE_HOST, REMOTE_SSH_PORT),
ssh_username=REMOTE_USERNAME,
ssh_password=REMOTE_PASSWORD,
remote_bind_address=('localhost', PORT),
local_bind_address=('localhost', PORT)):
conn = psycopg2.connect(...)

With sshtunnel package
I was not familiar with SSH tunnels, so I had some difficulties to use mrts's answer.
Maybe thoses precisions could help someone.
In psycopg2.connect(), host and port are the one you just created by connecting remote host by ssh tunnel.
Here is my code :
from sshtunnel import SSHTunnelForwarder
server = SSHTunnelForwarder((REMOTE_HOST, REMOTE_SSH_PORT),
ssh_username=REMOTE_USERNAME,
ssh_password=REMOTE_PASSWORD,
remote_bind_address=('localhost', PORT),
local_bind_address=('localhost', PORT))
server.start()
import psycopg2
conn = psycopg2.connect(
database=DATABASE,
user=USER,
host=server.local_bind_host,
port=server.local_bind_port,
password=PWD)
cur = conn.cursor()
cur.execute("select * from yourtable limit 1;")
data = cur.fetchall()
print(data)
I hope this example make it clearer.

Call your ssh via os.system in a separate thread/process. You can also use -N with ssh to avoid opening a remote shell.

Clodoaldo Neto's code worked for me perfectly but beware it doesn't clean up the process afterward.
The method shown by Luca Fiaschi also works for me. I updated it a bit for python3 and the updated psutil module. The changes were just that process.username and process.cmdline are now functions and that the iterator is process_iter() instead of get_process_list().
Here is an example of a very slightly modified version of the code Luca Fiaschi posted that works with python3 (requires psutil module). I hope it is at least mostly correct!
#!/usr/bin/env python3
import psutil
import psycopg2
import subprocess
import time
import os
# Tunnel Config
SSH_HOST = "111.222.333.444"
SSH_USER = "user"
SSH_KEYFILE = "key.pem"
SSH_FOREIGN_PORT = 5432 # Port that postgres is running on the foreign server
SSH_INTERNAL_PORT = 5432 # Port we open locally that is forwarded to
# FOREIGN_PORT on the server.
# Postgres Config
DB_HOST = "127.0.0.1"
DB_PORT = SSH_INTERNAL_PORT
DB_PASSWORD = "password"
DB_DATABASE = "postgres"
DB_USER = "user"
class SSHTunnel(object):
"""
A context manager implementation of an ssh tunnel opened from python
"""
def __init__(self, tunnel_command):
assert "-fN" in tunnel_command, "need to open the tunnel with -fN"
self._tunnel_command = tunnel_command
self._delay = 0.1
self.ssh_tunnel = None
def create_tunnel(self):
tunnel_cmd = self._tunnel_command
ssh_process = subprocess.Popen(tunnel_cmd, universal_newlines=True,
shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
stdin=subprocess.PIPE)
# Assuming that the tunnel command has "-f" and "ExitOnForwardFailure=yes", then the
# command will return immediately so we can check the return status with a poll().
while True:
p = ssh_process.poll()
if p is not None: break
time.sleep(self._delay)
if p == 0:
# Unfortunately there is no direct way to get the pid of the spawned ssh process, so we'll find it
# by finding a matching process using psutil.
current_username = psutil.Process(os.getpid()).username()
ssh_processes = [proc for proc in psutil.process_iter() if proc.cmdline() == tunnel_cmd.split() and proc.username() == current_username]
if len(ssh_processes) == 1:
self.ssh_tunnel = ssh_processes[0]
return ssh_processes[0]
else:
raise RuntimeError('multiple (or zero?) tunnel ssh processes found: ' + str(ssh_processes))
else:
raise RuntimeError('Error creating tunnel: ' + str(p) + ' :: ' + str(ssh_process.stdout.readlines()))
def release(self):
""" Get rid of the tunnel by killin the pid
"""
if self.ssh_tunnel:
self.ssh_tunnel.terminate()
def __enter__(self):
self.create_tunnel()
return self
def __exit__(self, type, value, traceback):
self.release()
def __del__(self):
self.release()
command = "ssh -i %s %s#%s -fNL %d:localhost:%d"\
% (SSH_KEYFILE, SSH_USER, SSH_HOST, SSH_INTERNAL_PORT, SSH_FOREIGN_PORT)
with SSHTunnel(command):
conn = psycopg2.connect(host = DB_HOST, password = DB_PASSWORD,
database = DB_DATABASE, user = DB_USER, port = DB_PORT)
curs = conn.cursor()
sql = "select * from table"
curs.execute(sql)
rows = curs.fetchall()
print(rows)

For the moment I am using a solution bsed on this gist:
class SSHTunnel(object):
"""
A context manager implementation of an ssh tunnel opened from python
"""
def __init__(self, tunnel_command):
assert "-fN" in tunnel_command, "need to open the tunnel with -fN"
self._tunnel_command = tunnel_command
self._delay = 0.1
def create_tunnel(self):
tunnel_cmd = self._tunnel_command
import time, psutil, subprocess
ssh_process = subprocess.Popen(tunnel_cmd, universal_newlines=True,
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
stdin=subprocess.PIPE)
# Assuming that the tunnel command has "-f" and "ExitOnForwardFailure=yes", then the
# command will return immediately so we can check the return status with a poll().
while True:
p = ssh_process.poll()
if p is not None: break
time.sleep(self._delay)
if p == 0:
# Unfortunately there is no direct way to get the pid of the spawned ssh process, so we'll find it
# by finding a matching process using psutil.
current_username = psutil.Process(os.getpid()).username
ssh_processes = [proc for proc in psutil.get_process_list() if proc.cmdline == tunnel_cmd.split() and proc.username == current_username]
if len(ssh_processes) == 1:
self.ssh_tunnel = ssh_processes[0]
return ssh_processes[0]
else:
raise RuntimeError, 'multiple (or zero?) tunnel ssh processes found: ' + str(ssh_processes)
else:
raise RuntimeError, 'Error creating tunnel: ' + str(p) + ' :: ' + str(ssh_process.stdout.readlines())
def release(self):
""" Get rid of the tunnel by killin the pid
"""
self.ssh_tunnel.terminate()
def __enter__(self):
self.create_tunnel()
return self
def __exit__(self, type, value, traceback):
self.release()
def __del__(self):
self.release()
def test():
#do things that will fail if the tunnel is not opened
print "done =========="
command = "ssh username#someserver.com -L %d:localhost:%d -p 222 -fN" % (someport, someport)
with SSHTunnel(command):
test()
Please let me know if anybody has a better idea

from time import sleep
os.system("ssh username1#remote.somewhere.com -fNL 5432:localhost:5432 -p 222")
while True:
try:
conn = psycopg2.connect(
"host=localhost dbname={0} user={1} password={2}".format(
conf.dbname, conf.user, conf.password
)
)
break
except psycopg2.OperationalError:
sleep(3)