Using database.ini versus hardcoding/using env with psycopg2 - python

I have a problem that gave me a lot of grievance and would like to understand why it happened.
I want to access our Redshift instance using psycopg2. My initial attempt just hardcoded the information in the script and used an env variable for the password.
import psycopg2
import os
def connect_to_db():
return psycopg2.connect(
host="host url",
database="db name",
user="username",
password=os.getenv("db_pw"),
port=12345
)
Whenever I tried this, I would get this error:
psycopg2.OperationalError: FATAL: password authentication failed for user "username"
FATAL: no pg_hba.conf entry for host "::ffff:<host IP>", user "username", database "db name", SSL off
However if I use this following project structure I'm able to get the script to run as intended.
- project
- db.py
- database.ini
And these are the files:
db.py:
import psycopg2
from configparser import ConfigParser, Error
def config(filename='database.ini', section='postgresql'):
# create a parser
parser = ConfigParser()
# read config file
parser.read(filename)
# get section, default to postgresql
db = {}
if parser.has_section(section):
params = parser.items(section)
for param in params:
db[param[0]] = param[1]
else:
raise Exception('Section {0} not found in the {1} file'.format(section, filename))
return db
def connect_to_db():
"""Connect to db and return the connection object"""
params = config()
return psycopg2.connect(**params)
database.ini
[postgresql]
host=hosturl
database=db
user=user
password=password
port=12345
Could someone explain why this is happening?
Mac Intel 2020 2.3 GHz Quad-Core Intel Core i7

Related

flask_mysqldb - MYSQL cursor is throwing an error - cursor is not a known member of "None"

I am writing a simple auth service in python using flask and flask_mysqldb. There is an error with the cursor.
import jwt
from flask import Flask, request
from flask_mysqldb import MySQL
server = Flask(__name__)
mysql = MySQL(server)
# server configuration
server.config["MYSQL_HOST"] = os.environ.get("MYSQL_HOST")
server.config["MYSQL_USER"] = os.environ.get("MYSQL_USER")
server.config["MYSQL_PASSWORD"] = os.environ.get("MYSQL_PASSWORD")
server.config["MYSQL_DB"] = os.environ.get("MYSQL_DB")
server.config["MYSQL_PORT"] = os.environ.get("MYSQL_PORT")
# print(server.config["MYSQL_HOST"])
# print(server.config["MYSQL_PORT"])
#server.route("/login", methods=["POST"])
def login():
auth = request.authorization
if not auth:
return "missing credentials",401
#check db for username and password
cur = mysql.connection.cursor()
res = cur.execute(
"SELECT email,password FROM user WHERE email=%s, (auth.username)"
)
This works on a virtual environment. All the specified packages are correctly installed.
Please try
cur = mysql.connect.cursor()
if you use a connection it will not suggest cursor(). once you use connect.cursor() it will not show the error. Thanks

Connecting to Cloud SQL with Private IP gives timeout when trying to connect in Cloud Functions despite successful connectivity test

I am trying to connect to a Cloud SQL instance with Private IP through a Cloud Function which runs all egress traffic through a VPC connector.
I did a connectivity test which gave the following results:
Then, I deployed a Cloud Function following the docs and run the following code:
import sqlalchemy
import os
from google.cloud import storage
# SQLAlchemy==1.4.28
# psycopg2-binary==2.9.2
# google-cloud-storage==2.2.1
def hello_world(request):
# Remember - storing secrets in plaintext is potentially unsafe. Consider using
# something like https://cloud.google.com/secret-manager/docs/overview to help keep
# secrets secret.
db_user = '<DB-USER>'
db_pass = '<DB-PASSWORD>'
db_name = 'postgres'
db_host = '172.16.0.5:5432'
# Extract port from db_host if present,
# otherwise use DB_PORT environment variable.
host_args = db_host.split(":")
if len(host_args) == 1:
db_hostname = db_host
db_port = os.environ["DB_PORT"]
elif len(host_args) == 2:
db_hostname, db_port = host_args[0], int(host_args[1])
print(f"{db_hostname}, {db_port}")
pool = sqlalchemy.create_engine(
# Equivalent URL:
# mysql+pymysql://<db_user>:<db_pass>#<db_host>:<db_port>/<db_name>
sqlalchemy.engine.url.URL.create(
drivername="postgresql+psycopg2",
username=db_user, # e.g. "my-database-user"
password=db_pass, # e.g. "my-database-password"
host=db_hostname, # e.g. "127.0.0.1"
port=db_port, # e.g. 5432
database=db_name, # e.g. "my-database-name"
)
)
print(f'Created engine : {pool}')
pool.connect()
print(f'Done engine : {pool}')
return 'Ended well', 200
However, it is unable to connect to the instance, and the function times out at pool.connect():
I confirm that the source IP is in line with the expectation of the connectivity test (11.0.0.11).
What could be the reason, or what could I do to find out the actual cause?

SQLAlchemy create_engine connection string with Microsoft ODBC datasource User DSN

Does anyone know how to pass connection string into create_engine function?
I use Window and has a ODBC datasource with DSN that set up by IT department.
My ODBC DSN connects to Postgres database.
Does anyone know the library or connection string to make this to work?
Note that I cannot ask them for the username and password to access the Postgres directly. I only can connect via ODBC only.
Thank you very much.
The below worked for me:
engine=create_engine("mssql+pyodbc://user:password#DSNSTRING")
Yes, you can use the winreg library.
Below is a function I adapted from Bart Jonk.
Original answer and function: https://stackoverflow.com/a/66528870/11080806
from winreg import (ConnectRegistry, HKEY_CURRENT_USER, HKEY_LOCAL_MACHINE, OpenKeyEx, QueryValueEx,)
def odbc_connection_string(odbc_dsn_name: str, system_dsn: bool = False):
"""
Converts a windows ODBC DSN to a dsn
that can be passed to sqlalchemy and used by psycopg2.
It reads connection parameters from the Windows Registry.
By: Bart Jonk on stackoverflow
https://stackoverflow.com/users/11100064/bart-jonk
https://stackoverflow.com/a/66528870/11080806
Parameters
-----
odbc_dsn_name: str
Name of the ODBC DSN to get connection details from.
system_dsn: bool, default False
If True look for DSN in system folder, otherwise look in current user folder.
"""
# connect to registry
if system_dsn:
hreg = ConnectRegistry(None, HKEY_LOCAL_MACHINE)
else:
hreg = ConnectRegistry(None, HKEY_CURRENT_USER)
# define key and open
key = f"SOFTWARE\\ODBC\\ODBC.INI\\{odbc_dsn_name}"
hkey = OpenKeyEx(hreg, key)
# get hostname, port, sslmode, credentials from odbc registry
hostname = QueryValueEx(hkey, "Servername")[0]
# port_number = QueryValueEx(hkey, "Port")[0]
# ssl_mode = QueryValueEx(hkey, "SSLmode")[0]
database = QueryValueEx(hkey, "Database")[0]
username = QueryValueEx(hkey, "UID")[0]
password = QueryValueEx(hkey, "Password")[0]
# set defaults for port number and SSL mode
# if not port_number:
# port_number = "5432"
# if not ssl_mode:
# ssl_mode = 'Prefer'
# create connection string in sqlalchemy format
return f"postgresql://{username}:{password}#{hostname}/{database}"

Python - MySQL SSL Connections

I have a MySQL Server set up to use SSL and I also have the CA Certificate.
When I connect to the server using MySQL Workbench, I do not need the certificate. I can also connect to the server using Python and MySQLdb on a Mac without the CA-certificate.
But when I try to connect using the exact same setup of Python and MySQLdb on a windows machine, I get access denied. It appears that I need the CA. And when I enter the CA, I get the following error
_mysql_exceptions.OperationalError: (2026, 'SSL connection error')
My code to open the connection is below:
db = MySQLdb.connect(host="host.name",
port=3306,
user="user",
passwd="secret_password",
db="database",
ssl={'ca': '/path/to/ca/cert'})
Could anyone point out what the problem is on a windows?
I just got the following to work with Python 2.7 and MySQLdb (1.2.4):
database = MySQLdb.connect(host='hostname', user='username', db='db_name',
passwd='PASSWORD', ssl={'ca': '/path/to/ca-file'})
This is what you had so there must be something else going on here. I wonder if you have something either incorrect with the your local CA file or possibly the cert on the server? Can you get a copy of the CA file from the server?
Try this
import ssl
from databases import Database
sslctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH,
cafile='ca.pem')
sslctx.verify_mode = ssl.CERT_REQUIRED
sslctx.check_hostname = True
sslctx.load_cert_chain(certfile='client.crt', keyfile='pkey.key')
database = Database(DATABASE_URL, ssl=sslctx)
databases library provides support for PostgreSQL, MySQL, and SQLite.
Also useful for async frameworks.
Im using pymysql and I had some problems to connect using SSL keys and certs: for the "ssl" attribute I set up as a dictionary inside. Try as below:
db = MySQLdb.connect(host="host.name",
port=3306,
user="user",
passwd="secret_password",
db="database",
ssl={'ssl':
{'ca': '/path/to/ca/ca',
'key': '/path/to/ca/key',
'cert': '/path/to/ca/cert'
}
}
)
I know this is a bit old but I found a way to get this to work. Use pymysql instead of MySQLdb and write the connection as:
import pymysql
conn = pymysql.connect(user = 'user', password = 'passwd'
, database = 'db', host = 'hst', ssl = {'ssl' : {'ca': 'pathtosll/something.pem'}})
The point people miss (including myself) is that ssl needs to be a dictionary containing
a key 'ssl' which has another dictionary as a value with a key 'ca'. This should work for you.
import pymysql
conn = pymysql.connect(host= # your host, usually localhost,
user = # your username,
passwd = # your password,
db = #your database name ,
ssl ={'ssl': r'path of your pem file'})

Can I use a config file to hold connection string parameters?

I have a script which connects to a database. I'd like to pull out the server,user,password,db from the script and put it into a config file. I'm successfully pulling in the values from the config file. The problem I'm having is the pymssql.connect fails when I have variables in the syntax. I pasted my sample code below. Is this possible?
###database connection
config = configparser.ConfigParser()
config.read('test.config')
server = config['DEFAULT']['SQLServer']
db = config['DEFAULT']['Database']
user = config['DEFAULT']['User']
password = config['DEFAULT']['Password']
###this works
####conn = pymssql.connect(host='Server1', user='Joe',password='MyPass', database='MyDB')
###this doesn't
try:
conn = pymssql.connect(host=server, user=user,password=password, database=db)
except Exception as e:
print(str(e))
sys.exit()
This is how I retrieved connection properties and connected sql server database
**App.ini file**
[CoreContext]
host=sservername.database.windows.net
user=dbuser#servername
password=password
database=DeltaXCore
**Connection.py file**
appConfig = ConfigParser.ConfigParser()
appConfig.read("App.ini")
ConnectionString.HOST = appConfig.get("CoreContext", "host")
CoreConnectionString.USER = appConfig.get("CoreContext", "user")
CoreConnectionString.PASSWORD = appConfig.get("CoreContext", "password")
CoreConnectionString.DATABASE = appConfig.get("CoreContext", "database")
pymssql.connect(host=objdeltaxConnectionString.HOST, user=objdeltaxConnectionString.USER,password=objdeltaxConnectionString.PASSWORD, database=objdeltaxConnectionString.DATABASE)

Categories