Does anyone know how to pass connection string into create_engine function?
I use Window and has a ODBC datasource with DSN that set up by IT department.
My ODBC DSN connects to Postgres database.
Does anyone know the library or connection string to make this to work?
Note that I cannot ask them for the username and password to access the Postgres directly. I only can connect via ODBC only.
Thank you very much.
The below worked for me:
engine=create_engine("mssql+pyodbc://user:password#DSNSTRING")
Yes, you can use the winreg library.
Below is a function I adapted from Bart Jonk.
Original answer and function: https://stackoverflow.com/a/66528870/11080806
from winreg import (ConnectRegistry, HKEY_CURRENT_USER, HKEY_LOCAL_MACHINE, OpenKeyEx, QueryValueEx,)
def odbc_connection_string(odbc_dsn_name: str, system_dsn: bool = False):
"""
Converts a windows ODBC DSN to a dsn
that can be passed to sqlalchemy and used by psycopg2.
It reads connection parameters from the Windows Registry.
By: Bart Jonk on stackoverflow
https://stackoverflow.com/users/11100064/bart-jonk
https://stackoverflow.com/a/66528870/11080806
Parameters
-----
odbc_dsn_name: str
Name of the ODBC DSN to get connection details from.
system_dsn: bool, default False
If True look for DSN in system folder, otherwise look in current user folder.
"""
# connect to registry
if system_dsn:
hreg = ConnectRegistry(None, HKEY_LOCAL_MACHINE)
else:
hreg = ConnectRegistry(None, HKEY_CURRENT_USER)
# define key and open
key = f"SOFTWARE\\ODBC\\ODBC.INI\\{odbc_dsn_name}"
hkey = OpenKeyEx(hreg, key)
# get hostname, port, sslmode, credentials from odbc registry
hostname = QueryValueEx(hkey, "Servername")[0]
# port_number = QueryValueEx(hkey, "Port")[0]
# ssl_mode = QueryValueEx(hkey, "SSLmode")[0]
database = QueryValueEx(hkey, "Database")[0]
username = QueryValueEx(hkey, "UID")[0]
password = QueryValueEx(hkey, "Password")[0]
# set defaults for port number and SSL mode
# if not port_number:
# port_number = "5432"
# if not ssl_mode:
# ssl_mode = 'Prefer'
# create connection string in sqlalchemy format
return f"postgresql://{username}:{password}#{hostname}/{database}"
Related
I am trying to connect to a Cloud SQL instance with Private IP through a Cloud Function which runs all egress traffic through a VPC connector.
I did a connectivity test which gave the following results:
Then, I deployed a Cloud Function following the docs and run the following code:
import sqlalchemy
import os
from google.cloud import storage
# SQLAlchemy==1.4.28
# psycopg2-binary==2.9.2
# google-cloud-storage==2.2.1
def hello_world(request):
# Remember - storing secrets in plaintext is potentially unsafe. Consider using
# something like https://cloud.google.com/secret-manager/docs/overview to help keep
# secrets secret.
db_user = '<DB-USER>'
db_pass = '<DB-PASSWORD>'
db_name = 'postgres'
db_host = '172.16.0.5:5432'
# Extract port from db_host if present,
# otherwise use DB_PORT environment variable.
host_args = db_host.split(":")
if len(host_args) == 1:
db_hostname = db_host
db_port = os.environ["DB_PORT"]
elif len(host_args) == 2:
db_hostname, db_port = host_args[0], int(host_args[1])
print(f"{db_hostname}, {db_port}")
pool = sqlalchemy.create_engine(
# Equivalent URL:
# mysql+pymysql://<db_user>:<db_pass>#<db_host>:<db_port>/<db_name>
sqlalchemy.engine.url.URL.create(
drivername="postgresql+psycopg2",
username=db_user, # e.g. "my-database-user"
password=db_pass, # e.g. "my-database-password"
host=db_hostname, # e.g. "127.0.0.1"
port=db_port, # e.g. 5432
database=db_name, # e.g. "my-database-name"
)
)
print(f'Created engine : {pool}')
pool.connect()
print(f'Done engine : {pool}')
return 'Ended well', 200
However, it is unable to connect to the instance, and the function times out at pool.connect():
I confirm that the source IP is in line with the expectation of the connectivity test (11.0.0.11).
What could be the reason, or what could I do to find out the actual cause?
I set up the Cloud SQL instance on Google Cloud Platform and followed the official instructions, but don't seem to be able to connect to the Cloud SQL instance. When I try to do a sanity check and access the PostgreSQL db through Cloud Shell, I'm able to connect successfully though.
Could someone please help - I would be much obliged.
Code:
from sqlalchemy import create_engine
engine = create_engine('postgresql+psycopg2://<user>:<pass>#<public IP Address/<table>')
engine.connect()
Error:
Is the server running on host "XX.XX.XXX.XX" and accepting
TCP/IP connections on port XXXX?
I found another way to connect to a PostgreSQL GCP instance without using the Cloud SQL Proxy.
Code:
import sqlalchemy
username = '' # DB username
password = '' # DB password
host = '' # Public IP address for your instance
port = '5432'
database = '' # Name of database ('postgres' by default)
db_url = 'postgresql+psycopg2://{}:{}#{}:{}/{}'.format(
username, password, host, port, database)
engine = sqlalchemy.create_engine(db_url)
conn = engine.connect()
I whitelisted my IP address before trying to connect. (https://cloud.google.com/sql/docs/postgres/connect-external-app#appaccessIP)
Use the Cloud SQL proxy to connect to Cloud SQL from external applications.
In order to achieve this please follow the relevant documentation.
The steps described would consist of:
Enabling the Cloud SQL Admin API on your Cloud Console.
Installing the relevant proxy client according to your OS.
Use any of the available methods to authenticate the Cloud SQL Proxy.
Invoke the proxy with ./cloud_sql_proxy -instances=INSTANCE_CONNECTION_NAME=tcp:5432 & ond your terminal and connect the proxy by changing your code and using SQLALCHEMY:
from sqlalchemy import create_engine
engine = create_engine('postgresql+psycopg2://DATABASE_USER:PASSWORD#localhost:5432/')
NOTE: the code above assumes you are not trying to connect to the proxy in a production environment and are using an authenticated Cloud SDK client in order to connect to the proxy.
This worked to me using the Cloud SQL Proxy on my personal computer and uploading the code to Google App Engine standard.
db_user = os.environ.get('CLOUD_SQL_USERNAME')
db_pass = os.environ.get('CLOUD_SQL_PASSWORD')
db_name = os.environ.get('CLOUD_SQL_DATABASE_NAME')
db_connection_name = os.environ.get('CLOUD_SQL_CONNECTION_NAME')
if os.environ.get('GAE_ENV') == 'standard':
db_uri = f'postgresql+psycopg2://{db_user}:{db_pass}#/{db_name}?host=/cloudsql/{db_connection_name}'
else:
db_uri = f'postgresql+psycopg2://{db_user}:{db_pass}#127.0.0.1:1234/{db_name}'
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = db_uri
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
Depending on the database client library, the socket (/cloudsql/INSTANCE_CONNECTION_NAME/.s.PGSQL.5432) needs to be specified.
The docs have this example for SQLAlchemy:
db_user = os.environ["DB_USER"]
db_pass = os.environ["DB_PASS"]
db_name = os.environ["DB_NAME"]
db_socket_dir = os.environ.get("DB_SOCKET_DIR", "/cloudsql")
cloud_sql_connection_name = os.environ["CLOUD_SQL_CONNECTION_NAME"]
pool = sqlalchemy.create_engine(
# Equivalent URL:
# postgresql+pg8000://<db_user>:<db_pass>#/<db_name>
# ?unix_sock=<socket_path>/<cloud_sql_instance_name>/.s.PGSQL.5432
sqlalchemy.engine.url.URL.create(
drivername="postgresql+pg8000",
username=db_user, # e.g. "my-database-user"
password=db_pass, # e.g. "my-database-password"
database=db_name, # e.g. "my-database-name"
query={
"unix_sock": "{}/{}/.s.PGSQL.5432".format(
db_socket_dir, # e.g. "/cloudsql"
cloud_sql_connection_name) # i.e "<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>"
}
),
**db_config
)
Be aware that this example is with pg8000 that uses unix_sock instead of unix_socket as socket identifier.
I have a MySQL Server set up to use SSL and I also have the CA Certificate.
When I connect to the server using MySQL Workbench, I do not need the certificate. I can also connect to the server using Python and MySQLdb on a Mac without the CA-certificate.
But when I try to connect using the exact same setup of Python and MySQLdb on a windows machine, I get access denied. It appears that I need the CA. And when I enter the CA, I get the following error
_mysql_exceptions.OperationalError: (2026, 'SSL connection error')
My code to open the connection is below:
db = MySQLdb.connect(host="host.name",
port=3306,
user="user",
passwd="secret_password",
db="database",
ssl={'ca': '/path/to/ca/cert'})
Could anyone point out what the problem is on a windows?
I just got the following to work with Python 2.7 and MySQLdb (1.2.4):
database = MySQLdb.connect(host='hostname', user='username', db='db_name',
passwd='PASSWORD', ssl={'ca': '/path/to/ca-file'})
This is what you had so there must be something else going on here. I wonder if you have something either incorrect with the your local CA file or possibly the cert on the server? Can you get a copy of the CA file from the server?
Try this
import ssl
from databases import Database
sslctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH,
cafile='ca.pem')
sslctx.verify_mode = ssl.CERT_REQUIRED
sslctx.check_hostname = True
sslctx.load_cert_chain(certfile='client.crt', keyfile='pkey.key')
database = Database(DATABASE_URL, ssl=sslctx)
databases library provides support for PostgreSQL, MySQL, and SQLite.
Also useful for async frameworks.
Im using pymysql and I had some problems to connect using SSL keys and certs: for the "ssl" attribute I set up as a dictionary inside. Try as below:
db = MySQLdb.connect(host="host.name",
port=3306,
user="user",
passwd="secret_password",
db="database",
ssl={'ssl':
{'ca': '/path/to/ca/ca',
'key': '/path/to/ca/key',
'cert': '/path/to/ca/cert'
}
}
)
I know this is a bit old but I found a way to get this to work. Use pymysql instead of MySQLdb and write the connection as:
import pymysql
conn = pymysql.connect(user = 'user', password = 'passwd'
, database = 'db', host = 'hst', ssl = {'ssl' : {'ca': 'pathtosll/something.pem'}})
The point people miss (including myself) is that ssl needs to be a dictionary containing
a key 'ssl' which has another dictionary as a value with a key 'ca'. This should work for you.
import pymysql
conn = pymysql.connect(host= # your host, usually localhost,
user = # your username,
passwd = # your password,
db = #your database name ,
ssl ={'ssl': r'path of your pem file'})
When I am trying to connect python with SQL Server, following error occurred.
"pyodbc.Error: ('08001', '[08001] [Microsoft][ODBC SQL Server
Driver][DBNETLIB]SQL Server does not exist or access denied. (17)
(SQLDriverConnect)')"
Following is the my code.
import pyodbc
connection = pyodbc.connect("Driver={SQL Server}; Server=localhost;
Database=emotionDetection; uid=uname ;pwd=pw;Trusted_Connection=yes")
cursor = connection.cursor()
SQLCommand = ("INSERT INTO emotion" "(happy, sad, angry) "
"VALUES (?,?,?)")
Values = ['smile','cry','blame']
cursor.execute(SQLCommand,Values)
connection.commit()
connection.close()
This is my first attempt to connect Python with sql server. I don't have an idea what would be the driver name, server name, username and password.Do you have any idea of what should be my configuration. Please help me.
CONNECTION FROM WINDOWS TO MS SQL SERVER DATABASE:
Here you have an example I use myself to connect to MS SQL database table with a Python script:
import pyodbc
server = 'ip_database_server'
database = 'database_name'
username = 'user_name'
password = 'user_password'
driver = '{SQL Server}' # Driver you need to connect to the database
port = '1433'
cnn = pyodbc.connect('DRIVER='+driver+';PORT=port;SERVER='+server+';PORT=1443;DATABASE='+database+';UID='+username+
';PWD='+password)
cursor = cnn.cursor()
'User' and 'password' and 'table_name' are attibutes defined by the DB administrator, and he should give them to you. The port to connect to is also defined by the admin. If you are trying to connect from a Windows device to the DB, go to ODBC Data Source Administrator from Windows, and check if you have installed the driver:
Where is the ODBC data source administrator in a Windows machine.
The image is in spanish, but you only have to click on 'Drivers' tab, and check if the driver is there as in the image.
CONNECTION FROM LINUX/UNIX TO MS SQL SERVER DATABASE:
If you are working in Linux/Unix, then you shoud install a ODBC manager like 'FreeTDS' and 'unixODBC'. To configure them, you have some examples in the following links:
Example: Connecting to Microsoft SQL Server from Linux/Unix
Example: Installing and Configuring ODBC
I think you should check out this.
stackoverflow answer about odbc
Also, what sql server do you use?
The library pymssql doesnot require any drivers and works on both Windows as well as Ubunutu.
import pymssql
import pandas as pd
server = 'yourusername'
username = 'yourusername'
password = 'yourpassword'
database = 'yourdatabase'
table_name = 'yourtablename'
conn = pymssql.connect(host=server,user=username,password=password,database=database)
dat = pd.read_sql("select * from table_name,conn)
Try pyodbc with SQLalchemy
try this:
import sqlalchemy
import pyodbc
from sqlalchemy import create_engine
engine = create_engine("mssql+pyodbc://user:password#host:port/databasename?driver=ODBC+Driver+17+for+SQL+Server")
cnxn = engine.connect()
Use your corresponding driver
It works for me
Luck!
Working Examples Work Best For Me:
Need Mac ODBC Drivers?
If you need the mac driver I used homebrew and found the commands here
Detail
I personally learn best by reverse enginerring, with that said I am sharing one of my examples, it may be a bit crude but I'm growing my Python skills.
My script I created allows me to connect my Mac OS to a AWS RDS instance.
The whole script is a copy paste with a little modification for you about your server info, and you are off and running.
just modify these lines to connect.
server = 'yourusername'
username = 'yourusername'
password = 'yourforgottencomplicatedpassword'
database = 'yourdatabase'
Then Run the file: python3 ~/Your/path/pyodbc_mssqldbtest.py and you should be set.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# =============================================================================
# Created By : Jeromie Kirchoff
# Created Date: Mon July 31 22:32:00 PDT 2018
# FILENAME: pyodbc_mssqldbtest.py
# =============================================================================
"""The Module Has Been Build for Interaction with MSSQL DBs To Test the con."""
# =============================================================================
# Thanks to this post for headers https://stackoverflow.com/q/12704305/1896134
# Answer to an SO question: https://stackoverflow.com/q/42433408/1896134
# =============================================================================
import pyodbc
def runningwithqueries(query):
"""The Module Has Been Build to {Open, Run & Close} query connection."""
print("\nRunning Query: " + str(query) + "\nResult :\n")
crsr = cnxn.execute(query)
columns = [column[0] for column in crsr.description]
print(columns)
for row in crsr.fetchall():
print(row)
crsr.close()
# =============================================================================
# SET VARIABLES NEEDED FOR SERVER CONNECTION
# =============================================================================
server = 'yourusername'
username = 'yourusername'
password = 'yourforgottencomplicatedpassword'
database = 'yourdatabase'
connStr = (r'DRIVER={ODBC Driver 17 for SQL Server};' +
r"Integrated Security=True;" +
r'SERVER=' + server +
r';UID=' + username +
r';PWD=' + password +
r';DSN=MSSQL-PYTHON' +
r';DATABASE=' + database + ';'
)
print("Your Connection String:\n" + str(connStr) + "\n\n")
# =============================================================================
# CONNECT TO THE DB
# =============================================================================
cnxn = pyodbc.connect(connStr, autocommit=True)
# =============================================================================
# SET QUERIES TO VARIABLES
# =============================================================================
SQLQUERY1 = ("SELECT ##VERSION;")
SQLQUERY2 = ("SELECT * FROM sys.schemas;")
SQLQUERY3 = ("SELECT * FROM INFORMATION_SCHEMA.TABLES;")
SQLQUERY4 = ("SELECT * FROM INFORMATION_SCHEMA.COLUMNS;")
SQLQUERY5 = ("SELECT * FROM INFORMATION_SCHEMA.CHECK_CONSTRAINTS;")
SQLQUERY6 = ("EXEC sp_databases;")
SQLQUERY7 = ("EXEC sp_who2 'active';")
# =============================================================================
# RUN QUERIES
# YOU CAN RUN AS MANY QUERIES AS LONG AS THE CONNECTION IS OPEN TO THE DB
# =============================================================================
runningwithqueries(SQLQUERY1)
runningwithqueries(SQLQUERY2)
runningwithqueries(SQLQUERY3)
runningwithqueries(SQLQUERY4)
runningwithqueries(SQLQUERY5)
runningwithqueries(SQLQUERY6)
runningwithqueries(SQLQUERY7)
# =============================================================================
# CLOSE THE CONNECTION TO THE DB
# =============================================================================
cnxn.close()
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};' 'Server=**SERVER NAME**;' 'Database=**DATABASE NAME**;' 'Trusted_Connection=yes;')
cursor = conn.cursor()
cursor.execute('SELECT * FROM Output3')
This works just check you specify the Respective Driver, Server and the Database names correctively!
I have a connection string that looks like this
con_str = "myuser/mypass#oracle.sub.example.com:1521/ora1"
Where ora1 is the SID of my database. Using this information in SQL Developer works fine, meaning that I can connect and query without problems.
However, if I attempt to connect to Oracle using this string, it fails.
cx_Oracle.connect(con_str)
DatabaseError: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
This connection string format works if the ora1 is a service name, though.
I have seen other questions that seem to have the reverse of my problem (it works with SID, but not Service name)
Using Oracle Service Names with SQLAlachemy
Oracle SID and Service name; connection problems
cx_Oracle & Connecting to Oracle DB Remotely
What is the proper way to connect to Oracle, using cx_Oracle, using an SID and not a service name? How do I do this without the need to adjust the TNSNAMES.ORA file? My application is distributed to many users internally and making changes to the TNSNAMES file is less than ideal when dealing with users without administrator privileges on their Windows machines. Additionally, when I use service name, I don't need to touch this file at all and would like it keep it that way.
I a similar scenario, I was able to connect to the database by using cx_Oracle.makedsn() to create a dsn string with a given SID (instead of the service name):
dsnStr = cx_Oracle.makedsn("oracle.sub.example.com", "1521", "ora1")
This returns something like
(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle.sub.example.com)(PORT=1521)))(CONNECT_DATA=(SID=ora1)))
which can then be used with cx_Oracle.connect() to connect to the database:
con = cx_Oracle.connect(user="myuser", password="mypass", dsn=dsnStr)
print con.version
con.close()
For those looking for how to specify service_name instead of SID.
From changelog for SQLAlchemy 1.0.0b1 (released on March 13, 2015):
[oracle] [feature] Added support for cx_oracle connections to a
specific service name, as opposed to a tns name, by passing
?service_name=<name> to the URL. Pull request courtesy Sławomir
Ehlert.
The change introduces new, Oracle dialect specific option service_name which can be used to build connect string like this:
from sqlalchemy import create_engine
from sqlalchemy.engine import url
connect_url = url.URL(
'oracle+cx_oracle',
username='some_username',
password='some_password',
host='some_host',
port='some_port',
query=dict(service_name='some_oracle_service_name'))
engine = create_engine(connect_url)
If you are using sqlalchemy and ORACLE 12, the following seems to work.
from sqlalchemy import create_engine
con='oracle://user:password#hostname:1521/?service_name=DDDD'
engine = create_engine(con)
Note, you have to use the service name and not the SID. I don't know why, but the simple connection string that uses SID does not work.
It still may not work. You need to take the output of dsnStr and modify the string by replacing SID with SERVICE_NAME and use that variable in the con string. This procedure worked for me.
SID's may not be easily accessible or you might not have it created for your database.
In my case, I'm working from the client side requesting access to a cloud database so creating an SID didn't really make sense.
Instead, you might have a string that looks similar to this:
"(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = something.cloud.company)
(PORT = 12345)) (ADDRESS = (PROTOCOL = TCP)(HOST = something.cloud.company)
(PORT = 12345)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME =
something.company)))"
You can use it in replacement of the SID.
connection = cx_Oracle.connect("username", "pw", "(DESCRIPTION = (ADDRESS =
(PROTOCOL = TCP)(HOST = something.cloud.company)(PORT = 12345)) (ADDRESS =
(PROTOCOL = TCP)(HOST = something.cloud.company)(PORT = 12345))
(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = something.company)))")
I thought during a while that I would not be able to use Magic SQL (%sql, %%sql) because of service name issue in connection that would force to use the alternative way described above with cx_Oracle.connect(), cx_Oracle.makedsn()... I finally found a solution working for me: declare and set a variable for the service name first and then use it in the command (since not working if literal string for service name put in the command !)
import cx_Oracle
user='youruser'
pwd='youruserpwd'
dbhost='xx.xx.xx.xx'
service='yourservice'
%load_ext sql
%sql oracle+cx_oracle://$user:$pwd#$dbhost:1521/?service_name=$service
output (what you get in successful connection):
u'Connected: youruser#'
If someone is looking to set oracle.jdbc.proxyClientName property for cx_oracle, to connect using proxyClient, they can use -
cx_Oracle.init_oracle_client("../../oracle_local_client", config_dir= "../../oracle_local_client/network/admin")
connectDsn = cx_Oracle.makedsn('db.svr.net', 'portNumberHere',service_name="TEST_READWRITE")
#replace all prams above
pool = cx_Oracle.SessionPool(externalauth=True, homogeneous=False, dsn = connectDsn)
connection = pool.acquire(user="[PROXY_CLIENT_NAME]")
Notice the use of '[' braces to depict that the user is proxyClient.
I am using Kerberos authentication for this and my SQLNET.ora file contains the below properties.
NAMES.DIRECTORY_PATH=(TNSNAMES,HOSTNAME,EZCONNECT)
SQLNET.AUTHENTICATION_SERVICES = (BEQ,KERBEROS5PRE,KERBEROS5)
SQLNET.AUTHENTICATION_KERBEROS5_SERVICE=oracle
SQLNET.KERBEROS5_CC_NAME=OSMSFT:
SQLNET.KERBEROS5_CONF_MIT=TRUE
SQLNET.KERBEROS5_CONF=I:\projects\poc\resources\krb5.conf # krb5 config file complete path.
For more information, refer the video embedded in this article.
I also met this issue.
The solution is:
1: get the service name at tnsnames.ora
2: put the service name in
con_str = "myuser/mypass#oracle.sub.example.com:1521/ora1"