I would like to verify the SSL connection that SQLAlchemy sets up when using create_engine to connect to a PostgreSQL database. For example, if I have the following Python 3 code:
from sqlalchemy import create_engine
conn_string = "postgresql+psycopg2://myuser:******#someserver:5432/somedb"
conn_args = {
"sslmode": "verify-full",
"sslrootcert": "/etc/ssl/certs/ca-certificates.crt",
}
engine = create_engine(conn_string, connect_args=conn_args)
I know that I can print the contents of engine.__dict__, but it doesn't contain any information about the SSL settings (TLS version, cipher suite, etc) that it's using to connect:
{
'_echo': False,
'dialect': <sqlalchemy.dialects.postgresql.psycopg2.PGDialect_psycopg2 object at 0x7f988a217978>,
'dispatch': <sqlalchemy.event.base.ConnectionEventsDispatch object at 0x7f988938e788>,
'engine': Engine(postgresql+psycopg2://myuser:******#someserver:5432/somedb),
'logger': <Logger sqlalchemy.engine.base.Engine (DEBUG)>,
'pool': <sqlalchemy.pool.impl.QueuePool object at 0x7f988a238c50>,
'url': postgresql+psycopg2://myuser:******#someserver:5432/somedb
}
I know I can do something like SELECT * FROM pg_stat_ssl;, but does the SQLAlchemy engine store this kind of information as a class attribute / method?
Thank you!
I don't use postgres so hopefully this holds true for you.
SQLAlchemy takes the info that you provide in the url and passes it down to the underlying dbapi library that is also specified in the url, in your case it's psycopg2.
Your engine instance only connects to the database when needed, and sqlalchemy just passes the connection info along to the driver specified in the url which returns a connection that sqlalchemy uses.
Forgive that this is mysql, but should be fundamentally the same for you:
>>> engine
Engine(mysql+mysqlconnector://test:***#localhost/test)
>>> conn = engine.connect()
>>> conn
<sqlalchemy.engine.base.Connection object at 0x000001614ACBE2B0>
>>> conn.connection
<sqlalchemy.pool._ConnectionFairy object at 0x000001614BF08630>
>>> conn.connection.connection
<mysql.connector.connection_cext.CMySQLConnection object at 0x000001614AB7E1D0>
Calling engine.connect() returns a sqlalchemy.engine.base.Connection instance that has a connection property for which the docstring says:
The underlying DB-API connection managed by this Connection.
However, you can see from above that it actually returns a sqlalchemy.pool._ConnectionFairy object which from it's docstring:
Proxies a DBAPI connection...
Here is the __init__() method of the connection fairy, and as you can see it has a connection attribute that is the actual underlying dbapi connection.
def __init__(self, dbapi_connection, connection_record, echo):
self.connection = dbapi_connection
self._connection_record = connection_record
self._echo = echo
As to what info is available on the dbapi connection object, it depends on the implementation of that particular driver. E.g psycopg2 connection objects have an info attribute:
A ConnectionInfo object exposing information about the native libpq
connection.
That info object has attributes such as ssl_in_use:
True if the connection uses SSL, False if not.
And ssl_attribute:
Returns SSL-related information about the connection.
So you don't have to dig too deep to get at the actual db connection to see what is really going on.
Also, if you want to ensure that all client connections are ssl, you can always force them to.
Here´s a quick and dirty of what SuperShoot spelled out in detail:
>>> from sqlalchemy import create_engine
>>> db_string = "postgresql+psycopg2://myuser:******#someserver:5432/somedb"
>>> db = create_engine(db_string)
>>> conn = db.connect()
>>> conn.connection.connection.info.ssl_in_use
Should return True if using SSL.
In case someone is looking for PostgreSQL and pg8000, see the pg8000 docs.
For SSL defaults, it is:
import sqlalchemy
sqlalchemy.create_engine(url, connect_args={'ssl_context':True})
Related
I've recently changed my project to use SQLAlchemy and my project runs fine, it used an external MySQL server.
Now I'm trying to work with a different MySQL server with SSL CA, and it doesn't connect.
(It did connect using MySQL Workbench, so the certificate should be fine)
I'm using the following code:
ssl_args = {'ssl': {'ca': ca_path}}
engine = create_engine("mysql+pymysql://<user>:<pass>#<addr>/<schema>",
connect_args=ssl_args)
and I get the following error:
Can't connect to MySQL server on '\addr\' ([WinError 10054] An existing connection was forcibly closed by the remote host)
Any suggestions?
I changed the DBAPI to MySQL-Connector, and used the following code:
ssl_args = {'ssl_ca': ca_path}
engine = create_engine("mysql+mysqlconnector://<user>:<pass>#<addr>/<schema>",
connect_args=ssl_args)
And now it works.
If you just connect from a client machine with an ssl connection (so you don't have access to the cert and key), you could simple add ssl=true to your uri.
Edit:
For example:
mysql_db = "mysql+mysqlconnector://<user>:<pass>#<addr>/<schema>?ssl=true"
The official doc is well documented:
engine = create_engine(
db_url,
connect_args={
"ssl": {
"ssl_ca": "ca.pem",
"ssl_cert": "client-cert.pem",
"ssl_key": "client-key.pem"
}
}
)
Another solution is to use sqlalchemy.engine.url.URL to define the URL and pass it to create_engine.
sqlUrl = sqlalchemy.engine.url.URL(
drivername="mysql+pymysql",
username=db_user,
password=db_pass,
host=db_host,
port=3306,
database=db_name,
query={"ssl_ca": "main_app/certs/BaltimoreCyberTrustRoot.crt.pem"},
)
create_engine(sqlUrl)
You can include SSL parameters as a dictionary in the query argument.
This approach is useful if you are using Flask to initialize the SqlAlchemy engine with a config parameter like SQLALCHEMY_DATABASE_URI rather than directly using create_engine.
I'm totally new using sqlalchemy and postgresql. I read this tutorial to build the following piece of code :
import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy import engine
def connect(user, password, db, host='localhost', port=5432):
'''Returns a connection and a metadata object'''
# We connect with the help of the PostgreSQL URL
# postgresql://federer:grandestslam#localhost:5432/tennis
url = 'postgresql://{}:{}#{}:{}/{}'
url = url.format(user, password, host, port, db)
# The return value of create_engine() is our connection object
con = sqlalchemy.create_engine(url, client_encoding='utf8')
# We then bind the connection to MetaData()
meta = sqlalchemy.MetaData(bind=con, reflect=True)
return con, meta
con, meta = connect('federer', 'grandestslam', 'tennis')
con
engine('postgresql://federer:***#localhost:5432/tennis')
meta
MetaData(bind=Engine('postgresql://federer:***#localhost:5432/tennis'))
When running it I have this error :
File "test.py", line 22, in <module>
engine('postgresql://federer:***#localhost:5432/tennis')
TypeError: 'module' object is not callable
what should I do ? thanks !
So, your problem is happening because you've made this call:
from sqlalchemy import engine
And then you've used this later in the file:
engine('postgresql://federer:***#localhost:5432/tennis')
Strangely, in that section, you have some statements that are just con and meta with no assignments or calls or anything. I'm not sure what you're doing there. I would suggest that you check out SQLalchemy's page on engine and connection use to help get you sorted.
It will of course depend on exactly how you've set up your database. I used the declarative_base module in one of my projects, so my process of setting up a session to connect to my DB looks like this:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# Connect to Database and create database session
engine = create_engine('postgresql://catalog:catalog#localhost/menus')
Base.metadata.bind = engine
DBSession = sessionmaker(bind=engine)
session = DBSession()
And in my database setup file, I've assigned:
Base = declarative_base()
But you'll have to customize it a bit to your particular setup. I hope that helps.
Edit: I see now where those calls to con and meta were coming from, as well as your other confusing lines, it's part of the tutorial you linked to. What he was doing in that tutorial was using the Python interpreter in command line. I'll explain a few of the things he did there in the hope that it helps you some more. Lines beginning with >>> are what he enters in as commands. The other lines are the output he receives back.
>>> con, meta = connect('federer', 'grandestslam', 'tennis') # he creates the connection and meta objects
>>> con # now he calls the connection by itself to have it show that it's connected to his DB
Engine(postgresql://federer:***#localhost:5432/tennis)
>>> meta # here he calls his meta object to show how it, too, is connected
MetaData(bind=Engine(postgresql://federer:***#localhost:5432/tennis))
Update: I've confirmed this is only a problem when using an Azure SQL instance. I can use the same conn string to connect to local, network, and remote SQL (AWS) instances - it is only failing when connecting to Azure. I can connect to the Azure instance with other tools, like Management Studio.
I am building a small Python(3.4.x)/Flask application. I'm a complete noob here so forgive me if I break any rules in posting.
I have created the database engine with:
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('mssql+pymssql://dbadmin:dbadminpass#somedomain.server.net/databasename?charset=utf8')
db_session = scoped_session(sessionmaker(autocommit = False, autoflush = False, bind = engine))
Base = declarative_base()
Base.query = db_session.query_property()
def init_db():
import models
Base.metadata.creat_all(bind=engine)
Everything builds/interprets correctly at runtime but I get an error on running the query:
usr = User.query.filter_by(username=form.user.data).first()
The error is:
sqlalchemy.exc.OperationalError: (OperationalError) (20002, b'DB-Lib error message 20002, severity 9:\nAdaptive Server connection failed\n') None None
packages are: Flask==0.10.1, pymssql==2.1.1, SQLAlchemy==0.9.8
Thanks in advance.
I had similar problem and solved it by explicitly setting tds version = 7.0. FreeTDS reads the user's ${HOME}/.freetds.conf before resorting to the system-wide sysconfdir/freetds.conf. So, I created ~/.freetds.conf with [global] section as:
[global]
tds version = 7.0
You can find more information on freetds.con: http://www.freetds.org/userguide/freetdsconf.htm
As I just had the same problem.
Since I could get pymssql to connect bypassing sqlalchemy, I figured everything else should be fine, so I used the create_engine parameter connect_args to pass everything straight to pymssql.connect.
server_name = "sql_server_name"
server_addres = server_name + ".database.windows.net"
database = "database_name"
username = "{}#{}".format("my_username", server_name)
password = "strong_password"
arguments = dict(server=server_addres, user=username,
password=password, database=database, charset="utf8")
AZURE_ENGINE = create_engine('mssql+pymssql:///', connect_args=arguments)
This works fine and does not require one to meddle with the .freetds.conf file at all.
Also, note that pymssql requires usernname to be in the form username#servername. For more information see the linked documentation.
I'm trying to connect to a SQL Server 2012 database using SQLAlchemy (with pyodbc) on Python 3.3 (Windows 7-64-bit). I am able to connect using straight pyodbc but have been unsuccessful at connecting using SQLAlchemy. I have dsn file setup for the database access.
I successfully connect using straight pyodbc like this:
con = pyodbc.connect('FILEDSN=c:\\users\\me\\mydbserver.dsn')
For sqlalchemy I have tried:
import sqlalchemy as sa
engine = sa.create_engine('mssql+pyodbc://c/users/me/mydbserver.dsn/mydbname')
The create_engine method doesn't actually set up the connection and succeeds, but
iIf I try something that causes sqlalchemy to actually setup the connection (like engine.table_names()), it takes a while but then returns this error:
DBAPIError: (Error) ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied. (17) (SQLDriverConnect)') None None
I'm not sure where thing are going wrong are how to see what connection string is actually being passed to pyodbc by sqlalchemy. I have successfully using the same sqlalchemy classes with SQLite and MySQL.
The file-based DSN string is being interpreted by SQLAlchemy as server name = c, database name = users.
I prefer connecting without using DSNs, it's one less configuration task to deal with during code migrations.
This syntax works using Windows Authentication:
engine = sa.create_engine('mssql+pyodbc://server/database')
Or with SQL Authentication:
engine = sa.create_engine('mssql+pyodbc://user:password#server/database')
SQLAlchemy has a thorough explanation of the different connection string options here.
In Python 3 you can use function quote_plus from module urllib.parse to create parameters for connection:
import urllib
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"UID=user;"
"PWD=password")
engine = sa.create_engine("mssql+pyodbc:///?odbc_connect={}".format(params))
In order to use Windows Authentication, you want to use Trusted_Connection as parameter:
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"Trusted_Connection=yes")
In Python 2 you should use function quote_plus from library urllib instead:
params = urllib.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"UID=user;"
"PWD=password")
I have an update info about the connection to MSSQL Server without using DSNs and using Windows Authentication. In my example I have next options:
My local server name is "(localdb)\ProjectsV12". Local server name I see from database properties (I am using Windows 10 / Visual Studio 2015).
My db name is "MainTest1"
engine = create_engine('mssql+pyodbc://(localdb)\ProjectsV12/MainTest1?driver=SQL+Server+Native+Client+11.0', echo=True)
It is needed to specify driver in connection.
You may find your client version in:
control panel>Systems and Security>Administrative Tools.>ODBC Data
Sources>System DSN tab>Add
Look on SQL Native client version from the list.
Just want to add some latest information here:
If you are connecting using DSN connections:
engine = create_engine("mssql+pyodbc://USERNAME:PASSWORD#SOME_DSN")
If you are connecting using Hostname connections:
engine = create_engine("mssql+pyodbc://USERNAME:PASSWORD#HOST_IP:PORT/DATABASENAME?driver=SQL+Server+Native+Client+11.0")
For more details, please refer to the "Official Document"
import pyodbc
import sqlalchemy as sa
engine = sa.create_engine('mssql+pyodbc://ServerName/DatabaseName?driver=SQL+Server+Native+Client+11.0',echo = True)
This works with Windows Authentication.
I did different and worked like a charm.
First you import the library:
import pandas as pd
from sqlalchemy import create_engine
import pyodbc
Create a function to create the engine
def mssql_engine(user = os.getenv('user'), password = os.getenv('password')
,host = os.getenv('SERVER_ADDRESS'),db = os.getenv('DATABASE')):
engine = create_engine(f'mssql+pyodbc://{user}:{password}#{host}/{db}?driver=SQL+Server')
return engine
Create a variable with your query
query = 'SELECT * FROM [Orders]'
Execute the Pandas command to create a Dataframe from a MSSQL Table
df = pd.read_sql(query, mssql_engine())
On the command line, this works:
$ mongo
> show dbs
mydatabase 1.0GB
However, this does not:
$ python
>>> import pymongo
>>> connection = pymongo.MongoClient()
>>> connection.mydatabase.find()
I read through docs here:
http://api.mongodb.org/python/current/tutorial.html
But do not understand how to either...
connect to an existing database (using pymongo)
query what databases exist in the mongodb connection.
Why can't I access my database?
Connect to an existing database
import pymongo
from pymongo import MongoClient
connection = MongoClient()
db = connection.mydatabase
List existing databases
import pymongo
from pymongo import MongoClient
connection = MongoClient()
# connection.database_names() # depreciated
connection.list_database_names()
The question implies user has a local MongoDB. However I found this question trying to connect to a remote MongoDB. I think the tutorial is worth mentioning (no other answer here mentioned how I can specify the host and the port)
The above code will connect on the default host and port. We can also specify the host and port explicitly, as follows:
client = MongoClient('localhost', 27017)
Or use the MongoDB URI format:
client = MongoClient('mongodb://localhost:27017/')
show dbs and find() are totally different commands as such you cannot compare the two.
connection.mydatabase.find()
Will actually do nothing because you cannot find() documents on database level. You are probably looking for:
cursor = connection.mydatabase.mycol.find()
I am no Python programmer but something like that and the foreach the cursor var to get your data.
As an added note you will want to replace mycol with the collection name that contains your documents.
As for querying for a list of databases you can do something like:
databases = connection.mydatabase.command({'listDatabases': 1});
As shown here: http://docs.mongodb.org/manual/reference/command/listDatabases/#listDatabases
However again I am no Python programmer but this should get you started.
On the python command line:
import pymongo
from pymongo import MongoClient
connection = MongoClient() ## connects by default to db at localhost:27017
connection.database_names() ## python binding equivalent to show dbs.
Although there doesn't seem to be a wealth of examples, it appears that the bindings are pretty complete within the Python Driver API Documentation.
database_names() is deprecated. One can use list_database_names() instead.
mongo_db_url will be something like "mongodb://localhost:27017/". 27017 is deafult port number, replace suitably.
from pymongo import MongoClient
client = MongoClient(<mongo_db_url>)
#or client = MongoClient('localhost', 27017)
client.list_database_names()