We Keep Coding

Using SSL with SQLAlchemy

I've recently changed my project to use SQLAlchemy and my project runs fine, it used an external MySQL server.
Now I'm trying to work with a different MySQL server with SSL CA, and it doesn't connect.
(It did connect using MySQL Workbench, so the certificate should be fine)
I'm using the following code:
ssl_args = {'ssl': {'ca': ca_path}}
engine = create_engine("mysql+pymysql://<user>:<pass>#<addr>/<schema>",
connect_args=ssl_args)
and I get the following error:
Can't connect to MySQL server on '\addr\' ([WinError 10054] An existing connection was forcibly closed by the remote host)
Any suggestions?

I changed the DBAPI to MySQL-Connector, and used the following code:
ssl_args = {'ssl_ca': ca_path}
engine = create_engine("mysql+mysqlconnector://<user>:<pass>#<addr>/<schema>",
connect_args=ssl_args)
And now it works.

If you just connect from a client machine with an ssl connection (so you don't have access to the cert and key), you could simple add ssl=true to your uri.
Edit:
For example:
mysql_db = "mysql+mysqlconnector://<user>:<pass>#<addr>/<schema>?ssl=true"

The official doc is well documented:
engine = create_engine(
db_url,
connect_args={
"ssl": {
"ssl_ca": "ca.pem",
"ssl_cert": "client-cert.pem",
"ssl_key": "client-key.pem"
}
}
)

Another solution is to use sqlalchemy.engine.url.URL to define the URL and pass it to create_engine.
sqlUrl = sqlalchemy.engine.url.URL(
drivername="mysql+pymysql",
username=db_user,
password=db_pass,
host=db_host,
port=3306,
database=db_name,
query={"ssl_ca": "main_app/certs/BaltimoreCyberTrustRoot.crt.pem"},
)
create_engine(sqlUrl)
You can include SSL parameters as a dictionary in the query argument.
This approach is useful if you are using Flask to initialize the SqlAlchemy engine with a config parameter like SQLALCHEMY_DATABASE_URI rather than directly using create_engine.

Python sqlalchemy trying to write pandas dataframe to SQL Server using .to_sql

I have a python code through which I am getting a pandas dataframe "df". I am trying to write this dataframe to Microsoft SQL server. I am trying to connect through the following code by I am getting an error
import pyodbc
from sqlalchemy import create_engine
engine = create_engine('mssql+pyodbc:///?odbc_connect=DRIVER={SQL Server};SERVER=bidept;DATABASE=BIDB;UID=sdcc\neils;PWD=neil!pass')
engine.connect()
df.to_sql(name='[BIDB].[dbo].[Test]',con=engine, if_exists='append')
However at the engine.connect() line I am getting the following error
sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('08001', '[08001] [Microsoft][ODBC SQL Server Driver]Neither DSN nor SERVER keyword supplied (0) (SQLDriverConnect)')
Can anyone tell me what I am missing. I am using Microsoft SQL Server Management Studio - 14.0.17177.0
I connect to the SQL server through the following
Server type: Database Engine
Server name: bidept
Authentication: Windows Authentication
for which I log into my windows using username : sdcc\neils
and password : neil!pass
I am new to databases and python. Kindly let me know if you need any additional details. Any help will be greatly appreciated. Thank you in advance.

I was finally able to make it run.
import pyodbc
from sqlalchemy import create_engine
import urllib
params = urllib.quote_plus(r'DRIVER={SQL Server};SERVER=bidept;DATABASE=BIDB;Trusted_Connection=yes')
### For python 3.5: urllib.parse.quote_plus
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine = create_engine(conn_str)
reload(sys)
sys.setdefaultencoding('utf8')
df.to_sql(name='Test',con=engine, if_exists='append',index=False)
Thanks to #gord-thompson who answered Here
Although my in my sql server, all the tables are under the 'dbo' schema (i.e. dbo.Test1, dbo.Other_Tables) and this query puts my table in 'sdcc\neils' schema (i.e. sdcc\neils.Test1, sdcc\neils.Other_Tables) any solution to this?

Sqlalchemy pyodbc DNS-less URL connection string does not seem to work

I am trying to connect to a Sybase ASE 15 database on windows using sqlalchemy (1.0.9) and pyodbc. If I use a DNS url everything works as expected:
url = r'sybase+pyodbc://usename:password#dns'
engine = create_engine(url, echo=True)
Session = sessionmaker(bind=engine)
sess = Session()
conn = sess.connection()
However, if I avoid the DNS I get an error message:
url = 'sybase+pyodbc://username:password#host:port/database?driver=Adaptive Server Enterprise'
, I get an error:
DBAPIError: (pyodbc.Error) ('01S00', '[01S00] [SAP][ASE ODBC
Driver]Invalid port number (30011) (SQLDriverConnect)')
The port number is correct and it is the same port as specified in the DNS.
Any ideas?

It may be worth trying to work with some version of the semicolon-separated DSN-less format used by pyODBC (and ODBC in general). Some examples here:
http://www.connectionstrings.com/adaptive-server-enterprise-odbc-driver/
This question tackles a similar issue with FreeTDS, but the concept is the same, as the connect string is basically passed through to the low-level ODBC connect:
SqlAlchemy equivalent of pyodbc connect string using FreeTDS
The URL is being parsed down to this type of string for ultimate connection by pyodbc through to SQLDriverConnect (in the ODBC API), so specifying the ASE ODBC DSN-less connection string directly may work better.
Update: Ran a quick test to see what connect arguments are produced for this URL:
from sqlalchemy.engine.url import *
from sqlalchemy.connectors.pyodbc import *
connector = PyODBCConnector()
url = make_url("sybase+pyodbc://username:password#host:5555/database?driver=Adaptive Server Enterprise")
print connector.create_connect_args(url)
This results in:
[['DRIVER={Adaptive Server Enterprise};Server=host,5555;Database=database;UID=username;PWD=password'], {}]
Note that the hostname and port are separated by a comma, per http://www.connectionstrings.com/adaptive-server-enterprise-odbc-driver/tds-based-odbc-driver-from-sybase-ocs-125/, this format works for TDS-based ODBC for Sybase 12.5:
Driver={Sybase ASE ODBC Driver};NetworkAddress=myServerAddress,5000;
Db=myDataBase;Uid=myUsername;Pwd=myPassword;
However, the ASE 15 format (http://www.connectionstrings.com/adaptive-server-enterprise-odbc-driver/adaptive-server-enterprise-150/) specifies server=myServerAddress;port=myPortnumber with port as a key passed in the semicolon-delimited string:
Driver={Adaptive Server Enterprise};app=myAppName;server=myServerAddress;
port=myPortnumber;db=myDataBase;uid=myUsername;pwd=myPassword;
If you "cheat" on the port spec by using host;port=5555, you get:
[['DRIVER={Adaptive Server Enterprise};Server=host;port=5555;Database=database;UID=username;PWD=password'], {}]
But this just feels like a Bad Idea™, even if it works. I'd also note that the generated string is using Database as the key vs. Db in the Sybase connection string reference. This may prove to be an issue as well.
Using ?odbc_connect as in the linked question is probably your best option for controlling the exact connect arguments being sent to ODBC.

Adaptive Server connection failed error in Python/Flask/sqlAlchemy environment with pymssql

Update: I've confirmed this is only a problem when using an Azure SQL instance. I can use the same conn string to connect to local, network, and remote SQL (AWS) instances - it is only failing when connecting to Azure. I can connect to the Azure instance with other tools, like Management Studio.
I am building a small Python(3.4.x)/Flask application. I'm a complete noob here so forgive me if I break any rules in posting.
I have created the database engine with:
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('mssql+pymssql://dbadmin:dbadminpass#somedomain.server.net/databasename?charset=utf8')
db_session = scoped_session(sessionmaker(autocommit = False, autoflush = False, bind = engine))
Base = declarative_base()
Base.query = db_session.query_property()
def init_db():
import models
Base.metadata.creat_all(bind=engine)
Everything builds/interprets correctly at runtime but I get an error on running the query:
usr = User.query.filter_by(username=form.user.data).first()
The error is:
sqlalchemy.exc.OperationalError: (OperationalError) (20002, b'DB-Lib error message 20002, severity 9:\nAdaptive Server connection failed\n') None None
packages are: Flask==0.10.1, pymssql==2.1.1, SQLAlchemy==0.9.8
Thanks in advance.

I had similar problem and solved it by explicitly setting tds version = 7.0. FreeTDS reads the user's ${HOME}/.freetds.conf before resorting to the system-wide sysconfdir/freetds.conf. So, I created ~/.freetds.conf with [global] section as:
[global]
tds version = 7.0
You can find more information on freetds.con: http://www.freetds.org/userguide/freetdsconf.htm

As I just had the same problem.
Since I could get pymssql to connect bypassing sqlalchemy, I figured everything else should be fine, so I used the create_engine parameter connect_args to pass everything straight to pymssql.connect.
server_name = "sql_server_name"
server_addres = server_name + ".database.windows.net"
database = "database_name"
username = "{}#{}".format("my_username", server_name)
password = "strong_password"
arguments = dict(server=server_addres, user=username,
password=password, database=database, charset="utf8")
AZURE_ENGINE = create_engine('mssql+pymssql:///', connect_args=arguments)
This works fine and does not require one to meddle with the .freetds.conf file at all.
Also, note that pymssql requires usernname to be in the form username#servername. For more information see the linked documentation.

Connecting to SQL Server 2012 using sqlalchemy and pyodbc

I'm trying to connect to a SQL Server 2012 database using SQLAlchemy (with pyodbc) on Python 3.3 (Windows 7-64-bit). I am able to connect using straight pyodbc but have been unsuccessful at connecting using SQLAlchemy. I have dsn file setup for the database access.
I successfully connect using straight pyodbc like this:
con = pyodbc.connect('FILEDSN=c:\\users\\me\\mydbserver.dsn')
For sqlalchemy I have tried:
import sqlalchemy as sa
engine = sa.create_engine('mssql+pyodbc://c/users/me/mydbserver.dsn/mydbname')
The create_engine method doesn't actually set up the connection and succeeds, but
iIf I try something that causes sqlalchemy to actually setup the connection (like engine.table_names()), it takes a while but then returns this error:
DBAPIError: (Error) ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied. (17) (SQLDriverConnect)') None None
I'm not sure where thing are going wrong are how to see what connection string is actually being passed to pyodbc by sqlalchemy. I have successfully using the same sqlalchemy classes with SQLite and MySQL.

The file-based DSN string is being interpreted by SQLAlchemy as server name = c, database name = users.
I prefer connecting without using DSNs, it's one less configuration task to deal with during code migrations.
This syntax works using Windows Authentication:
engine = sa.create_engine('mssql+pyodbc://server/database')
Or with SQL Authentication:
engine = sa.create_engine('mssql+pyodbc://user:password#server/database')
SQLAlchemy has a thorough explanation of the different connection string options here.

In Python 3 you can use function quote_plus from module urllib.parse to create parameters for connection:
import urllib
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"UID=user;"
"PWD=password")
engine = sa.create_engine("mssql+pyodbc:///?odbc_connect={}".format(params))
In order to use Windows Authentication, you want to use Trusted_Connection as parameter:
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"Trusted_Connection=yes")
In Python 2 you should use function quote_plus from library urllib instead:
params = urllib.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"UID=user;"
"PWD=password")

I have an update info about the connection to MSSQL Server without using DSNs and using Windows Authentication. In my example I have next options:
My local server name is "(localdb)\ProjectsV12". Local server name I see from database properties (I am using Windows 10 / Visual Studio 2015).
My db name is "MainTest1"
engine = create_engine('mssql+pyodbc://(localdb)\ProjectsV12/MainTest1?driver=SQL+Server+Native+Client+11.0', echo=True)
It is needed to specify driver in connection.
You may find your client version in:
control panel>Systems and Security>Administrative Tools.>ODBC Data
Sources>System DSN tab>Add
Look on SQL Native client version from the list.

Just want to add some latest information here:
If you are connecting using DSN connections:
engine = create_engine("mssql+pyodbc://USERNAME:PASSWORD#SOME_DSN")
If you are connecting using Hostname connections:
engine = create_engine("mssql+pyodbc://USERNAME:PASSWORD#HOST_IP:PORT/DATABASENAME?driver=SQL+Server+Native+Client+11.0")
For more details, please refer to the "Official Document"

import pyodbc
import sqlalchemy as sa
engine = sa.create_engine('mssql+pyodbc://ServerName/DatabaseName?driver=SQL+Server+Native+Client+11.0',echo = True)
This works with Windows Authentication.

I did different and worked like a charm.
First you import the library:
import pandas as pd
from sqlalchemy import create_engine
import pyodbc
Create a function to create the engine
def mssql_engine(user = os.getenv('user'), password = os.getenv('password')
,host = os.getenv('SERVER_ADDRESS'),db = os.getenv('DATABASE')):
engine = create_engine(f'mssql+pyodbc://{user}:{password}#{host}/{db}?driver=SQL+Server')
return engine
Create a variable with your query
query = 'SELECT * FROM [Orders]'
Execute the Pandas command to create a Dataframe from a MSSQL Table
df = pd.read_sql(query, mssql_engine())