I've recently changed my project to use SQLAlchemy and my project runs fine, it used an external MySQL server.
Now I'm trying to work with a different MySQL server with SSL CA, and it doesn't connect.
(It did connect using MySQL Workbench, so the certificate should be fine)
I'm using the following code:
ssl_args = {'ssl': {'ca': ca_path}}
engine = create_engine("mysql+pymysql://<user>:<pass>#<addr>/<schema>",
connect_args=ssl_args)
and I get the following error:
Can't connect to MySQL server on '\addr\' ([WinError 10054] An existing connection was forcibly closed by the remote host)
Any suggestions?
I changed the DBAPI to MySQL-Connector, and used the following code:
ssl_args = {'ssl_ca': ca_path}
engine = create_engine("mysql+mysqlconnector://<user>:<pass>#<addr>/<schema>",
connect_args=ssl_args)
And now it works.
If you just connect from a client machine with an ssl connection (so you don't have access to the cert and key), you could simple add ssl=true to your uri.
Edit:
For example:
mysql_db = "mysql+mysqlconnector://<user>:<pass>#<addr>/<schema>?ssl=true"
The official doc is well documented:
engine = create_engine(
db_url,
connect_args={
"ssl": {
"ssl_ca": "ca.pem",
"ssl_cert": "client-cert.pem",
"ssl_key": "client-key.pem"
}
}
)
Another solution is to use sqlalchemy.engine.url.URL to define the URL and pass it to create_engine.
sqlUrl = sqlalchemy.engine.url.URL(
drivername="mysql+pymysql",
username=db_user,
password=db_pass,
host=db_host,
port=3306,
database=db_name,
query={"ssl_ca": "main_app/certs/BaltimoreCyberTrustRoot.crt.pem"},
)
create_engine(sqlUrl)
You can include SSL parameters as a dictionary in the query argument.
This approach is useful if you are using Flask to initialize the SqlAlchemy engine with a config parameter like SQLALCHEMY_DATABASE_URI rather than directly using create_engine.
Related
Im trying to connect Sybase database with flask SQLalchemy using a ODBC connection .
My connection string :
'SQLALCHEMY_DATABASE_URI' = "sybase+pyodbc://username:passw#rd#host:port/dbname?driver=Adaptive+Server+Enterprise"
Getting this Error :
But I'm pretty sure i'm using the right port in the connection string. But when i try connect to connect to the same instance using this method .
Second method :
con = pyodbc.connect(server=server ,port=port ,username=username ,password=password ,driver=driver)
The connection works perfectly fine now with the same connection details.
Can anyone help me in building the connection string URL and help me fix this. Cause I want to use the "db Object" instead of "cursor Object ".
As noted in the Getting Connected wiki page, "Hostname Connections" are not supported. You can either create an ODBC DSN, or use your pyodbc connection string with an ODBC direct pass-through connection:
import urllib
from sqlalchemy import create_engine
connection_string = (
'DRIVER=SAP ASE ODBC driver;'
'SERVER=centos7-vm01;'
'PORT=5000;'
'UID=scott;PWD=tiger;'
'DATABASE=mydatabase;'
'charset=utf8;'
)
connection_uri = f"sybase+pyodbc:///?odbc_connect={urllib.parse.quote_plus(connection_string)}"
engine = create_engine(connection_uri)
After so much research I found that, this error is due incorrect parameter in the connection string.
Invalid port number error will be also fixed with this block of code .
Code here :
import urllib
connection_string = (
'DRIVER=Adaptive Server Enterprise;'
'SERVER=server;'
'PORT=port;'
'UID=username;PWD=password;'
'DATABASE=dbname;'
)
connection_uri = f"sybase+pyodbc:///?odbc_connect={urllib.parse.quote_plus(connection_string)}"
SQLALCHEMY_DATABASE_URI = connection_uri # connection string for SQLALchemy
This code is best solution .
I set up the Cloud SQL instance on Google Cloud Platform and followed the official instructions, but don't seem to be able to connect to the Cloud SQL instance. When I try to do a sanity check and access the PostgreSQL db through Cloud Shell, I'm able to connect successfully though.
Could someone please help - I would be much obliged.
Code:
from sqlalchemy import create_engine
engine = create_engine('postgresql+psycopg2://<user>:<pass>#<public IP Address/<table>')
engine.connect()
Error:
Is the server running on host "XX.XX.XXX.XX" and accepting
TCP/IP connections on port XXXX?
I found another way to connect to a PostgreSQL GCP instance without using the Cloud SQL Proxy.
Code:
import sqlalchemy
username = '' # DB username
password = '' # DB password
host = '' # Public IP address for your instance
port = '5432'
database = '' # Name of database ('postgres' by default)
db_url = 'postgresql+psycopg2://{}:{}#{}:{}/{}'.format(
username, password, host, port, database)
engine = sqlalchemy.create_engine(db_url)
conn = engine.connect()
I whitelisted my IP address before trying to connect. (https://cloud.google.com/sql/docs/postgres/connect-external-app#appaccessIP)
Use the Cloud SQL proxy to connect to Cloud SQL from external applications.
In order to achieve this please follow the relevant documentation.
The steps described would consist of:
Enabling the Cloud SQL Admin API on your Cloud Console.
Installing the relevant proxy client according to your OS.
Use any of the available methods to authenticate the Cloud SQL Proxy.
Invoke the proxy with ./cloud_sql_proxy -instances=INSTANCE_CONNECTION_NAME=tcp:5432 & ond your terminal and connect the proxy by changing your code and using SQLALCHEMY:
from sqlalchemy import create_engine
engine = create_engine('postgresql+psycopg2://DATABASE_USER:PASSWORD#localhost:5432/')
NOTE: the code above assumes you are not trying to connect to the proxy in a production environment and are using an authenticated Cloud SDK client in order to connect to the proxy.
This worked to me using the Cloud SQL Proxy on my personal computer and uploading the code to Google App Engine standard.
db_user = os.environ.get('CLOUD_SQL_USERNAME')
db_pass = os.environ.get('CLOUD_SQL_PASSWORD')
db_name = os.environ.get('CLOUD_SQL_DATABASE_NAME')
db_connection_name = os.environ.get('CLOUD_SQL_CONNECTION_NAME')
if os.environ.get('GAE_ENV') == 'standard':
db_uri = f'postgresql+psycopg2://{db_user}:{db_pass}#/{db_name}?host=/cloudsql/{db_connection_name}'
else:
db_uri = f'postgresql+psycopg2://{db_user}:{db_pass}#127.0.0.1:1234/{db_name}'
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = db_uri
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
Depending on the database client library, the socket (/cloudsql/INSTANCE_CONNECTION_NAME/.s.PGSQL.5432) needs to be specified.
The docs have this example for SQLAlchemy:
db_user = os.environ["DB_USER"]
db_pass = os.environ["DB_PASS"]
db_name = os.environ["DB_NAME"]
db_socket_dir = os.environ.get("DB_SOCKET_DIR", "/cloudsql")
cloud_sql_connection_name = os.environ["CLOUD_SQL_CONNECTION_NAME"]
pool = sqlalchemy.create_engine(
# Equivalent URL:
# postgresql+pg8000://<db_user>:<db_pass>#/<db_name>
# ?unix_sock=<socket_path>/<cloud_sql_instance_name>/.s.PGSQL.5432
sqlalchemy.engine.url.URL.create(
drivername="postgresql+pg8000",
username=db_user, # e.g. "my-database-user"
password=db_pass, # e.g. "my-database-password"
database=db_name, # e.g. "my-database-name"
query={
"unix_sock": "{}/{}/.s.PGSQL.5432".format(
db_socket_dir, # e.g. "/cloudsql"
cloud_sql_connection_name) # i.e "<PROJECT-NAME>:<INSTANCE-REGION>:<INSTANCE-NAME>"
}
),
**db_config
)
Be aware that this example is with pg8000 that uses unix_sock instead of unix_socket as socket identifier.
I have a MySQL Server set up to use SSL and I also have the CA Certificate.
When I connect to the server using MySQL Workbench, I do not need the certificate. I can also connect to the server using Python and MySQLdb on a Mac without the CA-certificate.
But when I try to connect using the exact same setup of Python and MySQLdb on a windows machine, I get access denied. It appears that I need the CA. And when I enter the CA, I get the following error
_mysql_exceptions.OperationalError: (2026, 'SSL connection error')
My code to open the connection is below:
db = MySQLdb.connect(host="host.name",
port=3306,
user="user",
passwd="secret_password",
db="database",
ssl={'ca': '/path/to/ca/cert'})
Could anyone point out what the problem is on a windows?
I just got the following to work with Python 2.7 and MySQLdb (1.2.4):
database = MySQLdb.connect(host='hostname', user='username', db='db_name',
passwd='PASSWORD', ssl={'ca': '/path/to/ca-file'})
This is what you had so there must be something else going on here. I wonder if you have something either incorrect with the your local CA file or possibly the cert on the server? Can you get a copy of the CA file from the server?
Try this
import ssl
from databases import Database
sslctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH,
cafile='ca.pem')
sslctx.verify_mode = ssl.CERT_REQUIRED
sslctx.check_hostname = True
sslctx.load_cert_chain(certfile='client.crt', keyfile='pkey.key')
database = Database(DATABASE_URL, ssl=sslctx)
databases library provides support for PostgreSQL, MySQL, and SQLite.
Also useful for async frameworks.
Im using pymysql and I had some problems to connect using SSL keys and certs: for the "ssl" attribute I set up as a dictionary inside. Try as below:
db = MySQLdb.connect(host="host.name",
port=3306,
user="user",
passwd="secret_password",
db="database",
ssl={'ssl':
{'ca': '/path/to/ca/ca',
'key': '/path/to/ca/key',
'cert': '/path/to/ca/cert'
}
}
)
I know this is a bit old but I found a way to get this to work. Use pymysql instead of MySQLdb and write the connection as:
import pymysql
conn = pymysql.connect(user = 'user', password = 'passwd'
, database = 'db', host = 'hst', ssl = {'ssl' : {'ca': 'pathtosll/something.pem'}})
The point people miss (including myself) is that ssl needs to be a dictionary containing
a key 'ssl' which has another dictionary as a value with a key 'ca'. This should work for you.
import pymysql
conn = pymysql.connect(host= # your host, usually localhost,
user = # your username,
passwd = # your password,
db = #your database name ,
ssl ={'ssl': r'path of your pem file'})
Update: I've confirmed this is only a problem when using an Azure SQL instance. I can use the same conn string to connect to local, network, and remote SQL (AWS) instances - it is only failing when connecting to Azure. I can connect to the Azure instance with other tools, like Management Studio.
I am building a small Python(3.4.x)/Flask application. I'm a complete noob here so forgive me if I break any rules in posting.
I have created the database engine with:
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('mssql+pymssql://dbadmin:dbadminpass#somedomain.server.net/databasename?charset=utf8')
db_session = scoped_session(sessionmaker(autocommit = False, autoflush = False, bind = engine))
Base = declarative_base()
Base.query = db_session.query_property()
def init_db():
import models
Base.metadata.creat_all(bind=engine)
Everything builds/interprets correctly at runtime but I get an error on running the query:
usr = User.query.filter_by(username=form.user.data).first()
The error is:
sqlalchemy.exc.OperationalError: (OperationalError) (20002, b'DB-Lib error message 20002, severity 9:\nAdaptive Server connection failed\n') None None
packages are: Flask==0.10.1, pymssql==2.1.1, SQLAlchemy==0.9.8
Thanks in advance.
I had similar problem and solved it by explicitly setting tds version = 7.0. FreeTDS reads the user's ${HOME}/.freetds.conf before resorting to the system-wide sysconfdir/freetds.conf. So, I created ~/.freetds.conf with [global] section as:
[global]
tds version = 7.0
You can find more information on freetds.con: http://www.freetds.org/userguide/freetdsconf.htm
As I just had the same problem.
Since I could get pymssql to connect bypassing sqlalchemy, I figured everything else should be fine, so I used the create_engine parameter connect_args to pass everything straight to pymssql.connect.
server_name = "sql_server_name"
server_addres = server_name + ".database.windows.net"
database = "database_name"
username = "{}#{}".format("my_username", server_name)
password = "strong_password"
arguments = dict(server=server_addres, user=username,
password=password, database=database, charset="utf8")
AZURE_ENGINE = create_engine('mssql+pymssql:///', connect_args=arguments)
This works fine and does not require one to meddle with the .freetds.conf file at all.
Also, note that pymssql requires usernname to be in the form username#servername. For more information see the linked documentation.
I'm trying to connect to a SQL Server 2012 database using SQLAlchemy (with pyodbc) on Python 3.3 (Windows 7-64-bit). I am able to connect using straight pyodbc but have been unsuccessful at connecting using SQLAlchemy. I have dsn file setup for the database access.
I successfully connect using straight pyodbc like this:
con = pyodbc.connect('FILEDSN=c:\\users\\me\\mydbserver.dsn')
For sqlalchemy I have tried:
import sqlalchemy as sa
engine = sa.create_engine('mssql+pyodbc://c/users/me/mydbserver.dsn/mydbname')
The create_engine method doesn't actually set up the connection and succeeds, but
iIf I try something that causes sqlalchemy to actually setup the connection (like engine.table_names()), it takes a while but then returns this error:
DBAPIError: (Error) ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied. (17) (SQLDriverConnect)') None None
I'm not sure where thing are going wrong are how to see what connection string is actually being passed to pyodbc by sqlalchemy. I have successfully using the same sqlalchemy classes with SQLite and MySQL.
The file-based DSN string is being interpreted by SQLAlchemy as server name = c, database name = users.
I prefer connecting without using DSNs, it's one less configuration task to deal with during code migrations.
This syntax works using Windows Authentication:
engine = sa.create_engine('mssql+pyodbc://server/database')
Or with SQL Authentication:
engine = sa.create_engine('mssql+pyodbc://user:password#server/database')
SQLAlchemy has a thorough explanation of the different connection string options here.
In Python 3 you can use function quote_plus from module urllib.parse to create parameters for connection:
import urllib
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"UID=user;"
"PWD=password")
engine = sa.create_engine("mssql+pyodbc:///?odbc_connect={}".format(params))
In order to use Windows Authentication, you want to use Trusted_Connection as parameter:
params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"Trusted_Connection=yes")
In Python 2 you should use function quote_plus from library urllib instead:
params = urllib.quote_plus("DRIVER={SQL Server Native Client 11.0};"
"SERVER=dagger;"
"DATABASE=test;"
"UID=user;"
"PWD=password")
I have an update info about the connection to MSSQL Server without using DSNs and using Windows Authentication. In my example I have next options:
My local server name is "(localdb)\ProjectsV12". Local server name I see from database properties (I am using Windows 10 / Visual Studio 2015).
My db name is "MainTest1"
engine = create_engine('mssql+pyodbc://(localdb)\ProjectsV12/MainTest1?driver=SQL+Server+Native+Client+11.0', echo=True)
It is needed to specify driver in connection.
You may find your client version in:
control panel>Systems and Security>Administrative Tools.>ODBC Data
Sources>System DSN tab>Add
Look on SQL Native client version from the list.
Just want to add some latest information here:
If you are connecting using DSN connections:
engine = create_engine("mssql+pyodbc://USERNAME:PASSWORD#SOME_DSN")
If you are connecting using Hostname connections:
engine = create_engine("mssql+pyodbc://USERNAME:PASSWORD#HOST_IP:PORT/DATABASENAME?driver=SQL+Server+Native+Client+11.0")
For more details, please refer to the "Official Document"
import pyodbc
import sqlalchemy as sa
engine = sa.create_engine('mssql+pyodbc://ServerName/DatabaseName?driver=SQL+Server+Native+Client+11.0',echo = True)
This works with Windows Authentication.
I did different and worked like a charm.
First you import the library:
import pandas as pd
from sqlalchemy import create_engine
import pyodbc
Create a function to create the engine
def mssql_engine(user = os.getenv('user'), password = os.getenv('password')
,host = os.getenv('SERVER_ADDRESS'),db = os.getenv('DATABASE')):
engine = create_engine(f'mssql+pyodbc://{user}:{password}#{host}/{db}?driver=SQL+Server')
return engine
Create a variable with your query
query = 'SELECT * FROM [Orders]'
Execute the Pandas command to create a Dataframe from a MSSQL Table
df = pd.read_sql(query, mssql_engine())