I'm using SQLAlchemy (Core only, not ORM) to create a connection to a SQL Server 2008 SP3.
When looking at the process' network connections, I noticed that the TCP/IP connection to the SQL Server (port 1433) remains open (ESTABLISHED).
Sample code:
from urllib.parse import quote_plus
from sqlalchemy.pool import NullPool
import sqlalchemy as sa
# parameters are read from a config file
db_params = quote_plus(';'.join(['{}={}'.format(key, val) for key, val in db_config.items()]))
# Hostname based connection
engine = sa.create_engine('mssql:///?odbc_connect={}'.format(db_params),
poolclass=NullPool)
conn = engine.connect()
conn.close()
engine.dispose()
engine = None
I added the NullPool and the engine.dispose() afterwards, thinking they might solve the lingering connection, but alas.
I'm using as hostname based connection as specified here.
Versions:
Python 3.5.0 (x32 on Win7)
SQLAlchemy 1.0.10
pyODBC 3.0.10
Edit: I've rewritten my code to solely use pyODBC instead of SQLAlchemy + pyODBC, and the issue remains. So as far as I can see, the issue is caused by pyODBC keeping the connection open.
When only pyODBC, the issue is because of connection pooling as discussed here.
As described in the docs:
pooling
A Boolean indicating whether connection pooling is enabled.
This is a global (HENV) setting, so it can only be modified before the
first connection is made. The default is True, which enables ODBC
connection pooling.
Thus:
import pyodbc
pyodbc.pooling = False
conn = pyodbc.connect(db_connection_string)
conn.close()
It seems that when using SQLAlchemy and disabling the SA pooling by using the NullPool, this isn't passed down to pyODBC.
Related
How do I port forward to an rds instance from a bastion host to a rds sql server instance?
Typically, I port forward from a bastion host like so: ssh -L 20004:targeted-mssql-rds.aws.com:1433 knn0926#my_bastion_host.com and that will allow me to call localhost:20004 for my db connection. However, that doesn't seem to work or to be the correct process for connecting to a db through port forwarding out of a python script. Every time I try to connect I get the following error: pyodbc.OperationalError: ('HYT00, [HYT00] [Mircosoft][ODBC Driver 17 for SQL Server]Login timeout expired(0) (SqlDriverConnect)). I'm also on a mac so I'm not sure if that makes a difference compared to windows.
I think my code is pretty straightforward for connecting:
import pyodbc
import pandas as pd
import io
import os
#listing drivers
for driver in pyodbc.drivers():
print(driver)
#Make db connection
sql_connect = pyodbc.connect("Driver = {ODBC Driver for SQL Server}; SERVER=localhost,20004; database=simpsons;uid=my_user;PWD=my_pwd, autocommit=True")
cursor.execute('''select * from spp.transactions where acx_num > 1000''')
Am I supposed to be connecting to my bastion and mssql instance in the python code itself? I'm confused as to why its failing and how to properly connect it to my code.
I'm trying to connect my database using SSL with PyMySQL, but I can't find good documentation on what the syntax is.
These credentials work in Workbench and with the CLI, but I get this error when using PyMySQL.
Can't connect to MySQL server on 'server.domain.com' ([WinError 10061] No connection could be made because the target machine actively refused it)")
db_conn = pymysql.connect(
host=db_creds['host'],
user=db_creds['user'],
passwd=db_creds['passwd'],
db=db_creds['db'],
charset=db_creds['charset'],
ssl={'ssl':{'ca': 'C:/SSL_CERTS/ca-cert.pem',
'key' : 'C:/SSL_CERTS/client-key.pem',
'cert' : 'C:/SSL_CERTS/client-cert.pem'
}
}
)
If I shut SSL off and drop the SSL parameter, I can connect unsecured just fine. What am I doing wrong with the SSL parameter?
Edit: PyMySQL now wants ssl parameters listed like this instead of in a dict.
db_conn = pymysql.connect(
host=db_creds['host'],
user=db_creds['user'],
passwd=db_creds['passwd'],
db=db_creds['db'],
charset=db_creds['charset'],
ssl_ca='C:/SSL_CERTS/ca-cert.pem',
ssl_key='C:/SSL_CERTS/client-key.pem',
ssl_cert='C:/SSL_CERTS/client-cert.pem'
)
Thanks for the help everyone. The syntax listed in the question is right, but the server I was attempting a connection to was using a non-standard port. I needed to add
port = db_creds['port']
Thanks, MannyKary, for the clue.
I had the same problem connecting pyMysql using client-side cert and key for users that REQUIRE X509, the TiDB (mySQL 5.7 compatible) server complained that no cert was supplied!!!
[2021/05/18 16:31:23.881 +00:00] [INFO] [privileges.go:258] ["ssl check failure, require x509 but no verified cert"] [user=mindline_root] [host=%]
Looking through the sourcecode of PyMysql 1.0.2, it appears that the ssl parameter is now a boolean instead of a ssl_dict, so you should put all your ssl parameters into individual arguements, e.g.,
db_conn = pymysql.connect(
host=db_creds['host'],
user=db_creds['user'],
passwd=db_creds['passwd'],
db=db_creds['db'],
charset=db_creds['charset'],
ssl_ca='C:/SSL_CERTS/ca-cert.pem',
ssl_key='C:/SSL_CERTS/client-key.pem',
ssl_cert='C:/SSL_CERTS/client-cert.pem'
)
I just started to learn python and try to connect to oracle 11g, but I always get following error
cx_Oracle.InternalError: No Oracle error?
Here is my simple script to connect to oracle
import cx_Oracle as oracle
con = oracle.connect('user/password#ip:port/service')
Already try to look for any reference in other sites including here but can't find the solution. I don't think I have connection issue to oracle, because I use the same PC to connect to oracle using PHP.
Any advise would be appreciated, thanks.
One thing to keep in mind anytime you work with Oracle is that they use a proprietary connection protocol TNS (Transparent Network Substrate).
Therefore, you might need to use the cx_Oracle.makedsn(ip, port, SID) method and then pass it to cx_Oracle.connect() method to create your connection. Thus the general format on how to set up Oracle connection is:
import cx_Oracle
ip = 'xxx.xxx.xx.xxx'
port = 'xxxx'
SID = 'SID'
username = 'username'
password = 'password'
dsn_tns = cx_Oracle.makedsn(ip, port, SID)
db = cx_Oracle.connect(username, password, dsn_tns)
This is assuming you have already gotten cx_Oracle to work and import properly, which can be finicky depending on your environment.
Is there an equivalent way to connect to a postgres database in a similar way that sqlite connects to a database using python?
For example, in sqlite, a connection will be defined by conn = sqlite3.connect(curruser.dbname). What is the similar connection syntax for postgres?
You can use the psycopg connector then, and the connection syntax will be similar, except that you'll need to specify some more information in a connection string:
conn_string = "host='localhost' dbname='my_database' user='postgres' password='secret'"
conn = psycopg2.connect(conn_string)
Here are some examples: Using psycopg2 with PostgreSQL
The location for mysql.sock on my system is /usr/local/mysql5/mysqld.sock
thrilllap-2:tmp reuven$ mysqld --print-defaults
mysqld would have been started with the following arguments:
--socket=/usr/local/mysql5/mysqld.sock --port=3306
When I try to use mysql via sqlalchemy from flask, I get:
File "build/bdist.macosx-10.6-intel/egg/MySQLdb/connections.py", line 187, in __init__
sqlalchemy.exc.OperationalError: (OperationalError) (2002, "Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)") None None
The mysql program connects correctly to the database, as does every other mysql client on my system.
My my.cnf has the correct location for the socket
[client]
port = 3306
socket = /usr/local/mysql5/mysqld.sock
[safe_mysqld]
socket = /usr/local/mysql5/mysqld.sock
[mysqld_safe]
socket = /usr/local/mysql5/mysqld.sock
[mysqld]
socket = /usr/local/mysql5/mysqld.sock
port = 3306
The base "SQLAlchemy" library has an option where you can specify the location of the mysql.sock, but this isn't exposed through the sqlalchemy / flask library
http://packages.python.org/Flask-SQLAlchemy/config.html
My questions:
Where does sqlalchemy get the idea that /tmp/mysql.sock is the correct location?
Is there a way to change the default via the Flash-SQLAlchemy connector
You'll have to dig up the exact syntax, but for MySQL I think they use a unix_socket query opt. Something like:
mysql:///dbname?unix_socket=/opt/mysql/mysql.sock'
Should be your connect URI for SQLAlchemy.
Yes! Sean was right
app.config['SQLALCHEMY_DATABASE_URI'] = ''mysql://dayenu:secret.word#localhost/dayenu?unix_socket=/usr/local/mysql5/mysqld.sock
db = SQLAlchemy(app)
works fine! I think this parameter is used by pyodbc, which is what SQLAlchemy uses to talk to mysql, but I couldn't find this parameter anywhere in the pyodbc documentation.
You could create the conexión using
sqlalchemy.create_engine(
mysql_str = sqlalchemy.engine.url.URL(
drivername='mysql+pymysql',
username="db_user",
password="db_pass",
database=db_name,
query={
'unix_socket': '/usr/local/mysql5/mysqld.sock'
}
)
)