I have a daemon that uses sqlalchemy to interact with MySQL database. Since interaction is seldom, the connections are prone to timing out. I've tried to fix the problem by setting various flags when creating the database engine, e.g. pool_recycle=3600, but nothing seems to help.
To help me debug the problem, I set the timeout of my local mysql server to 10 seconds, and tried the following program.
import time
import sqlalchemy
engine = sqlalchemy.engine.create_engine("mysql://localhost")
while True:
connection = engine.connect()
result = connection.execute("SELECT 1")
print result.fetchone()
connection.close()
time.sleep(15)
Surprisingly, I continue to get exceptions like the following:
sqlalchemy.exc.OperationalError: (OperationalError) (2006, 'MySQL server has gone away')
However, if I remove the call to connection.close(), the problem goes away. What is going on here? Why isn't sqlalchemy trying to establish a new connection each time I call connect()?
I'm using Python 2.7.3 with sqlalchemy 0.9.8 and MySQL 5.5.40.
the document mention:
MySQL features an automatic connection close behavior,
for connections that have been idle for eight hours or more.
To circumvent having this issue, use the pool_recycle option
which controls the maximum age of any connection:
engine = create_engine('mysql+mysqldb://...', pool_recycle=3600)
You can just put "pool_recycle" parameter when the create_engine is invoked.
I'm not 100% sure if this is going to fix your problem or not, but I ran into a similar issue when dealing with mysql. It was only fixed when I used pymysql to connect to the database.
You can install like this:
pip install pymysql
Which will work for both linux and windows (if you have it installed)
Then give your connection string like this:
import time
import sqlalchemy
engine = sqlalchemy.engine.create_engine("mysql+pymysql://localhost")
while True:
connection = engine.connect()
result = connection.execute("SELECT 1")
print result.fetchone()
connection.close()
time.sleep(15)
I get the following output when I run it:
(1,)
(1,)
(1,)
(1,)
On another note, I have found certain queries to break with SQLAlchemy 0.9.8. I had to install version 0.9.3 in order for my applications to not break anymore.
Related
I am trying to switch a pyodbc connection to sqlalchemy engine. My working pyodbc connection is:
con = pyodbc.connect('DSN=SageLine50v23;UID=#####;PWD=#####;')
This is what I've tried.
con = create_engine('pyodbc://'+username+':'+password+'#'+url+'/'+db_name+'?driver=SageLine50v23')
I am trying to connect to my Sage 50 accounting data but just can't work out how to build the connection string. This is where I downloaded the odbc driver https://my.sage.co.uk/public/help/askarticle.aspx?articleid=19136.
I got some orginal help for the pyodbc connection using this website (which is working) https://www.cdata.com/kb/tech/sageuk-odbc-python-linux.rst but would like to use SQLAlchemy for it connection with pandas. Any ideas? Assume the issue is with this part pyodbc://
According to this thread Sage 50 uses MySQL to store its data. However, Sage also provides its own ODBC driver which may or may not use the same SQL dialect as MySQL itself.
SQLAlchemy needs to know which SQL dialect to use, so you could try using the mysql+pyodbc://... prefix for your connection URI. If that doesn't work (presumably because "Sage SQL" is too different from "MySQL SQL") then you may want to ask Sage support if they know of a SQLAlchemy dialect for their product.
I am trying to pull a query using pyodbc and it keeps returning a reset by peer error, I can run the query no issue 100% of the time in Teradata SQL Assistant.
This happens with one other query I use but everything else I have done works using the same code.
I have tried running it multiple times, always works in Teradata SQL Assistant but not in python. Hundreds of other queries have worked with no issue, only two have given issues. I have tried slightly changing the queries with no luck.
I also tried in R with RODBC and it worked there.
I asked our DBA team and they said they have no process that would automatically boot that process and there is no issues with the query.
import pyodbc
import pandas.io.sql as psql
connection_info = 'DSN=xxxxxx'
conn = pyodbc.connect(connection_info)
sql1 = '''
QUERY HERE
'''
df = psql.read_sql_query(sql1, conn)
expect df to = resuls, instead get the following error:
10054 WSA E ConnReset: Connection reset by peer
I am using sqlalchemy to connect to MySQL database and found a strange behavior.
If I query
LOAD DATA LOCAL INFILE
'C:\\\\Temp\\\\JaydenW\\\\iata_processing\\\\icer\\\\rename\\\\ICER_2017-10-
12T09033
7Z023870.csv
It pops an error:
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1148, u'The used
command is not allowed with this MySQL versi
on') [SQL: u"LOAD DATA LOCAL INFILE
'C:\\\\Temp\\\\JaydenW\\\\iata_processing\\\\icer\\\\rename\\\\ICER_2017-10-
12T090337Z023870.csv' INTO TABLE genie_etl.iata_icer_etl LINES TERMINATED BY
'\\n'
IGNORE 1 Lines (rtxt);"] (Background on this error at:
http://sqlalche.me/e/2j85)
And I find the reason is that:
I need to set the parameter as
args = "mysql+pymysql://"+username+":"+password+"#"+hostname+"/"+database+"?
local_infile=1"
If I use MySQL official connection library. I do not need to do so.
myConnection = MySQLdb.connect(host=hostname, user=username, passwd=password, db=database)
Can anyone help me to understand the difference between the two mechanisms?
The reason is that the mechanisms use different drivers.
In SQLAlchemy you appear to be using the pymysql engine, which uses the PyMySQL Connection class to create the DB connection. That one requires the user to explicitly pass the local_infile parameter if they want to use the LOAD DATA LOCAL command.
The other example uses MySQLdb, which is basically a wrapper around the MySQL C API (and to my knowledge not the official connection library; that would be MySQL Connector Python, which is also available on SQLAlchemy as mysqlconnector). This one apparently creates the connection in a way that the LOAD DATA LOCAL is enabled by default.
I'm using python 3.4 (ActiveState) and pyodbc 3.0.7 on a Windows 7 box to connect to a SQL Server 2008 RC2 database running on Window NT 6.1.
The problem I'm having is that the code below fails silently. No changes are made to the database.
connection = pyodbc.connect("DRIVER={SQL Server};SERVER=(local);DATABASE=Kerb;UID=sa;PWD=password", autocommit=True)
cursor = connection.cursor()
cursor.execute('''INSERT INTO [My].[Sample] (Case) VALUES (1);''')
I've also attempted to force the insert with a commit statement (which, unless I'm mistaken, shouldn't be necessary due to the autocommit=True), this also fails with no output.
cursor.execute('''INSERT INTO [My].[Sample] (Case) VALUES (1);''')
cursor.commit()
So my solution so far has been to add a sleep, which has solved the problem. But I worry about implementing this solution in production as it doesn't take into account network lag, etc.
cursor.execute('''INSERT INTO [My].[Sample] (Case) VALUES (1);''')
time.sleep(1)
I believe my question may be related to:
pyODBC and SQL Server 2008 and Python 3
If anyone has any ideas for further debugging or has documentation regarding this bit of asynchronous behavior I would love to hear it.
Thanks!
Unfortunately it appears that PyODBC cannot execute insert statements without the use of a timeout. I have started using PyMSSQL and the timeout is no longer required for a successful commit.
I can connect via pyODBC to an unsupported database over ODBC. Queries appear to execute correctly. If I try to connect using mssql+pyodbc, I can't connect properly (image not found).
I've tried "base:///", "base+pyodbc:///", or "pyodbc:///".
Do I need to write my own "dialect" that doesn't make any changes to base, and are there any useful (up to date) guides on how to do this?
EDIT:
import pyodbc
conn = pyodbc.connect(DSN = "ODBCCONNECTIONNAME", UID = "ODBCUSER", PWD="PASSWORD")
cursor = conn.cursor()
Also works with this line replacing the connection above:
conn = pyodbc.connect("DSN=fmp_production;UID=odbc_user;PWD=Pwd222")
I can then run selects, and modifications fine, using standard SQL.
EDIT:
Okay, so I'm getting an error:
"Abort trap: 6" basically my python process is crashing out. I've tried testing with a functioning ODBC connection to a MySQL database, using:
engine = create_engine('mysql+pyodbc://root:rootpwd#testenvironment')
If I include the name of the database, then I get an image not found error instead. But I think the actual problem is whatever is crashing my python.