So, basically,someone was telling me today that the way I Query my database was bad before I never close the connection after finishing and thus I probably had many open connection with the database running simultaneously.
I looked online however, I did not get a straight answer. I am using MySQL Mariadb.
and this is how I am doing it
db = mysql.connector.connect(host='localhost', username, passwd, db='mydb')
# and actually call this function and pass query
def db_execute(query):
cursor = db.cursor()
cursor.execute(query)
db.commit()
is this decent? how can I close the connection after I finish? what's better?
db = mysql.connector.connect(host='localhost', username, passwd, db='mydb')
# and actually call this function and pass query
def db_execute(query):
cursor = db.cursor()
try:
cursor.execute(query)
db.commit()
#You can also catch any exceptions you are interested in here
finally:
cursor.close()
db.close()
Finally block gets executed even if you code throws an exception, hence your connection is always closed even if something unexpected happens. If you do not use finally and there is an exception in your code before the close statement, the connection remains open, which is undesired.
Promptly close anything inside the database. This applies to "cursors", "transactions", etc.
As for disconnect() (however it is spelled), the rules are different.
Do not do a connect-disconnect around each SQL statement; this is terribly inefficient.
In a web application, do connect at early in the "web page" and either disconnect when finished, or let the containing code (PHP, etc) deal with closing when it goes away. Web pages are short-lived and cannot leave "state" for the next page when the user clicks a link.
In a long-running application, be aware that after some number of minutes (configurable) of not touching the database, the connection to the database will be closed. If you then try to do some SQL, you will get some form of error. For such rare applications, you may need to connect+disconnect more than once. And you may as well do them at times convenient to the application.
Generally having lots of idle connections is "not a problem". However, if it exceeds max_connections, newcomers will get an error about 'cannot connect'.
"Connection pooling" adds another dimension to this discussion. I won't go there.
cursor.close() and then db.close()
Related
I have written a script that executes inside docker and writes to an sqlite3 db on host. At a time, there are 8-10 such docker containers writing to the same database (each having their own connection).
In my script, I create an sqlite connection and get a cursor and use it for the lifecycle of the whole program (multiple cur.execute(), con.commit(), con.rollback()).
I initiate the connection with 20 second timeout:
con = sqlite3.connect('database.db', timeout=20)
but I still get sqlite3: database is locked. errors.
I have tried this solution like this:
while True:
try:
cur.execute(f"...insert into db statement...")
break
except sqlite3.OperationalError:
continue
and similarly for con.commit() and con.rollback() in place of cur.execute().
My question is, is there a better way to do this? I am not checking if the while loop for any of the containers is stuck in infinite loop. I have seen answers creating connection and cursor inside and cursor inside the while loop, does creating a connection lock the database? does cur.execute() lock it or only con.commit()?
I'm kind of new to Python and its MySQLdb connector.
I'm writing an API to return some data from a database using the RESTful approach. In PHP, I wrapped the Connection management part in a class, acting as an abstraction layer for MySQL queries.
In Python:
I define the connection early on in the script: con = mdb.connect('localhost', 'user', 'passwd', 'dbname')
Then, in all subsequent methods:
import MySQLdb as mdb
def insert_func():
with con:
cur = con.cursor(mdb.cursors.DictCursor)
cur.execute("INSERT INTO table (col1, col2, col3) VALUES (%s, %s, %s)", (val1, val2, val3) )
rows = cur.fetchall()
#do something with the results
return someval
etc.
I use mdb.cursors.DictCursor because I prefer to be able to access database columns in an associative array manner.
Now the problems start popping up:
in one function, I issue an insert query to create a 'group' with unique 'groupid'.
This 'group' has a creator. Every user in the database holds a JSON array in the 'groups' column of his/her row in the table.
So when I create a new group, I want to assign the groupid to the user that created it.
I update the user's record using a similar function.
I've wrapped the 'insert' and 'update' parts in two separate function defs.
The first time I run the script, everything works fine.
The second time I run the script, the script runs endlessly (I suspect due to some idle connection to the MySQL database).
When I interrupt it using CTRL + C, I get one of the following errors:
"'Cursor' object has no attribute 'connection'"
"commands out of sync; you can't run this command now"
or any other KeyboardInterrupt exception, as would be expected.
It seems to me that these errors are caused by some erroneous way of handling connections and cursors in my code.
I read it was good practice to use with con: so that the connection will automatically close itself after the query. I use 'with' on 'con' in each function, so the connection is closed, but I decided to define the connection globally, for any function to use it. This seems incompatible with the with con: context management. I suspect the cursor needs to be 'context managed' in a similar way, but I do not know how to do this (To my knowledge, PHP doesn't use cursors for MySQL, so I have no experience using them).
I now have the following questions:
Why does it work the first time but not the second? (it will however, work again, once, after the CTRL + C interrupt).
How should I go about using connections and cursors when using multiple functions (that can be called upon in sequence)?
I think there are two main issues going on here- one appears to be python code and the other is the structure of how you're interacting to your DB.
First, you're not closing your connection. This depends on your application's needs - you have to decide how long it should stay open. Reference this SO question
from contextlib import closing
with closing( connection.cursor() ) as cursor:
... use the cursor ...
# cursor closed. Guaranteed.
connection.close()
Right now, you have to interrupt your program with Ctl+C because there's no reason for your with statement to stop running.
Second, start thinking about your interactions with the DB in terms of 'transactions'. Do something, commit it to the DB, if it didn't work, rollback, if it did, close the connection. Here's a tutorial.
With connections, as with file handles the rule of thumb is open late, close early.
So I would recommend share connections only where they are trying to do one thing. Or if you multiprocess, then each process gets a connection, again following open late, close early. And if you are doing sequential operation (say in a loop) open and close outside the loop. Having global connections can get messy. Mainly because now you have to keep track of which function uses it at what time, and what it tries to do with it.
The issue of "cannot run command now", is because your keyboard interrupt kills the active connection.
As to part one of your question - endlessly could be anywhere. Each instance of python will get its own connection. So when you run it the second time it should get its own connection. Open up a mysql client and do
show full processlist
to see whats going on.
I making a simple python script which checks a mysql table every x seconds and print the result to the console. I use the MySQL Connector Driver.
However, running the script only prints the initalial values. By that I mean, that if I change the values in the database while my script is running, it's not registered by the script and it's keeps on writing the initial values.
The code which retrieves the values in a while loop is as follows:
def get_fans():
global cnx
query = 'SELECT * FROM config'
while True:
cursor = cnx.cursor()
cursor.execute(query)
for (config_name, config_value) in cursor:
print config_name, config_value
print "\n"
cursor.close()
time.sleep(3)
Why is this happening?
Most likely, it's an autocommit issue. MySQL Connector Driver documentation states, that it has autocommit turned off. Make sure you commit your implicit transactions while changing your table. Also because default isolation is REPEATABLE READ you have:
All consistent reads within the same transaction read the snapshot
established by the first read.
So I guess you have to manage transaction even for your polling script. Or change isolation level to READ COMMITTED.
Though, the better way is to restore to MySQL client default autocommit-on mode. Even though PEP 249 guides to have it initially disabled, it's mere a proposal and most likely a serious design mistake. Not only it makes novices wonder about uncommited changes, makes even your read-only workload slower, it complicates data-layer design and breaks explicit is better than implicit Python zen. Even sluggish things like Django have it rethought.
I'm using Flask-SQLAlchemy 1.0, Flask 0.10, SQLAlchemy 0.8.2, and Python 2.7.5. I'm connecting to MySQL 5.6 with Oracle's MySQL Connector/Python 1.0.12.
When I restart my web server (either Apache2 or Flask's built-in), I receive the exception OperationalError: MySQL Connection not available after MySQL's wait_timeout expires (default 8 hours).
I've found people with similar problems and explicitly set SQLALCHEMY_POOL_RECYCLE = 7200, even though that's Flask-SQLAlchemy's default. When I put a breakpoint here, I see that the teardown function is successfully calling session.remove() after each request. Any ideas?
Update 7/21/2014:
Since this question continues to receive attention, I must add that I did try some of the proposals. Two of my attempts looked like the following:
First:
#contextmanager
def safe_commit():
try:
yield
db.session.commit()
except:
db.session.rollback()
raise
This allowed me to wrap my commit calls like so:
with safe_commit():
model = Model(prop=value)
db.session.add(model)
I am 99% certain that I did not miss any db.session.commit calls with this method and I still had problems.
Second:
def managed_session():
def decorator(f):
#wraps(f)
def decorated_function(*args, **kwargs):
try:
response = f(*args, **kwargs)
db.session.commit()
return response
except:
db.session.rollback()
raise
finally:
db.session.close()
return decorated_function
return decorator
To further ensure I wasn't missing any commit calls, I made a Flask wrapper that enabled code such as (if I remember correctly):
#managed_session()
def hello(self):
model = Model(prop=value)
db.session.add(model)
return render_template(...
Unfortunately, neither method worked. I also recall trying to issue SELECT(1) calls in an attempt to re-establish the connection, but I don't have that code anymore.
To me, the bottom line is MySQL/SQL Alchemy has issues. When I migrated to Postgres, I didn't have to worry about my commits. Everything just worked.
I was having this problem and it was driving me nuts. I tried playing with SQLALCHEMY_POOL_RECYCLE but this didn't seem to fix the problem.
I finally found http://docs.sqlalchemy.org/en/latest/orm/session.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it , and adapted for flask-sqlalchemy.
After I started using the following pattern, I haven't seen the problem. The key seems to be always assuring that a commit() or rollback() is executed. So if there is if-then-else which doesn't flow through commit() (e.g., for detected error), also do commit() or rollback() before redirect, abort, render_template call.
class DoSomething(MethodView):
def get(self):
try:
# do stuff
db.session.commit()
return flask.render_template('sometemplate.html')
except:
db.session.rollback()
raise
app.add_url_rule('/someurl',view_func=DoSomething.as_view('dosomething'),methods=['GET'])
UPDATE 7/22/2014
I discovered that I also had to change the SQLALCHEMY_POOL_RECYCLE to be less than the MySQL interactive_timeout. On the godaddy server, interactive_timeout was set to 60, so I set SQLALCHEMY_POOL_RECYCLE to 50. I think both the pattern I used, and this timeout were necessary to make the problem go away, but at this point I'm not positive. However, I'm pretty sure that when SQLALCHEMY_POOL_RECYCLE was greater than interactive_timeout, I was still getting the operational error.
I ran across the same issue recently - the first request to the MYSQL database after a long period of FLASK & SQLAlchemy application inactivity (at least 8 hours) results in an unhandled exception, which in turn implies 500 Internal Server Error: Connection Unavailable. All subsequent requests are just fine.
I managed to boil down the problem to MYSQL connection by decreasing the ##session.wait_timeout (and ##global just in case) value to 5 seconds. Then every odd request was just alright, while every second after 5-plus-second pause failed. The conclusion was obvious - SQLAlchemy was using open, but timeouted on the database end connection.
Solution
In my case it turned out the solution is spelled out in the SQLAlchemy – MYSQL has gone away blog post:
The first thing to make sure of is [...] the value of pool_recycle should be less than your MYSQLs wait_timeout value.
In MYSQL documentation you can find wait_timeout defaults to 8 hours (28 800 seconds), while SQLAlchemy engine's pool_recycle default value is -1, that entails no connection recycle whatsoever. I simply passed the value of 21 600 (6 hours) to the create_engine function and the error is gone.
sqlalchemy provides 2 ways of handling with disconnections, details in the documentation
Short version:
Optimistically
use try...except block to catch disconnection exceptions. This will return a 500 on the failing request, then the web application continues as normal. So use this one if disconnection happens infrequently. Note: you'll need to wrap each potential-to-fail operations in the try...except block.
Pessimistically (the one I'm using)
Basically do an extra ping operation (something like SELECT 1) each time a connection is checked out from the pool. If the ping fails raise DisconnectionError, upon which the host pool will attempt to force a new connection to be created (in fact the pool will try 3 times before officially give up). In this way your application won't see 500 error. The tradeoff is the extra SQL executed, although according to the doc the overhead is small.
I am writing a program on python which interacts with MySQL database.
For sql queries I use MySQLdb.
The problem is that fetchone() returns None but with the database browser I can see that that row exists.
This piece of code:
query = "SELECT * FROM revision WHERE rev_id=%s;"
cursor.execute(query % revision_id)
row = cursor.fetchone()
if row == None:
raise Exception("there isn't revision with id %s" % revision_id)
I have no idea what is going on here. Any ideas?
EDIT: okay, in some cases it works in some cases it doesn't but anyway when it
does not work the row exists in the table. I am passing a cursor object to a function and the code above is in the function. The problem is connected with this cursor object. Could the problem be that I pass the cursor as an argument to the function? How can I test it?
EDIT2: yes, the problem is that cursor does not work after I use it several times. Wether because other program connects to the DB or I am doing something wrong.
I have while loop in which I call a function to get info from the DB. After some iterations it does not work again. There is another program which writes to
the DB while while loop works.
Okay, db.autocommit(True) solved my problem.
This is related to transaction isolation level on your MySQL server. In the case of REPEATABLE_READ which is the default level for InnoDb, a snapshot is created at the time of first read, and subsequent read by the same cursor are made from this snapshot. Read more about isolation levels here
What we usually require while reusing the same cursor to run multiple queries, is READ_COMMITTED. Thankfully, if you can not change this on your SQL server, you can set your cursor to a particular isolation level.
cur = conn.cursor()
cur.execute("SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED")
This makes sure that every query you make, there is a fresh latest committed snapshot is used.
Best Practice is to commit db, after all query executed db.commit()