I am used to (spoiled by?) python's SQLite interface to deal with SQL databases. One nice feature in python's SQLite's API the "context manager," i.e., python's with statement. I usually execute queries in the following way:
import as sqlite
with sqlite.connect(db_filename) as conn:
query = "INSERT OR IGNORE INTO shapes VALUES (?,?);"
results = conn.execute(query, ("ID1","triangle"))
With the code above, if my query modifies the database and I forget to run conn.commit(),the context manager runs it for me automatically upon exiting the with statement. It also handles exceptions nicely: if an exception occurs before I commit anything, then the database is rolled back.
I am now using the MySQLdb interface, which doesn't seem to support a similar context manager out of the box. How do I create my own? There is a related question here, but it doesn't offer a complete solution.
Previously, MySQLdb connections were context managers.
As of this commit on 2018-12-04, however, MySQLdb connections are no longer context managers,
and users must explicitly call conn.commit() or conn.rollback(), or write their own context manager, such as the one below.
You could use something like this:
import config
import MySQLdb
import MySQLdb.cursors as mc
import _mysql_exceptions
import contextlib
DictCursor = mc.DictCursor
SSCursor = mc.SSCursor
SSDictCursor = mc.SSDictCursor
Cursor = mc.Cursor
#contextlib.contextmanager
def connection(cursorclass=Cursor,
host=config.HOST, user=config.USER,
passwd=config.PASS, dbname=config.MYDB,
driver=MySQLdb):
connection = driver.connect(
host=host, user=user, passwd=passwd, db=dbname,
cursorclass=cursorclass)
try:
yield connection
except Exception:
connection.rollback()
raise
else:
connection.commit()
finally:
connection.close()
#contextlib.contextmanager
def cursor(cursorclass=Cursor, host=config.HOST, user=config.USER,
passwd=config.PASS, dbname=config.MYDB):
with connection(cursorclass, host, user, passwd, dbname) as conn:
cursor = conn.cursor()
try:
yield cursor
finally:
cursor.close()
with cursor(SSDictCursor) as cur:
print(cur)
connection = cur.connection
print(connection)
sql = 'select * from table'
cur.execute(sql)
for row in cur:
print(row)
To use it you would place config.py in your PYTHONPATH and define the HOST, USER, PASS, MYDB variables there.
Think things have changed since this question was originally asked. Somewhat confusingly (from my point of view at least), for recent versions of MySQLdb, if you use a connection in a context you get a cursor (as per the oursql example), not something that closes automatically (as you would if you opened a file for instance).
Here's what I do:
from contextlib import closing
with closing(getConnection()) as conn: #ensure that the connection is closed
with conn as cursor: #cursor will now auto-commit
cursor.execute('SELECT * FROM tablename')
Related
Kindly look at the code below. I'm using opentelemetry for tracing. Psycopg2Instrumentor for PostgreSQL tracing. Here only the "show server_version" SQL statement is getting traced. But the SQL statement in execute method is not traced. I think it's because of using NamedTupleCursor cursor_factory. If I remove NamedTupleCursor, it's tracing the main SQL statements. Could you please help me to trace the main SQL statement without removing NamedTupleCursor?
def self.get_connection():
#conn = create_connection()
with conn.cursor() as curs:
curs.execute("show server_version") ---> this sql statement is getting tracked
return conn
def execute()
with self.get_connection() as conn:
with conn.cursor(cursor_factory=NamedTupleCursor) as curs:
curs.execute("Sql statements"). ---> this sql statement is **not** getting tracked```
Below is the code snippet for working with Psycopg2Instrumentor for PostgreSQL tracing. The instrumentation code to be updated on passing cursor_factory in cursor parameter, rather than setting it in connection. For now, the below works for me and tracing got captured.
import psycopg2
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor
Psycopg2Instrumentor().instrument()
#add your cursor factory in connection method
cnx = psycopg2.connect(
host=host, database=DBname, user=user, password=password, cursor_factory=RealDictCursor)
#remove the cursor factory from cursor method
cursor = cnx.cursor()
cursor.execute("SELECT statement")
cursor.close()
cnx.close()
Thanks to the thread (Psycopg2Instrumentor doesn't work for cursors with non-default cursor_factory) and #RaguramGopi
The below is my way of handling database connection, but it is a bit clumsy than I want...So the question is whether or not there are some other even more proper ways to close the database while returning an error message to client if DB operations return some errors.
#app.route('/get-data/', methods=['GET'])
def get_data():
db_error = False
try:
conn = pymysql.connect(db_url, db_username, db_password, db_name)
cursor = conn.cursor()
cursor.execute('SELECT a_column FROM a_table WHERE a_condition = 0')
results = cursor.fetchall()
except Exception as ex:
logging.error('{ex}')
db_error = True # Cannot simply return here; otherwise DB connection is left open
finally:
cursor.close()
conn.close()
if db_error:
return Response('Database error', 500)
return jsonify(results) # Let's assume the jsonify() function will not throw an error...
Suppose I use a context manager, does it mean that both conn and cursor will definitely be closed even when an exception is thrown? Or is it something implementation-dependent, i.e., some packages, say, pymysql, will make sure all cursors and conns are closed, regardless of errors are thrown or not; while other packages, say, pyodbc, will NOT ensure this. (Here pymysql and pyodbc are just two examples of course...)
This is probably a stupid question but I cannot find the information in the documentation for pymysql. What is the pymysql default cursorclass? When I do not specify a cursor class on connection to the database my queries return a list for each row in the response.
When I specify pymysql.cursors.DictCursor I get a dictionary response. I would like to be able to change between them for different connections within a script.
I've written a little function with a context manager to yield the cursor but it requires me to specify the name of the cursorclass each time. I know I can get around this, but knowing the name of the default cursorclass would be nice.
from contextlib import contextmanager
import pymysql
#contextmanager
def openDb(host=DB_HOST, database=DB_DATABASE,
user=DB_USER, cursor=DB_CURSOR):
"""
Simple context manager for opening a db connection
"""
with pymysql.connect(host=host, database=database, user=user,
cursorclass=cursor) as cur:
yield cur
I could probably write this as:
#contextmanager
def openDb(host=DB_HOST, database=DB_DATABASE,
user=DB_USER, cursor=None):
"""
Simple context manager for opening a db connection
"""
if cursor:
with pymysql.connect(host=host, database=database, user=user,
cursorclass=cursor) as cur:
yield cur
else:
with pymysql.connect(host=host, database=database, user=user) as cur:
yield cur
and let it default to whatever the default cursorclass is, but I would prefer to be explicit.
Of course as soon as I post this I find the answer in via:
>>> import pymysql
>>> help(pymysql.cursors)
Help on module pymysql.cursors in pymysql:
NAME
pymysql.cursors - # -*- coding: utf-8 -*-
CLASSES
builtins.object
Cursor
SSCursor
DictCursorMixin
DictCursor(DictCursorMixin, Cursor)
SSDictCursor(DictCursorMixin, SSCursor)
pymysql.cursors.Cursor is the answer. Documentation...
PyMySQL, a python package to access MySQL database, seems not support SELECT ... FOR UPDATE.
In the code below, I used SELECT...FOR UPDATE to read some_table in function f(), used UPDATE to modify the table in g(), and spawned 50 threads for each function.
I expected deadlock to happen since SELECT...FOR UPDATE should block the threads spawned by g. But actually no deadlock happened. Can some one explain why?
from threading import Thread
import pymysql
def f():
db = pymysql.connect("localhost", "tester","pwd", "testDB")
cur = db.cursor()
sql = "SELECT * FROM some_table FOR UPDATE"
try:
cur.execute(sql)
except:
print("Exception in select")
def g():
db = pymysql.connect("localhost", "tester", "pwd","testDB")
cur = db.cursor()
sql = "UPDATE some_table SET val=20 WHERE id=2"
try:
cur.execute(sql)
db.commit()
except:
print("Exception in update")
db.rollback()
for _ in range(50):
Thread(target=f, args=()).start()
Thread(target=g, args=()).start()
I am using Python 3.4 and PyMySQL 0.6.6. Thank you in advance.
PyMySQL does support SELECT ... FOR UPDATE
But you need start the transaction using connection.begin()
Here's an example:
connection= pymysql.connect("localhost", "tester", "pwd","testDB")
connection.begin()
cursor = db.cursor()
cursor.execute("SELECT * FROM some_table FOR UPDATE")
Table/row (depending upon your select query) is now in locked state. Only the current connection can make changes.
To release the lock. You can,
commit the changes (if any)
connection.commit()
close the connection
connection.close()
Code which I used with PostgeSQL 9.4 and Python 2.7.9
import psycopg2
import psycopg2.extras
def init_pg94_from_sql_file(filename, connection):
...
cursor = connection.cursor()
cursor.execute(...)
connection.commit()
cursor.close()
def store_values_to_pg94(fileHeader, file_size, eventStarts, eventEnds, connection):
cursor = connection.cursor()
cursor.execute(...);
connection.commit()
cursor.close()
def(main):
...
conn_string = "dbname=detector_development user=postgres"
conn = psycopg2.connect(conn_string)
init_pg94_from_sql_file("12.7.2015.sql", conn)
store_values_to_pg94(fileHeader, file_size, eventStarts, eventEnds, conn)
conn.commit()
conn.close()
where you see that
cursor is initiated, executed and commited in each function
in the main, everything once again committed, and connection closed
I think there can be much duplicates here, because writing to disk by commit in every function sounds to be inefficient.
Also closing things three times sounds too inefficient.
How can you handle the cursor initialization and commits better?
Cursors are lightweight. Closing a cursor is releasing unused memory. In a database connection every transaction should be commited. Commit every task. The efficiency is part of the database's job. If non-commited data is written to disk or when commited data is written, is not your concern.