Below is a database pooling example. I don't understand the following.
Why the getcursor function use "yield"?
What is the context manager?
from psycopg2.pool import SimpleConnectionPool
from contextlib import contextmanager
dbConnection = "dbname='dbname' user='postgres' host='localhost' password='postgres'"
# pool define with 10 live connections
connectionpool = SimpleConnectionPool(1,10,dsn=dbConnection)
#contextmanager
def getcursor():
con = connectionpool.getconn()
try:
yield con.cursor()
finally:
connectionpool.putconn(con)
def main_work():
try:
# with here will take care of put connection when its done
with getcursor() as cur:
cur.execute("select * from \"TableName\"")
result_set = cur.fetchall()
except Exception as e:
print "error in executing with exception: ", e**
Both of your questions are related. In python context managers are what we're using whenever you see a with statement. Classically, they're written like this.
class getcursor(object):
def __enter__(self):
con = connectionpool.getconn()
return con
def __exit__(self, *args):
connectionpool.putconn(con)
Now when you use a context manager, it calls the __enter__ method on the with statement and the __exit__ method when it exits the context. Think of it like this.
cursor = getcursor()
with cursor as cur: # calls cursor.__enter__()
cur.execute("select * from \"TableName\"")
result_set = cur.fetchall()
# We're now exiting the context so it calls `cursor.__exit__()`
# with some exception info if relevant
x = 1
The #contextmanager decorator is some sugar to make creating a context manager easier. Basically, it uses the yield statement to give the execution back to the caller. Everything up to and including the yield statement is the __enter__ method and everything after that is effectively the __exit__ statement.
Related
In the PyMySQL library, in cursors.py the following functions are called:
def __enter__(self):
return self
def __exit__(self, *exc_info):
del exc_info
self.close()
That's mean that if I use the cursor class in the with statement, the cursor should close whenever I go out from the nested block. Why instead it remain setted?
db = pymysql.connect(config)
with pymysql.cursors.Cursor(db) as cursor:
print(cursor)
print(cursor)
also:
db = pymysql.connect(config)
with db.cursor() as cursor:
print(cursor)
print(cursor)
both forms return the cursor object printing two times (one time inside the with statement and one time out from the with statement?. Am I doing something wrong?
Closing a cursor doesn't null out the cursor, just detaches it from the database. Try printing cursor.connection instead.
Also, I think you're expecting the "with" keyword to delete the object in question, but it's really just syntactic sugar around the enter and exit functions.
Instead of using:
import sqlite3
conn = sqlite3.connect(':memory:')
c = conn.cursor()
c.execute(...)
c.close()
would it be possible to use the Pythonic idiom:
with conn.cursor() as c:
c.execute(...)
It doesn't seem to work:
AttributeError: __exit__
Note: it's important to close a cursor because of this.
You can use contextlib.closing:
import sqlite3
from contextlib import closing
conn = sqlite3.connect(':memory:')
with closing(conn.cursor()) as cursor:
cursor.execute(...)
This works because closing(object) automatically calls the close() method of the passed in object after the with block.
A simpler alternative would be to use the connection object with the context manager, as specified in the docs.
with con:
con.execute(...)
If you insist on working with the cursor (because reasons), then why not make your own wrapper class?
class SafeCursor:
def __init__(self, connection):
self.con = connection
def __enter__(self):
self.cursor = self.con.cursor()
return self.cursor
def __exit__(self, typ, value, traceback):
self.cursor.close()
You'll then call your class like this:
with SafeCursor(conn) as c:
c.execute(...)
Adding to sudormrfbin's post. I've recently experienced an issue where an INSERT statement wasn't committing to the database. Turns out I was missing the with context manager for just the Connection object.
Also, it is a good practice to always close the Cursor object as well, as mentioned in this post.
Therefore, use two contextlib.closing() methods, each within a with context manager:
import contextlib
import sqlite3
# Auto-closes the Connection object
with contextlib.closing(sqlite3.connect("path_to_db_file")) as conn:
# Auto-commit to the database
with conn:
# Auto-close the Cursor object
with contextlib.closing(conn.cursor()) as cursor:
# Execute method(s)
cursor.execute(""" SQL statements here """)
I have a script that processes some data and if a database/file is present, writes some data into it. I specify the database or file as configargparse(argparse) argument. I need to clean (close file, db) in some organized way in case exceptions occur.
Here is my init:
import sqlite3
import confargparse
import sys
parser.ArgParser(...)
parser.add('--database', dest='database',
help='path to database with grabbers', metavar='FILE',
type=lambda x: arghelper.is_valid_file(parser, x))
parser.add('-f', '--file', type=configargparse.FileType(mode='r'))
args = parser.parse_args()
I did it using if and try:
if args.database:
conn = sqlite3.connect(args.database)
c = conn.cursor()
# same init for file
try:
while True: # do something, it might be moved to some main() function
result = foo()
if args.database:
c.execute('Write to database {}'.format(result))
# same
# for file
finally:
if args.database:
conn.close()
# same
# for file
except KeyboardInterrupt:
print 'keyboard interrupt'
Can it be done with the with statement? Something like (here comes ()?():() from C):
with ((args.database)?
(conn = sqlite3.connect(args.database)):
(None)) as db, same for file:
and then refer to the db inside the with clause and check if they exist?
To answer your question first. It can be done, using contextlib. But I'm not sure how much you would gain from this.
from contextlib import contextmanager
#contextmanager
def uncertain_conn(args):
yield sqlite3.connect(args.database) if args.database else None
# Then you use it like this
with uncertain_conn(args) as conn:
# conn will be the value yielded by uncertain_conn(args)
if conn is not None:
try:
# ...
But as I said, while turning a generator function into a context manager is cool and personally I really like the contextmanager decorator, and it does give you the functionality you say you want, I don't know if it really helps you that much here. If I were you I'd probably just be happy with if:
if args.database:
conn = sqlite3.connect(args.database)
try:
# ...
There are a couple of things you can simplify with with, though. Check out closing, also from contextlib (really simple, I'll just quote the documentation):
contextlib.closing(thing)
Return a context manager that closes thing
upon completion of the block. This is basically equivalent to:
from contextlib import contextmanager
#contextmanager def closing(thing):
try:
yield thing
finally:
thing.close()
So the above code can become:
if args.database:
conn = sqlite3.connect(args.database)
with closing(conn):
# do something; conn.close() will be called no matter what
But this won't print a nice message for KeyboardInterrupt. If you really need that, then I guess you still to have to write out the try-except-finally yourself. Doing anything more fanciful is probably not worth it. (And note that except must precede finally, otherwise you get a syntax error.)
And you can even do this with suppress (but requires a bit of caution; see below)
from contextlib import suppress
with suppress(TypeError):
conn = sqlite3.connect(args.database or None)
with closing(conn):
# do business
with suppress(error): do_thing is equivalent to
try:
do_thing
except error:
pass
So if args.database evaluates to False, the second line is effectively connect(None), which raises a TypeError, which will be caught by the context manager and the code below will be skipped. But the risk is that it will suppress all TypeErrors in its scope, and you may not want that.
You can create your own context manager in such cases. Create one which handles both connections. A context manager is a class which has methods __enter__() and __exit__(). One is called before entering the with clause, one is called when it is left (how ever).
Here's an example for how to do this in your case:
def f(cond1, cond2):
class MultiConnectionContextManager(object):
def __init__(self, cond1, cond2):
self.cond1 = cond1
self.cond2 = cond2
def __enter__(self):
print "entering ..."
if self.cond1:
# self.connection1 = open(...)
print "opening connection1"
if self.cond2:
# self.connection1 = open(...)
print "opening connection2"
return self
def __exit__(self, exc_type, exc_value, traceback):
print "exiting ..."
if self.cond1:
# self.connection1.close()
print "closing connection1"
if self.cond2:
# self.connection2.close()
print "closing connection2"
with MultiConnectionContextManager(cond1, cond2) as handle:
if cond1:
# handle.connection1.read()
print "using handle.connection1"
if cond2:
# handle.connection2.read()
print "using handle.connection2"
for cond1 in (False, True):
for cond2 in (False, True):
print "=====", cond1, cond2
f(cond1, cond2)
You can call this directly to see the outcome. Replace the prints with your real statements for opening, using, and closing the connections.
cx_Oracle contains __enter__ and __exit__ on Connection objects, but not on Cursor objects. Thus, I use this everywhere to wrap cursors :
class CursorWrapper(object):
def __init__(self, connection):
self.connection = connection
self.cursor = None
def __enter__(self):
self.cursor = self.connection.cursor()
return self.cursor
def __exit__(self, exc_type, exc_value, traceback):
self.cursor.close()
then, when I want a cursor
with CursorWrapper(cnx) as cursor:
cursor.execute("whatever sql statement")
It suits my needs fairly well.
However, I was then wondering what could prevent __enter__ and __exit__ methods to be added directly in cx_Oracle ?
Or is there a better way to use cursors with context managements, which would explain why it is not defined in the module ?
Solution moved from #MathieuC.'s question post.
I can just use contextlib.closing.
import contextlib
with contextlib.closing(cnx.cursor()) as cursor:
I'm using MySQLdb to connect to MySQL using python. My tables are all InnoDB and I'm using transactions.
I'm struggling to come up with a way to 'share' transactions across functions. Consider the following pseudocode:
def foo():
db = connect()
cur = db.cursor()
try:
cur.execute(...)
conn.commit()
except:
conn.rollback()
def bar():
db = connect()
cur = db.cursor()
try:
cur.execute(...)
foo() # note this call
conn.commit()
except:
conn.rollback()
At some points in my code, I need to call foo() and at some points I need to call bar(). What's the best practice here? How would I tell the call to foo() to commit() if called outside bar() but not inside bar()? This is obviously more complex if there are multiple threads calling foo() and bar() and the calls to connect() don't return the same connection object.
UPDATE
I found a solution which works for me. I've wrapped connect() to increment a value when called. Calling commit() decrements that value. If commit() is called and that counter's > 0, no commit happens and the value is decremented. You therefore get this:
def foo():
db = connect() # internal counter = 1
...
db.commit() # internal counter = 0, so commit
def bar():
db = connect() # internal counter = 1
...
foo() # internal counter goes to 2, then to 1 when commit() is called, so no commit happens
db.commit() # internal counter = 0, so commit
You can take advantage of Python's default function arguments in this case:
def foo(cur=None):
inside_larger_transaction = False
if cursor is None:
db = connect()
cur = db.cursor()
inside_larger_transaction = True
try:
cur.execute(...)
if not inside_larger_transaction:
conn.commit()
except:
conn.rollback()
So, if bar is calling foo, it just pass in the cursor object as a parameter.
Not that I don't see much sense in creating a different cursor object for each small function - you should either write your several functions as methods of an object, and have a cursor attribute - or pass the cursos explicitly always (in this case, use another named parameter to indicate whether the current function is part of a major transaction or not)
Another option is to create a context-manager class to make your commits, and encapsulate all transactions within it - therefore, none of your functions should do transaction commit - you would keep both transaction.commit and transaction.rollback calls on the __exit__method of this object.
class Transaction(object):
def __enter__(self):
self.db = connect()
cursor = self.db.cursor()
return cursor
def __exit__(self, exc_type, exc_value, traceback):
if exc_type is None:
self.db.commit()
else:
self.db.rollback()
And just use it like this:
def foo(cursor):
...
def foo(cur):
cur.execute(...)
def bar(cur):
cur.execute(...)
foo(cur)
with Transaction() as cursor:
foo(cursor)
with Transaction() as cursor:
bar(cursor)
The cleanest way IMO is to pass the connection object to foo and bar
Declare you connections outside the functions and pass them to the function as arguements
foo(cur, conn)
bar(cur, conn)