We Keep Coding

How to initialize a database connection only once and reuse it in run-time in python?

I am currently working on a huge project, which constantly executes queries. My problem is, that my old code always created a new database connection and cursor, which decreased the speed immensivly. So I thought it's time to make a new database class, which looks like this at the moment:
class Database(object):
_instance = None
def __new__(cls):
if cls._instance is None:
cls._instance = object.__new__(cls)
try:
connection = Database._instance.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test")
cursor = Database._instance.cursor = connection.cursor()
except Exception as error:
print("Error: Connection not established {}".format(error))
else:
print("Connection established")
return cls._instance
def __init__(self):
self.connection = self._instance.connection
self.cursor = self._instance.cursor
# Do database stuff here
The queries will use the class like so:
def foo():
with Database() as cursor:
cursor.execute("STATEMENT")
I am not absolutly sure, if this creates the connection only once regardless of how often the class is created. Maybe someone knows how to initialize a connection only once and how to make use of it in the class afterwards or maybe knows if my solution is correct. I am thankful for any help!

Explanation
The keyword here is clearly class variables. Taking a look in the official documentation, we can see that class variables, other than instance variables, are shared by all class instances regardless of how many class instances exists.
Generally speaking, instance variables are for data unique to each instance and class variables are for attributes and methods shared by all instances of the class:
So let us asume you have multiple instances of the class. The class itself is defined like below.
class Dog:
kind = "canine" # class variable shared by all instances
def __init__(self, name):
self.name = name # instance variable unique to each instance
In order to better understand the differences between class variables and instance variables, I would like to include a small example here:
>>> d = Dog("Fido")
>>> e = Dog("Buddy")
>>> d.kind # shared by all dogs
"canine"
>>> e.kind # shared by all dogs
"canine"
>>> d.name # unique to d
"Fido"
>>> e.name # unique to e
"Buddy"
Solution
Now that we know that class variables are shared by all instances of the class, we can simply define the connection and cursor like shown below.
class Database(object):
connection = None
cursor = None
def __init__(self):
if Database.connection is None:
try:
Database.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test")
Database.cursor = Database.connection.cursor()
except Exception as error:
print("Error: Connection not established {}".format(error))
else:
print("Connection established")
self.connection = Database.connection
self.cursor = Database.cursor
As a result, the connection to the database is created once at the beginning and can then be used by every further instance.

Kind of like this. It's a cheap way of using a global.
class Database(object):
connection = None
def __init__(self):
if not Database.connection:
Database.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test")
def query(self,sql):
cursor = Database.connection.cursor()
cursor.execute(sql)
# Do database stuff here

This too does work and you are guaranteed to always have one instance of the database
def singleton(class_):
instances = {}
def get_instance(*args, **kwargs):
if class_ not in instances:
instances[class_] = class_(*args, **kwargs)
return instances[class_]
return get_instance
#singleton
class SingletonDatabase:
def __init__(self) -> None:
print('Initializing singleton database connection... ', random.randint(1, 100))

The Reason you have to do all this is if you just create
a connection once and leave it at that you then
will end up trying to use a connection which is dropped
so you create a connection and attach it to your app
then whenever you get a new request check if the connection
still exists, with before request hook if not then recreate the
connection and proceeed.
on create_app
def create_app(self):
if not app.config.get('connection_created'):
app.database_connection = Database()
app.config['connection_created'] = True
on run app
#app.before_request
def check_database_connection(self):
if not app.config.get('connection_created') or not app.database_connection:
app.database_connection = Database()
app.config['connection_created'] = True
this will insure that your application always runs with an active connection
and that it gets created only once per app
if connection is dropped on any subsequent call then it gets recreated again...

Is there a "with conn.cursor() as..." way to work with Sqlite?

Instead of using:
import sqlite3
conn = sqlite3.connect(':memory:')
c = conn.cursor()
c.execute(...)
c.close()
would it be possible to use the Pythonic idiom:
with conn.cursor() as c:
c.execute(...)
It doesn't seem to work:
AttributeError: __exit__
Note: it's important to close a cursor because of this.

You can use contextlib.closing:
import sqlite3
from contextlib import closing
conn = sqlite3.connect(':memory:')
with closing(conn.cursor()) as cursor:
cursor.execute(...)
This works because closing(object) automatically calls the close() method of the passed in object after the with block.

A simpler alternative would be to use the connection object with the context manager, as specified in the docs.
with con:
con.execute(...)
If you insist on working with the cursor (because reasons), then why not make your own wrapper class?
class SafeCursor:
def __init__(self, connection):
self.con = connection
def __enter__(self):
self.cursor = self.con.cursor()
return self.cursor
def __exit__(self, typ, value, traceback):
self.cursor.close()
You'll then call your class like this:
with SafeCursor(conn) as c:
c.execute(...)

Adding to sudormrfbin's post. I've recently experienced an issue where an INSERT statement wasn't committing to the database. Turns out I was missing the with context manager for just the Connection object.
Also, it is a good practice to always close the Cursor object as well, as mentioned in this post.
Therefore, use two contextlib.closing() methods, each within a with context manager:
import contextlib
import sqlite3
# Auto-closes the Connection object
with contextlib.closing(sqlite3.connect("path_to_db_file")) as conn:
# Auto-commit to the database
with conn:
# Auto-close the Cursor object
with contextlib.closing(conn.cursor()) as cursor:
# Execute method(s)
cursor.execute(""" SQL statements here """)

Python Database Pooling with psycopg2

Below is a database pooling example. I don't understand the following.
Why the getcursor function use "yield"?
What is the context manager?
from psycopg2.pool import SimpleConnectionPool
from contextlib import contextmanager
dbConnection = "dbname='dbname' user='postgres' host='localhost' password='postgres'"
# pool define with 10 live connections
connectionpool = SimpleConnectionPool(1,10,dsn=dbConnection)
#contextmanager
def getcursor():
con = connectionpool.getconn()
try:
yield con.cursor()
finally:
connectionpool.putconn(con)
def main_work():
try:
# with here will take care of put connection when its done
with getcursor() as cur:
cur.execute("select * from \"TableName\"")
result_set = cur.fetchall()
except Exception as e:
print "error in executing with exception: ", e**

Both of your questions are related. In python context managers are what we're using whenever you see a with statement. Classically, they're written like this.
class getcursor(object):
def __enter__(self):
con = connectionpool.getconn()
return con
def __exit__(self, *args):
connectionpool.putconn(con)
Now when you use a context manager, it calls the __enter__ method on the with statement and the __exit__ method when it exits the context. Think of it like this.
cursor = getcursor()
with cursor as cur: # calls cursor.__enter__()
cur.execute("select * from \"TableName\"")
result_set = cur.fetchall()
# We're now exiting the context so it calls `cursor.__exit__()`
# with some exception info if relevant
x = 1
The #contextmanager decorator is some sugar to make creating a context manager easier. Basically, it uses the yield statement to give the execution back to the caller. Everything up to and including the yield statement is the __enter__ method and everything after that is effectively the __exit__ statement.

how to use python cx_Oracle with spool

I'm using python3.4 to interact with oracle(11g)/sql developer.
Is it true that cx_Oracle could not deal with sqlPlus statements? It seems that the page https://sourceforge.net/p/cx-oracle/mailman/message/2932119/ said so.
So how could we execute 'spool' command by python?
The code:
import cx_Oracle
db_conn = cx_Oracle.connect(...)
cursor = db_conn.cursor()
cursor.execute('spool C:\\Users\Administrator\Desktop\mycsv.csv')
...
the error: cx_Oracle.DatabaseError: ORA-00900:

The "spool" command is very specific to SQL*Plus and is not available in cx_Oracle or any other application that uses the OCI (Oracle Call Interface). You can do something similar, however, without too much trouble.
You can create your own Connection class subclassed from cx_Oracle.Connection and your own Cursor class subclassed from cx_Oracle.Cursor that would perform any logging and have a special command "spool" that would turn it on and off at will. Something like this:
class Connection(cx_Oracle.Connection):
def __init__(self, *args, **kwargs):
self.spoolFile = None
return super(Connection, self).__init__(*args, **kwargs)
def cursor(self):
return Cursor(self)
def spool(self, fileName):
self.spoolFile = open(fileName, "w")
class Cursor(cx_Oracle.Cursor):
def execute(self, statement, args):
result = super(Cursor, self).execute(statement, args)
if self.connection.spoolFile is not None:
self.connection.spoolFile.write("Headers for query\n")
self.connection.spoolFile.write("use cursor.description")
def fetchall(self):
rows = super(Cursor, self).fetchall()
if self.connection.spoolFile is not None:
for row in rows:
self.connection.spoolFile.write("row details")
That should give you some idea on where to go with this.

Using with statement on cursor in cx_Oracle

cx_Oracle contains __enter__ and __exit__ on Connection objects, but not on Cursor objects. Thus, I use this everywhere to wrap cursors :
class CursorWrapper(object):
def __init__(self, connection):
self.connection = connection
self.cursor = None
def __enter__(self):
self.cursor = self.connection.cursor()
return self.cursor
def __exit__(self, exc_type, exc_value, traceback):
self.cursor.close()
then, when I want a cursor
with CursorWrapper(cnx) as cursor:
cursor.execute("whatever sql statement")
It suits my needs fairly well.
However, I was then wondering what could prevent __enter__ and __exit__ methods to be added directly in cx_Oracle ?
Or is there a better way to use cursors with context managements, which would explain why it is not defined in the module ?

Solution moved from #MathieuC.'s question post.
I can just use contextlib.closing.
import contextlib
with contextlib.closing(cnx.cursor()) as cursor: