We Keep Coding

How to initialize a database connection only once and reuse it in run-time in python?

I am currently working on a huge project, which constantly executes queries. My problem is, that my old code always created a new database connection and cursor, which decreased the speed immensivly. So I thought it's time to make a new database class, which looks like this at the moment:
class Database(object):
_instance = None
def __new__(cls):
if cls._instance is None:
cls._instance = object.__new__(cls)
try:
connection = Database._instance.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test")
cursor = Database._instance.cursor = connection.cursor()
except Exception as error:
print("Error: Connection not established {}".format(error))
else:
print("Connection established")
return cls._instance
def __init__(self):
self.connection = self._instance.connection
self.cursor = self._instance.cursor
# Do database stuff here
The queries will use the class like so:
def foo():
with Database() as cursor:
cursor.execute("STATEMENT")
I am not absolutly sure, if this creates the connection only once regardless of how often the class is created. Maybe someone knows how to initialize a connection only once and how to make use of it in the class afterwards or maybe knows if my solution is correct. I am thankful for any help!

Explanation
The keyword here is clearly class variables. Taking a look in the official documentation, we can see that class variables, other than instance variables, are shared by all class instances regardless of how many class instances exists.
Generally speaking, instance variables are for data unique to each instance and class variables are for attributes and methods shared by all instances of the class:
So let us asume you have multiple instances of the class. The class itself is defined like below.
class Dog:
kind = "canine" # class variable shared by all instances
def __init__(self, name):
self.name = name # instance variable unique to each instance
In order to better understand the differences between class variables and instance variables, I would like to include a small example here:
>>> d = Dog("Fido")
>>> e = Dog("Buddy")
>>> d.kind # shared by all dogs
"canine"
>>> e.kind # shared by all dogs
"canine"
>>> d.name # unique to d
"Fido"
>>> e.name # unique to e
"Buddy"
Solution
Now that we know that class variables are shared by all instances of the class, we can simply define the connection and cursor like shown below.
class Database(object):
connection = None
cursor = None
def __init__(self):
if Database.connection is None:
try:
Database.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test")
Database.cursor = Database.connection.cursor()
except Exception as error:
print("Error: Connection not established {}".format(error))
else:
print("Connection established")
self.connection = Database.connection
self.cursor = Database.cursor
As a result, the connection to the database is created once at the beginning and can then be used by every further instance.

Kind of like this. It's a cheap way of using a global.
class Database(object):
connection = None
def __init__(self):
if not Database.connection:
Database.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test")
def query(self,sql):
cursor = Database.connection.cursor()
cursor.execute(sql)
# Do database stuff here

This too does work and you are guaranteed to always have one instance of the database
def singleton(class_):
instances = {}
def get_instance(*args, **kwargs):
if class_ not in instances:
instances[class_] = class_(*args, **kwargs)
return instances[class_]
return get_instance
#singleton
class SingletonDatabase:
def __init__(self) -> None:
print('Initializing singleton database connection... ', random.randint(1, 100))

The Reason you have to do all this is if you just create
a connection once and leave it at that you then
will end up trying to use a connection which is dropped
so you create a connection and attach it to your app
then whenever you get a new request check if the connection
still exists, with before request hook if not then recreate the
connection and proceeed.
on create_app
def create_app(self):
if not app.config.get('connection_created'):
app.database_connection = Database()
app.config['connection_created'] = True
on run app
#app.before_request
def check_database_connection(self):
if not app.config.get('connection_created') or not app.database_connection:
app.database_connection = Database()
app.config['connection_created'] = True
this will insure that your application always runs with an active connection
and that it gets created only once per app
if connection is dropped on any subsequent call then it gets recreated again...

How do I add a method to a class in a compiled module?

The file scheduling.py currently contains nothing. What does it need to contain, so the last statement, cursor.chronology(), prints Hello World? I do want this method to be available to anything that might be defined by means of sqlite.connect().
import sqlite3
import scheduling
db = sqlite3.connect(':memory:')
cursor = db.cursor()
cursor.chronology()

Both the sqlite3.connect() and sqlite3.onnection.cursor() methods allow you to specify factory argument to replace the normal connection or cursor class with your own subclass. You can use these paths to provide your own cursor.chronology() method.
So you'd subclass the sqlite3.Cursor class to add your custom method:
import sqlite3
class ChronologyCursor(sqlite3.Cursor):
def chronology(self):
print("Hello World")
# ...
You can then use that class as the factory argument to the cursor() call:
>>> db = sqlite3.connect(':memory:')
>>> cursor = db.cursor(factory=ChronologyCursor)
>>> type(cursor)
<class '__main__.ChronologyCursor'>
>>> cursor.chronology()
Hello World
You can also use a connection factory (subclassing sqlite3.connection()) to always use your cursor class:
class ChronologyConnection(sqlite3.Connection):
def cursor(self, *args, **kwargs):
if kwargs.get('factory') is None:
kwargs['factory'] = ChronologyCursor
return super().cursor(*args, **kwargs)
then use db = sqlite3.connect(':memory:', factory=ChronologyConnection) to use the new connection class:
>>> db = sqlite3.connect(':memory:', factory=ChronologyConnection)
>>> type(db)
<class '__main__.ChronologyConnection'>
>>> cursor = db.cursor()
>>> cursor.chronology()
Hello World
I strongly recommend against patching the sqlite3.connect() function to make the above factory the default, but if you must have this be transparent, you can put your own connect() function on the sqlite3 module. The base package is pure Python so you can monkey-patch an alternative connect() function into it.
So in your scheduling module, you'd place the above classes, then use:
_sqlite3_connect = sqlite3.connect
def chronology_connect(*args, **kwargs):
if kwargs.get('factory') is None:
kwargs['factory'] = ChronologyConnection
return _sqlite3_connect(*args, **kwargs)
sqlite3.connect = chronology_connect
Now importing scheduling is enough to make calls to sqlite3.connect() use your alternate connection factory:
>>> import sqlite3
>>> import scheduling
>>> db = sqlite3.connect(':memory:')
>>> cursor = db.cursor()
>>> cursor.chronology()
Hello World
>>> type(db)
<class 'scheduling.ChronologyConnection'>
>>> type(cursor)
<class 'scheduling.ChronologyCursor'>
The reason you would not want to do the above is that it makes sqlite3.connect() calls deviate from the norm, act in ways that clash with what the documenation states. That makes your code harder to maintain in the long run. I'd stick with explicit sqlite3.connect(...., factory=ChronologyConnection) calls instead.

How can I inherit from psycopg2?

I'm trying inherit psycopg2 like this:
import psycopg2
class myp(psycopg):
pass
ii = myp
ii.connect(database = "myDataBase", user = "myUser", password="myPassword")
Then it throws an error:
class myp(psycopg2._psycopg):
TypeError: Error when calling the metaclass bases
module.__init__() takes at most 2 arguments (3 given)
Is it possible to inherit from psycopg2 library?
EDIT:
I want to use different databases, so I just have to change the class MyDatabase. something like a wrapper.
example:
import psycopg2
class MyDatabase(psycopg2):
def connect(self):
#do some stuff
return psycopg2.connect(database = "myDataBase", user = "myUser", password="myPassword")
for mysqldb
import MySQLdb
class MyDatabase(MySQLdb)
def connect(self):
#do some stuff
return psycopg2.connect(database = "myDataBase", user = "myUser", password="myPassword")
and derived class
class MyDataBaseApp(MyDatabase):
def add(self, myObjectClass):
db = MyDatabase()
cn = None
try:
cn = db.connect()
cur = cn.cursor()
cur.execute ("INSERT ...",(myObjectClass.parameter1, myObjectClass.parameter2))
cn.commit()
except MyDatabase.DatabaseError, e:
print e
if cn:
cn.rollback()
finally:
if cn:
cn.close()
but according to the documentation I have to do it another way, suggestions?

Disclaimer: I'm not familiar with psycopg
Update
Seems the documentation recommends to subclass psycopg2.extensions.connection. Then, connect() is a factory function that can still be used to create new connections, but you have to provide your class as a factory, again according to the docs
Full code may have to look more like (untested):
import psycopg2
class myp(psycopg2.extensions.connection):
pass
ii = connect(connection_factory=myp,
database = "myDataBase", user = "myUser", password="myPassword")
Update 2
With the updated approach, you're trying to build new classes with different/divergent interfaces. Often, composition is better than inheritance, see wikipedia and this question.

Class instance in Python

I have problem with using class instance in Python.
Ive created a new class ora which inherit connect class from cx_Oracle package.
When I try tu run this code I recive information
File "pyt.py", line 12, in myquery
ora.myConnect.cursor()
AttributeError: 'NoneType' object has no attribute 'cursor'
So Python cannote recognize that in ora.myConnect is stored reference to instance.
I dont know what can be reason of this error and what its wrong with code.
from cx_Oracle import connect
class ora(connect):
myConnect = None
def __init__(self,connstr):
ora.myConnect = connect.__init__(self,connstr)
def myquery(self):
ora.myConnect.cursor()
ora.myConnect.cursor.execute("SELECT * FROM table")
ora.myConnect.cursor.close()
connstr = 'user/passwd#host:port/sid'
connection = ora(connstr)
connection.myquery()
connection.close()
EDIT
Ive tried to replace ora to self but still Python dont have access to instance
from cx_Oracle import connect
class ora(connect):
myConnect = None
def __init__(self,connstr):
self.myConnect = connect.__init__(self,connstr)
def myquery(self):
self.myConnect.cursor()
self.myConnect.cursor.execute("SELECT * FROM table")
self.myConnect.cursor.close()
Error:
self.myConnect.cursor()
AttributeError: 'NoneType' object has no attribute 'cursor'
EDIT2
This code works without OOP, for me self.myConnect sholud reference to object instance and this object should contain method cursor()
import cx_oracle
connstr = 'user/passwd#host:port/sid'
connection = cx_oracle.connect(connstr)
cursor = connection.cursor()
cursor.execute("SELECT * FROM table")
cursor.close()
connection.close()

It seems like you want self:
class ora(connect):
myConnect = None
def __init__(self, connstr):
self.myConnect = connect.__init__(self, connstr)
# ...
ora is the name of the class, not the instance.
Update Try the following:
from cx_Oracle import connect
class ora:
myConnect = None
def __init__(self, connstr):
self.myConnect = connect(connstr)
def myquery(self):
self.myConnect.cursor()
self.myConnect.cursor.execute("SELECT * FROM table")
self.myConnect.cursor.close()

Why do you want self.myConnect to refer to the connect instance? That's a complete misunderstanding of OOP. The ora instance is the connect instance. self.cursor is where you find the cursor.
Here's how your code should look:
class ora(connect):
def __init__(self,connstr):
super(ora, self).__init__(connstr)
def myquery(self):
self.cursor.execute("SELECT * FROM table")
self.cursor.close()
In any case, __init__ must never return anything, so setting self.myConnect to the return value will always result in it being bound to None.

How do I use a MySQLdb connect object on imported function

Noobish question to be sure. I have tried to move some common code into a separate module and use the code as an imported function. The code makes a MySQL query through MySQLdb. When the function is part of the main script, it runs just fine. When I import the function from a separate module, the function fails because the cursor object is no longer defined. Is there a way to import functions without defining a separate cursor object for just the imported function?
Here is an coded example. This works:
import MySQLdb
#from mod2 import lookup_value
def get_db_connection(database_name):
db = MySQLdb.connect('localhost', 'user', 'pswrd', database_name)
cur = db.cursor()
return db, cur
def lookup_value(user_name):
query = "SELECT COUNT(*) FROM x_user_%s" % (user_name)
cur.execute("%s" % (query))
return cur.fetchone()
db_name = 'mg_test' # database name
user_name = 'test' # name of a specific table in the database
db, cur = get_db_connection(db_name)
value = lookup_value(user_name)
When I move the code for lookup_value to a second file, and import it ('from mod2 import lookup_value'), the code fails because the cursor object is undefined. The imported version of lookup_value only works if I create a cursor object for its use. This seems very inefficient. What is the best way to handle this problem?

That's because lookup_value searches for the cur within the file you import it in. You could put this all in a class.
class DB():
def __init__(self,database_name):
db = MySQLdb.connect('localhost', 'user', 'pswrd', database_name)
self.cur = db.cursor()
def lookup_value(self,user_name):
query = "SELECT COUNT(*) FROM x_user_%s"
self.cur.execute(query, (user_name,))
self.result = self.cur.fetchone()
return self.result
now you can do
....
db = DB(db_name)
value = db.lookup_value(user_name)
when you import the DB from mod2 import DB the last part should still work.
Also take note of how i execute the query in lookup_value this ensures the data is sanitized

You should pass a cursor variable to your functions if you want them to be independent.
In any case you should use in a function only local variables and variables passed as parameters.