We Keep Coding

How to initialize a database connection only once and reuse it in run-time in python?

I am currently working on a huge project, which constantly executes queries. My problem is, that my old code always created a new database connection and cursor, which decreased the speed immensivly. So I thought it's time to make a new database class, which looks like this at the moment:
class Database(object):
_instance = None
def __new__(cls):
if cls._instance is None:
cls._instance = object.__new__(cls)
try:
connection = Database._instance.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test")
cursor = Database._instance.cursor = connection.cursor()
except Exception as error:
print("Error: Connection not established {}".format(error))
else:
print("Connection established")
return cls._instance
def __init__(self):
self.connection = self._instance.connection
self.cursor = self._instance.cursor
# Do database stuff here
The queries will use the class like so:
def foo():
with Database() as cursor:
cursor.execute("STATEMENT")
I am not absolutly sure, if this creates the connection only once regardless of how often the class is created. Maybe someone knows how to initialize a connection only once and how to make use of it in the class afterwards or maybe knows if my solution is correct. I am thankful for any help!

Explanation
The keyword here is clearly class variables. Taking a look in the official documentation, we can see that class variables, other than instance variables, are shared by all class instances regardless of how many class instances exists.
Generally speaking, instance variables are for data unique to each instance and class variables are for attributes and methods shared by all instances of the class:
So let us asume you have multiple instances of the class. The class itself is defined like below.
class Dog:
kind = "canine" # class variable shared by all instances
def __init__(self, name):
self.name = name # instance variable unique to each instance
In order to better understand the differences between class variables and instance variables, I would like to include a small example here:
>>> d = Dog("Fido")
>>> e = Dog("Buddy")
>>> d.kind # shared by all dogs
"canine"
>>> e.kind # shared by all dogs
"canine"
>>> d.name # unique to d
"Fido"
>>> e.name # unique to e
"Buddy"
Solution
Now that we know that class variables are shared by all instances of the class, we can simply define the connection and cursor like shown below.
class Database(object):
connection = None
cursor = None
def __init__(self):
if Database.connection is None:
try:
Database.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test")
Database.cursor = Database.connection.cursor()
except Exception as error:
print("Error: Connection not established {}".format(error))
else:
print("Connection established")
self.connection = Database.connection
self.cursor = Database.cursor
As a result, the connection to the database is created once at the beginning and can then be used by every further instance.

Kind of like this. It's a cheap way of using a global.
class Database(object):
connection = None
def __init__(self):
if not Database.connection:
Database.connection = mysql.connector.connect(host="127.0.0.1", user="root", password="", database="db_test")
def query(self,sql):
cursor = Database.connection.cursor()
cursor.execute(sql)
# Do database stuff here

This too does work and you are guaranteed to always have one instance of the database
def singleton(class_):
instances = {}
def get_instance(*args, **kwargs):
if class_ not in instances:
instances[class_] = class_(*args, **kwargs)
return instances[class_]
return get_instance
#singleton
class SingletonDatabase:
def __init__(self) -> None:
print('Initializing singleton database connection... ', random.randint(1, 100))

The Reason you have to do all this is if you just create
a connection once and leave it at that you then
will end up trying to use a connection which is dropped
so you create a connection and attach it to your app
then whenever you get a new request check if the connection
still exists, with before request hook if not then recreate the
connection and proceeed.
on create_app
def create_app(self):
if not app.config.get('connection_created'):
app.database_connection = Database()
app.config['connection_created'] = True
on run app
#app.before_request
def check_database_connection(self):
if not app.config.get('connection_created') or not app.database_connection:
app.database_connection = Database()
app.config['connection_created'] = True
this will insure that your application always runs with an active connection
and that it gets created only once per app
if connection is dropped on any subsequent call then it gets recreated again...

How to extend OracleCursor class from cx_Oracle

Using Python 2.7.12 and package cx_Oracle I'm trying to create an extended class of the what the package call OracleCursor. I simply want to inherit the methods from the superclass and extend with some custom methods.
First I get the OracleCursor by
import cx_Oracle
conn = cx_Oracle.connect(username, password, dsn)
cursor = conn.cursor()
and I then have the following
>>> type(cursor)Out[6]:
OracleCursor
>>> isinstance(cursor, cx_Oracle.Cursor)
True
One would think that it is achieved by
class ExtendedCursor(cx_Oracle.Cursor):
def hello_world(self):
print('Hello world')
extended = ExtendedCursor(cursor)
but I get TypeError: argument 1 must be cx_Oracle.Connection, not OracleCursor. To me that error doesn't make sense. Also, I can't use OracleCursor as my superclass since it isn't recognized as a class.

The cursor is returned from the Connection object. You need to create a custom connection that returns your ExtendedCursor.
import cx_Oracle as cxo
class MyCursor(cxo.Cursor):
def helloWorld(self):
print "helloWorld"
class MyConnection(cxo.Connection):
def cursor(self):
return MyCursor(self)
if __name__ == '__main__':
conStr = '<user>/<password>#127.0.0.1:1521/xe'
db = MyConnection(conStr)
c = db.cursor()
print c
c.execute('select 1+1 from dual')
print(c.fetchall())
c.helloWorld()
returns:
<__main__.MyCursor on <__main__.MyConnection to ...#127.0.0.1:1521/xe>>
[(2,)]
helloWorld

how to use python cx_Oracle with spool

I'm using python3.4 to interact with oracle(11g)/sql developer.
Is it true that cx_Oracle could not deal with sqlPlus statements? It seems that the page https://sourceforge.net/p/cx-oracle/mailman/message/2932119/ said so.
So how could we execute 'spool' command by python?
The code:
import cx_Oracle
db_conn = cx_Oracle.connect(...)
cursor = db_conn.cursor()
cursor.execute('spool C:\\Users\Administrator\Desktop\mycsv.csv')
...
the error: cx_Oracle.DatabaseError: ORA-00900:

The "spool" command is very specific to SQL*Plus and is not available in cx_Oracle or any other application that uses the OCI (Oracle Call Interface). You can do something similar, however, without too much trouble.
You can create your own Connection class subclassed from cx_Oracle.Connection and your own Cursor class subclassed from cx_Oracle.Cursor that would perform any logging and have a special command "spool" that would turn it on and off at will. Something like this:
class Connection(cx_Oracle.Connection):
def __init__(self, *args, **kwargs):
self.spoolFile = None
return super(Connection, self).__init__(*args, **kwargs)
def cursor(self):
return Cursor(self)
def spool(self, fileName):
self.spoolFile = open(fileName, "w")
class Cursor(cx_Oracle.Cursor):
def execute(self, statement, args):
result = super(Cursor, self).execute(statement, args)
if self.connection.spoolFile is not None:
self.connection.spoolFile.write("Headers for query\n")
self.connection.spoolFile.write("use cursor.description")
def fetchall(self):
rows = super(Cursor, self).fetchall()
if self.connection.spoolFile is not None:
for row in rows:
self.connection.spoolFile.write("row details")
That should give you some idea on where to go with this.

Mock entire python class

I'm trying to make a simple test in python, but I'm not able to figure it out how to accomplish the mocking process.
This is the class and def code:
class FileRemoveOp(...)
#apply_defaults
def __init__(
self,
source_conn_keys,
source_conn_id='conn_default',
*args, **kwargs):
super(v4FileRemoveOperator, self).__init__(*args, **kwargs)
self.source_conn_keys = source_conn_keys
self.source_conn_id = source_conn_id
def execute (self, context)
source_conn = Connection(conn_id)
try:
for source_conn_key in self.source_keys:
if not source_conn.check_for_key(source_conn_key):
logging.info("The source key does not exist")
source_conn.remove_file(source_conn_key,'')
finally:
logging.info("Remove operation successful.")
And this is my test for the execute function:
#mock.patch('main.Connection')
def test_remove_execute(self,MockConn):
mock_coon = MockConn.return_value
mock_coon.value = #I'm not sure what to put here#
remove_operator = FileRemoveOp(...)
remove_operator.execute(self)
Since the execute method try to make a connection, I need to mock that, I don't want to make a real connection, just return something mock. How can I make that? I'm used to do testing in Java but I never did on python..

First it is very important to understand that you always need to Mock where it the thing you are trying to mock out is used as stated in the unittest.mock documentation.
The basic principle is that you patch where an object is looked up,
which is not necessarily the same place as where it is defined.
Next what you would need to do is to return a MagicMock instance as return_value of the patched object. So to summarize this you would need to use the following sequence.
Patch Object
prepare MagicMock to be used
return the MagicMock we've just created as return_value
Here a quick example of a project.
connection.py (Class we would like to Mock)
class Connection(object):
def execute(self):
return "Connection to server made"
file.py (Where the Class is used)
from project.connection import Connection
class FileRemoveOp(object):
def __init__(self, foo):
self.foo = foo
def execute(self):
conn = Connection()
result = conn.execute()
return result
tests/test_file.py
import unittest
from unittest.mock import patch, MagicMock
from project.file import FileRemoveOp
class TestFileRemoveOp(unittest.TestCase):
def setUp(self):
self.fileremoveop = FileRemoveOp('foobar')
#patch('project.file.Connection')
def test_execute(self, connection_mock):
# Create a new MagickMock instance which will be the
# `return_value` of our patched object
connection_instance = MagicMock()
connection_instance.execute.return_value = "testing"
# Return the above created `connection_instance`
connection_mock.return_value = connection_instance
result = self.fileremoveop.execute()
expected = "testing"
self.assertEqual(result, expected)
def test_not_mocked(self):
# No mocking involved will execute the `Connection.execute` method
result = self.fileremoveop.execute()
expected = "Connection to server made"
self.assertEqual(result, expected)

I found that this simple solution works in python3: you can substitute a whole class before it is being imported for the first time. Say I have to mock class 'Manager' from real.manager
class MockManager:
...
import real.manager
real.manager.Manager = MockManager
It is possible to do this substitution in init.py if there is no better place.
It may work in python2 too but I did not check.

How can I inherit from psycopg2?

I'm trying inherit psycopg2 like this:
import psycopg2
class myp(psycopg):
pass
ii = myp
ii.connect(database = "myDataBase", user = "myUser", password="myPassword")
Then it throws an error:
class myp(psycopg2._psycopg):
TypeError: Error when calling the metaclass bases
module.__init__() takes at most 2 arguments (3 given)
Is it possible to inherit from psycopg2 library?
EDIT:
I want to use different databases, so I just have to change the class MyDatabase. something like a wrapper.
example:
import psycopg2
class MyDatabase(psycopg2):
def connect(self):
#do some stuff
return psycopg2.connect(database = "myDataBase", user = "myUser", password="myPassword")
for mysqldb
import MySQLdb
class MyDatabase(MySQLdb)
def connect(self):
#do some stuff
return psycopg2.connect(database = "myDataBase", user = "myUser", password="myPassword")
and derived class
class MyDataBaseApp(MyDatabase):
def add(self, myObjectClass):
db = MyDatabase()
cn = None
try:
cn = db.connect()
cur = cn.cursor()
cur.execute ("INSERT ...",(myObjectClass.parameter1, myObjectClass.parameter2))
cn.commit()
except MyDatabase.DatabaseError, e:
print e
if cn:
cn.rollback()
finally:
if cn:
cn.close()
but according to the documentation I have to do it another way, suggestions?

Disclaimer: I'm not familiar with psycopg
Update
Seems the documentation recommends to subclass psycopg2.extensions.connection. Then, connect() is a factory function that can still be used to create new connections, but you have to provide your class as a factory, again according to the docs
Full code may have to look more like (untested):
import psycopg2
class myp(psycopg2.extensions.connection):
pass
ii = connect(connection_factory=myp,
database = "myDataBase", user = "myUser", password="myPassword")
Update 2
With the updated approach, you're trying to build new classes with different/divergent interfaces. Often, composition is better than inheritance, see wikipedia and this question.