How can I perform post processing on my SQL3 database via python? The following code doesn't work, but what I am trying to do is first create a new database if not exists already, then insert some data, and finally execute the query and close the connection. But I what to do so separately, so as to add additional functionality later on, such as delete / updater / etc... Any ideas?
class TitlesDB:
# initiate global variables
conn = None
c = None
# perform pre - processing
def __init__(self, name):
import os
os.chdir('/../../')
import sqlite3
conn = sqlite3.connect(name)
c = conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS titles (title VARCHAR(100) UNIQUE)')
# insert a bunch of new titles
def InsertTitles(self, list):
c.executemany('INSERT OR IGNORE INTO titles VALUES (?)', list)
# perform post - processing
def __fina__(self):
conn.commit()
conn.close()
You could create a context manager to do the pre- and postprocessing.
import contextlib
#contextlib.contextmanager
def titles_cursor():
# perform pre - processing
conn = sqlite3.connect(name)
c = conn.cursor()
c.execute('CREATE TABLE IF NOT EXISTS titles (title VARCHAR(100) UNIQUE)')
yield c
# perform post - processing
conn.commit()
conn.close()
Use it in a with statement:
with titles_cursor() as c:
c.executemany('INSERT OR IGNORE INTO titles VALUES (?)', list)
First, wouldn't it be better to avoid having the sql connection inside the __init__?
You will have a problem if you want to use this class in the same instance after using __fina__.
You could have it in another method and call it and call the connection closing method when needed and commit after each method is executed.
Here is what I use : Create a class method that connects to the db and executes a query from an argument,commits and closes connection and you pass any query you want executed as an argument of that method.You can simply call this method anytime you want.
And the best about this is that you can create a method that passes multiple queries as arguments before closing db connection.
This is specially usefull if have to use sql connections to the same db in another class without using a set of methods each time you need to execute a sql query.
Here is a little example I used with MySQLdb module, It's pretty simple but it worked.
import MySQLdb
class DbQuery:
'''Here is the class I talked about'''
def __init__(self):
'''You can define the main queries here but it's not necessary
They can be global variables
If you don't have class dependency from which you get variables
you might not even need to define __init__'''
def Sql_Connect(self):
self.db = MySQLdb.connect("localhost","root","","data_db" )
self.cursor = db.cursor()
def Sql_Commit(self):
self.db.commit()
print "Info : Database updated"
except:
self.db.rollback()
print "Error : Database rollback"
self.db.close()
def Query(self,query):
self.Sql_Connect()
try :
self.cursor.execute(query)
self.Sql_Commit()
The only thing important is to remember the query structure.
Related
In the PyMySQL library, in cursors.py the following functions are called:
def __enter__(self):
return self
def __exit__(self, *exc_info):
del exc_info
self.close()
That's mean that if I use the cursor class in the with statement, the cursor should close whenever I go out from the nested block. Why instead it remain setted?
db = pymysql.connect(config)
with pymysql.cursors.Cursor(db) as cursor:
print(cursor)
print(cursor)
also:
db = pymysql.connect(config)
with db.cursor() as cursor:
print(cursor)
print(cursor)
both forms return the cursor object printing two times (one time inside the with statement and one time out from the with statement?. Am I doing something wrong?
Closing a cursor doesn't null out the cursor, just detaches it from the database. Try printing cursor.connection instead.
Also, I think you're expecting the "with" keyword to delete the object in question, but it's really just syntactic sugar around the enter and exit functions.
Instead of using:
import sqlite3
conn = sqlite3.connect(':memory:')
c = conn.cursor()
c.execute(...)
c.close()
would it be possible to use the Pythonic idiom:
with conn.cursor() as c:
c.execute(...)
It doesn't seem to work:
AttributeError: __exit__
Note: it's important to close a cursor because of this.
You can use contextlib.closing:
import sqlite3
from contextlib import closing
conn = sqlite3.connect(':memory:')
with closing(conn.cursor()) as cursor:
cursor.execute(...)
This works because closing(object) automatically calls the close() method of the passed in object after the with block.
A simpler alternative would be to use the connection object with the context manager, as specified in the docs.
with con:
con.execute(...)
If you insist on working with the cursor (because reasons), then why not make your own wrapper class?
class SafeCursor:
def __init__(self, connection):
self.con = connection
def __enter__(self):
self.cursor = self.con.cursor()
return self.cursor
def __exit__(self, typ, value, traceback):
self.cursor.close()
You'll then call your class like this:
with SafeCursor(conn) as c:
c.execute(...)
Adding to sudormrfbin's post. I've recently experienced an issue where an INSERT statement wasn't committing to the database. Turns out I was missing the with context manager for just the Connection object.
Also, it is a good practice to always close the Cursor object as well, as mentioned in this post.
Therefore, use two contextlib.closing() methods, each within a with context manager:
import contextlib
import sqlite3
# Auto-closes the Connection object
with contextlib.closing(sqlite3.connect("path_to_db_file")) as conn:
# Auto-commit to the database
with conn:
# Auto-close the Cursor object
with contextlib.closing(conn.cursor()) as cursor:
# Execute method(s)
cursor.execute(""" SQL statements here """)
How I can pass the data from object oriented programming to mysql in python? Do I need to make connection in every class?
Update:
This is my object orinted
class AttentionDataPoint(DataPoint):
def __init__ (self, _dataValueBytes):
DataPoint._init_(self, _dataValueBytes)
self.attentionValue=self._dataValueBytes[0]
def __str__(self):
if(self.attentionValue):
return "Attention Level: " + str(self.attentionValue)
class MeditationDataPoint(DataPoint):
def __init__ (self, _dataValueBytes):
DataPoint._init_(self, _dataValueBytes)
self.meditationValue=self._dataValueBytes[0]
def __str__(self):
if(self.meditationValue):
return "Meditation Level: " + str(self.meditationValue)
And I try to get the data to mysql using this coding.
import time
import smtplib
import datetime
import MySQLdb
db = MySQLdb.connect("192.168.0.101", "fyp", "123456", "system")
cur = db.cursor()
while True:
Meditation_Level = meditationValue()
Attention_Level = attentionValue()
current_time = datetime.datetime.now()
sql = "INSERT INTO table (id, Meditation_Level, Attention_Level, current_time) VALUES ('test1', %s, %s, %s)"
data = (Meditation_Level, Attention_Level, current_time)
cur.execute(sql, data)
db.commit()
db.close()
Update:
DataPoint
class DataPoint:
def __init__(self, dataValueBytes):
self._dataValueBytes = dataValueBytes
I am thinking of something like a class that has a function for connection and you just pass the instance of that class to every class where you need the connection (here the db instance), or may be not!
It's just an idea that I used in a other languages.
A few options:
Implement a save/load method for each DataPoint child to build the select/insert/update sql commands according with the needs of each class. Here is a basic tutorial: Python MySQL Database Access
Implement a class to build the sql commands above automatically from your class attributes.
The above is called Object Relational Mapping and, so, you don't need to reinvent the wheel: https://www.fullstackpython.com/object-relational-mappers-orms.html
I'd go with the first if it is a simple project with no chance to become big and with an ORM tool for all the other cases.
The most popular for python is SQLAlchemy:
http://docs.sqlalchemy.org/en/latest/orm/tutorial.html
I'm trying to figure out how to chain class methods to improve a utility class I've been writing - for reasons I'd prefer not to get into :)
Now suppose I wanted to chain a chain class methods on a class instance (in this case for setting the cursor) e.g.:
# initialize the class instance
db = CRUD(table='users', public_fields=['name', 'username', 'email'])
#the desired interface class_instance.cursor(<cursor>).method(...)
with sql.read_pool.cursor() as c:
db.cursor(c).get(target='username', where="omarlittle")
The part that's confusing is I would prefer the cursor not to persist as a class attribute after .get(...) has been called and has returned, I'd like to require that .cursor(cursor) must be first called.
class CRUD(object):
def __init__(self, table, public_fields):
self.table = table
self.public_fields = public_fields
def fields(self):
return ', '.join([f for f in self.public_fields])
def get(self, target, where):
#this is strictly for illustration purposes, I realize all
#the vulnerabilities this leaves me exposed to.
query = "SELECT {fields} FROM {table} WHERE {target} = {where}"
query.format(fields=self.fields, table=self.table, target=target,
where=where)
self.cursor.execute(query)
def cursor(self, cursor):
pass # this is where I get lost.
If I understand what you're asking, what you want is for the cursor method to return some object with a get method that works as desired. There's no reason the object it returns has to be self; it can instead return an instance of some cursor type.
That instance could have a back-reference to self, or it could get its own copy of whatever internals are needed to be a cursor, or it could be a wrapper around an underlying object from your low-level database library that knows how to be a cursor.
If you look at the DB API 2.0 spec, or implementations of it like the stdlib's sqlite3, that's exactly how they do it: A Database or Connection object (the thing you get from the top-level connect function) has a cursor method that returns a Cursor object, and that Cursor object has an execute method.
So:
class CRUDCursor(object):
def __init__(self, c, crud):
self.crud = crud
self.cursor = however_you_get_an_actual_sql_cursor(c)
def get(self, target, where):
#this is strictly for illustration purposes, I realize all
#the vulnerabilities this leaves me exposed to.
query = "SELECT {fields} FROM {table} WHERE {target} = {where}"
query.format(fields=self.crud.fields, table=self.crud.table,
target=target, where=where)
self.cursor.execute(query)
# you may want this to return something as well?
class CRUD(object):
def __init__(self, table, public_fields):
self.table = table
self.public_fields = public_fields
def fields(self):
return ', '.join([f for f in self.public_fields])
# no get method
def cursor(self, cursor):
return CRUDCursor(self, cursor)
However, there still seems to be a major problem with your example. Normally, after you execute a SELECT statement on a cursor, you want to fetch the rows from that cursor. You're not keeping the cursor object around in your "user" code, and you explicitly don't want the CRUD object to keep its cursor around, so… how do you expect to do that? Maybe get is supposed to return self.cursor.fetch_all() at the end or something?
Noobish question to be sure. I have tried to move some common code into a separate module and use the code as an imported function. The code makes a MySQL query through MySQLdb. When the function is part of the main script, it runs just fine. When I import the function from a separate module, the function fails because the cursor object is no longer defined. Is there a way to import functions without defining a separate cursor object for just the imported function?
Here is an coded example. This works:
import MySQLdb
#from mod2 import lookup_value
def get_db_connection(database_name):
db = MySQLdb.connect('localhost', 'user', 'pswrd', database_name)
cur = db.cursor()
return db, cur
def lookup_value(user_name):
query = "SELECT COUNT(*) FROM x_user_%s" % (user_name)
cur.execute("%s" % (query))
return cur.fetchone()
db_name = 'mg_test' # database name
user_name = 'test' # name of a specific table in the database
db, cur = get_db_connection(db_name)
value = lookup_value(user_name)
When I move the code for lookup_value to a second file, and import it ('from mod2 import lookup_value'), the code fails because the cursor object is undefined. The imported version of lookup_value only works if I create a cursor object for its use. This seems very inefficient. What is the best way to handle this problem?
That's because lookup_value searches for the cur within the file you import it in. You could put this all in a class.
class DB():
def __init__(self,database_name):
db = MySQLdb.connect('localhost', 'user', 'pswrd', database_name)
self.cur = db.cursor()
def lookup_value(self,user_name):
query = "SELECT COUNT(*) FROM x_user_%s"
self.cur.execute(query, (user_name,))
self.result = self.cur.fetchone()
return self.result
now you can do
....
db = DB(db_name)
value = db.lookup_value(user_name)
when you import the DB from mod2 import DB the last part should still work.
Also take note of how i execute the query in lookup_value this ensures the data is sanitized
You should pass a cursor variable to your functions if you want them to be independent.
In any case you should use in a function only local variables and variables passed as parameters.