Being new to both Python and sqlite, I've been playing around with them both recently trying to figure things out. In particular with sqlite I've learned how to open/close/commit data to a db. But now I'm trying to clean things up a bit so that I can open/close the db via function calls. For instance, I'd like to do something like:
def open_db():
conn = sqlite3.connect("path)
c = conn.cursor()
def close_db():
c.close()
conn.close()
def create_db():
open_db()
c.execute("CREATE STUFF")
close_db()
Then when I run the program, before I query or write to the table, I could do something like:
open_db()
c.execute('SELECT * DO STUFF')
OR
c.execute('DELETE * DO OTHER STUFF')
conn.commit
close_db()
I've read about context managers but I'm not sure I understand entirely whats going on with them. What would be the easiest solution to cleaning up the way I open/close my DB connections so I'm not always having to type in the cursor command.
This is because the connection you define is local to the open db function. Change it as follows
def open_db():
conn = sqlite3.connect("path)
return conn.cursor()
and then
c = open_db()
c.execute('SELECT * DO STUFF')
It should be noted that writing function like this purely as a learning exercise might be ok, but generally it's not very useful to write a thin wrapper around a database connectivity api.
I don't know that there is an easy way. As already suggested, if you make the name of a database cursor or connection local to a function then these will be lost upon exit from that function. The answer might be to write code using the contextlib module (which is included with the Python distribution, and documented in the help file); I wouldn't call that easy. The documentation for sqlite3 does mention that connection objects can be used as context managers; I suspect you've already noticed that. I also see that there's some sort of context manager for MySQL but I haven't used it.
Related
I am struggling to unittest a function that doesn't return anything and executes delete operation. The function is as follows:
def removeReportParseData(self, report_id, conn=None):
table_id = dbFind(
"select table_id from table_table where report_id=%s", (report_id), conn
)
for t in table_id:
self.removeTableParseData(int(t["table_id"]), conn)
dbUpdate("delete from table_table where table_id=%s", t["table_id"], conn)
I want to make sure that the commands were executed but don't want to affect the actual db. My current code is:
def test_remove_report_parse_data(self):
with patch("com.pdfgather.GlobalHelper.dbFind") as mocked_find:
mocked_find.return_value = [123, 232, 431]
mocked_find.assert_called_once()
Thank you in advance for any support.
I don't believe it is possible to execute SQL without executing it, so to speak, however if you are on MySQL you might be able to get fairly close to what you want by wrapping your queries in START TRANSACTION; and ROLLBACK;
i.e. you might replace your queries with:
START TRANSACTION;
YOUR QUERY HERE;
ROLLBACK
This will prove that your function works without actually changing the contents of the database.
However, if it is sufficient to simply test that these functions would execute ANY query, you could alternatively opt to test them with empty queries, and simply assert that your dbFind and dbUpdate methods were called as many times as you would expect.
Again, though, as I alluded to in my comment, I would strongly suggest NOT having your test suite interact with even your development database.
While there is certainly some configuration involved in setting up another database for your tests, you should be able to find some boilerplate code to do this quite easily as it is very common practice.
I have a program, which must work with database (and it can work in few threads). I wrote a special class, which starts from:
class DataBase:
def __init__(self):
self.connection = sqlite3.connect("some_db.db", check_same_thread=False)
self.cursor = self.connection.cursor()
Is it okay to use such class? Or I should go another way and use few cursors? My program works fine now and I didn't notice any problems, but I'm not sure can it lead to big problems in future
I have sqlite3 working in classes and I don't need it to be self-ed. Ive got it like...
connection = sqlite3.connect("some_db.db")
c = connection.cursor()
Yeah I think its fine, until you need multiple databases where all you need to do is name you connection and cursors something different like...
connection2 = sqlite3.connect("another_db.db")
c2 = connection2.cursor()
You can have them running at the same time too. Just remember to commit and close when you don't need them anymore.
I am constructing a model that does large parts of its calculations in a Postgresql database (for performance reasons). It looks somewhat like this:
def sql_func1(conn):
# prepare some data, crunch some number, etc.
curs = conn.cursor()
curs.execute("SOME SQL COMMAND")
curs.commit()
curs.close()
if __name__ == "__main__":
connection = psycopg2.connect(dbname='name', user='user', password='pass', host='localhost', port=1234)
sql_func1(conn)
sql_func2(conn)
sql_func3(conn)
connection.close()
The script uses around 30 individual functions like sql_func1. Obviously it is a little awkward to manage the connection and cursor in each function all the time. Thus I started using a decorator as described here. Now I can simply wrap sql_func1 with a decorator #db_connect and pass the connection from there. However, that means I am opening and closing the connection all the time, which is not good practice either. The psycopg2 FAQ says:
Creating a connection can be slow (think of SSL over TCP) so the best
practice is to create a single connection and keep it open as long as
required. It is also good practice to rollback or commit frequently
(even after a single SELECT statement) to make sure the backend is
never left “idle in transaction”. See also psycopg2.pool for
lightweight connection pooling.
Could you please give me some insights which would be an ideal practice im my case. Should I rather use a decorator that passes the cursor object instead of the connection? If so, please provide a code sample for the decorator. As I am rather new to programming, please let me also know in case you think my overall approach is wrong.
What about storing the connection in a global variable without closing it in the finally block? Something like this (according to the example you linked):
cnn = None
def with_connection(f):
def with_connection_(*args, **kwargs):
global cnn
if not cnn:
cnn = psycopg.connect(DSN)
try:
rv = f(cnn, *args, **kwargs)
except Exception, e:
cnn.rollback()
raise
else:
cnn.commit() # or maybe not
return rv
return with_connection_
I want to write a Python module to abstract away database transactions for my application. My question is whether I need to call connect() and close() for every transaction? In code:
import sqlite3
# Can I put connect() here?
conn = sqlite3.connect('db.py')
def insert(args):
# Or should I put it here?
conn = sqlite3.connect('db.py')
# Perform the transaction.
c = conn.cursor()
c.execute(''' insert args ''')
conn.commit()
# Do I close the connection here?
conn.close()
# Or can I close the connection whenever the application restarts (ideally, very rarely)
conn.close()
I have don't much experience with databases, so I'd appreciate an explanation for why one method is preferred over the other.
You can use the same connection repeatedly. You can also use the connection (and the cursor) as a context manager so that you don't need to explicitly call close on either.
def insert(conn, args):
with conn.cursor() as c:
c.execute(...)
conn.commit()
with connect('db.py') as conn:
insert(conn, ...)
insert(conn, ...)
insert(conn, ...)
There's no reason to close the connection to the database, and re-opening the connection each time can be expensive. (For example, you may need to establish a TCP session to connect to a remote database.)
Using a single connection will be faster, and operationally should be fine.
Use the atexit module if you want to ensure the closing eventually happens (even if your program is terminated by an exception). Specifically, import atexit at the start of your program, and atexit.register(conn.close) right after you connect -- note, no () after close, you want to register the function to be called at program exist (whether normal or via an exception), not to call the function.
Unfortunately if Python should crash due e.g to an error in a C-coded module that Python can't catch, or a kill -9, etc, the registered exit function(s) may end up not being called. Fortunately in this case it shouldn't hurt anyway (besides being, one hopes, a rare and extreme occurrence).
I was doing a tutorial and came across a way to handle connections with sqlite3,
Then I studied about the WITH keyword and found out that it is an alternative to try,except,finally way of doing things
It was said that in case of file-handling, 'WITH' automatically handles closing of files and I thought similar with the connection as said in zetcode tutorial:-
"With the with keyword, the Python interpreter automatically releases
the resources. It also provides error handling." http://zetcode.com/db/sqlitepythontutorial/
so I thought it would be good to use this way of handling things, but I couldn't figure out why both (inner scope and outer scope) statements work? shouldn't the WITH release the connection?
import sqlite3
con = sqlite3.connect('test.db')
with con:
cur = con.cursor()
cur.execute('SELECT 1,SQLITE_VERSION()')
data = cur.fetchone()
print data
cur.execute('SELECT 2,SQLITE_VERSION()')
data = cur.fetchone()
print data
which outputs
(1, u'3.6.21')
(2, u'3.6.21')
I don't know what exactly the WITH is doing here(or does in general), so, if you will please elaborate on the use of WITH over TRY CATCH in this context.
And should the connections be opened and closed on each query? (I am formulating queries inside a function which I call each time with an argument) Would it be a good practice?
In general, a context manager is free to do whatever its author wants it to do when used. Set/reset a certain system state, cleaning up resources after use, acquiring/releasing a lock, etc.
In particular, as Jon already writes, a database connection object creates a transaction when used as a context manager. If you want to auto-close the connection, you can do
with contextlib.closing(sqlite3.connect('test.db')) as con:
with con as cur:
cur.execute('SELECT 1,SQLITE_VERSION()')
data = cur.fetchone()
print data
with con as cur:
cur.execute('SELECT 2,SQLITE_VERSION()')
data = cur.fetchone()
print data
You could also write your own wrapper around sqlite3 to support with:
class SQLite():
def __init__(self, file='sqlite.db'):
self.file=file
def __enter__(self):
self.conn = sqlite3.connect(self.file)
self.conn.row_factory = sqlite3.Row
return self.conn.cursor()
def __exit__(self, type, value, traceback):
self.conn.commit()
self.conn.close()
with SQLite('test.db') as cur:
print(cur.execute('select sqlite_version();').fetchall()[0][0])
https://docs.python.org/2.5/whatsnew/pep-343.html#SECTION000910000000000000000
From the docs: http://docs.python.org/2/library/sqlite3.html#using-the-connection-as-a-context-manager
Connection objects can be used as context managers that automatically commit or rollback transactions. In the event of an exception, the transaction is rolled back; otherwise, the transaction is committed:
So, the context manager doesn't release the connection, instead, it ensures that any transactions occurring on the connection are rolled back if any exception occurs, or committed otherwise... Useful for DELETE, UPDATE and INSERT queries for instance.