We Keep Coding

When does sqlite3 roll back transactions in python, and when doesn't it?

I am creating a little workshop to teach how to use python and SQL and came across this oddity. I wanted to show how to use the with statement to create a transaction with sqlite:
import sqlite3
filename = 'data/transaction.db'
print("_________________________")
print("Create Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
sqls = [
'DROP TABLE IF EXISTS test',
'CREATE TABLE test (i integer)',
'INSERT INTO "test" VALUES(99)',
'SELECT * FROM test']
for sql in sqls:
cursor.execute(sql)
print(cursor.fetchall())
print("_________________________")
print("Create Error with 'with'")
try:
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
sqls = [
'update test set i = 1',
'SELECT * FROM test',
'fnord', # <-- trigger error
'update test set i = 0',]
for sql in sqls:
cursor.execute(sql)
print(cursor.fetchall())
except sqlite3.OperationalError as err:
print(err)
# near "fnord": syntax error
print("_________________________")
print("Show Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM test')
for row in cursor:
print(row)
# (99,)
This works exactly as expected. However to prove that without the with block the executions would be done halfway I tried the following:
print("_________________________")
print("Create Error without 'with'")
conn = sqlite3.connect(filename)
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
cursor.execute( 'UPDATE test SET i = 1 WHERE i = 99')
print(cursor.fetchall())
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
cursor.execute( 'update test set i = 0')
print(cursor.fetchall())
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
conn.close()
print("_________________________")
print("Show Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM test')
for row in cursor:
print(row)
# (99,)`
The whole output is:
_________________________
Create Table
[]
[]
[]
[(99,)]
_________________________
Create Error with 'with'
[]
[(1,)]
near "fnord": syntax error
_________________________
Show Table
(99,)
_________________________
Create Error without 'with'
[(99,)]
[]
[(1,)]
[]
[(0,)]
_________________________
Show Table
(99,) # Why is this not (0,)???
I am very confused as to why the last Block shows a 99 again. Eventually the plan is to add a try,except block with an exception, such that the SQL code mimics the first block - however I am confused without this already :).
Thanks for clarifying

From the python sqlite3 API doc:
The underlying sqlite3 library operates in autocommit mode by default,
but the Python sqlite3 module by default does not.
autocommit mode means that statements that modify the database take
effect immediately. A BEGIN or SAVEPOINT statement disables autocommit
mode, and a COMMIT, a ROLLBACK, or a RELEASE that ends the outermost
transaction, turns autocommit mode back on.
The Python sqlite3 module by default issues a BEGIN statement
implicitly before a Data Modification Language (DML) statement (i.e.
INSERT/UPDATE/DELETE/REPLACE).
Python will rollback transactions if the connection is closed without a commit (or an explicit ROLLBACK is issued). No transactions are committed in this program.
FYI a new connection is created in the "Create error without 'with'" block, but no new cursor is instantiated.

The Python with statement works with context managers. Whereas some context managers will release resources and possibly close an object, it seems that at least with the sqlite3.connection object, it merely commits or rolls back transactions but does not close the connection. This can be confirmed for the DB-API 2.0 interface:
Connection objects can be used as context managers that automatically commit or rollback transactions. In the event of an exception, the transaction is rolled back; otherwise, the transaction is committed:
...
# Successful, con.commit() is called automatically afterwards
...
# con.rollback() is called after the with block finishes with an exception, the
# exception is still raised and must be caught
...
# Connection object used as context manager only commits or rollbacks transactions,
# so the connection object should be closed manually
For the non-with statement block, you are never committing the transactions. When the connection is closed, all changes are automatically rolled back.
You need to call
conn.commit();
See Why the need to commit explicitly when doing an UPDATE? for more details.
As a side note, the section of your code titled "Create Error without 'with'" does not actually cause an error/exception.

Python psycopg2 Execute Hangs

I am using Python 3.x within an app hosted on Heroku with Basic PostgreSQL instance and Im using the psycopg2 library (as "lite")
Recently started hanging when I call Execute on the cursor object.
I am not sure how to troubleshoot this and would appreciate any thoughts.
Here is how I instantiate Connection:
def getConnection():
urlparse.uses_netloc.append("postgres")
url = urlparse.urlparse(os.environ["HEROKU_STUFF_HERE"])
con = lite.connect(
database=url.path[1:],
user=url.username,
password=url.password,
host=url.hostname,
port=url.port
)
return con
Here is how I generally execute non-return type statements:
def ExecuteSQL(sql):
con = DB.getConnection()
with con:
cur = con.cursor()
print("In")
cur.execute(sql)
print("out")
The app NEVER sees the light after printing the word "In" to the console.
I've left try except blocks out intentionally so as to blow the thing up.....no dice.
pretty much doesnt matter what the SQL Statement is, I get the same result
Running the same sql in a sql client tool executes instantly.
Im also not sure how to detect if this statement even makes it into Postgresql.....
Thanks for any help you can offer

psycopg2 opened the transaction which is not closed until a commit() or rollback(). Therefore your changes were not persistent (docs).
def ExecuteSQL(sql):
con = DB.getConnection()
with con:
cur = con.cursor()
cur.execute(sql)
con.commit()
or you could just do the following:
def ExecuteSQL(sql):
con = DB.getConnection()
with con:
con.autocommit = True
cur = con.cursor()
cur.execute(sql)

To init, execute and commit better in PostgreSQL with Python?

Code which I used with PostgeSQL 9.4 and Python 2.7.9
import psycopg2
import psycopg2.extras
def init_pg94_from_sql_file(filename, connection):
...
cursor = connection.cursor()
cursor.execute(...)
connection.commit()
cursor.close()
def store_values_to_pg94(fileHeader, file_size, eventStarts, eventEnds, connection):
cursor = connection.cursor()
cursor.execute(...);
connection.commit()
cursor.close()
def(main):
...
conn_string = "dbname=detector_development user=postgres"
conn = psycopg2.connect(conn_string)
init_pg94_from_sql_file("12.7.2015.sql", conn)
store_values_to_pg94(fileHeader, file_size, eventStarts, eventEnds, conn)
conn.commit()
conn.close()
where you see that
cursor is initiated, executed and commited in each function
in the main, everything once again committed, and connection closed
I think there can be much duplicates here, because writing to disk by commit in every function sounds to be inefficient.
Also closing things three times sounds too inefficient.
How can you handle the cursor initialization and commits better?

Cursors are lightweight. Closing a cursor is releasing unused memory. In a database connection every transaction should be commited. Commit every task. The efficiency is part of the database's job. If non-commited data is written to disk or when commited data is written, is not your concern.

Python mysqldb db.commit() not working

I have some code like this
db = MySQLdb.connect(host="localhost", user="root", passwd="root", db="domains")
cursor = db.cursor()
def check_urls(res):
pool = Pool(25)
for row in res:
pool.spawn(fetch, row[0], row[1])
pool.join()
def fetch(*args):
"""code trimmed for brevity"""
cursor.execute("""UPDATE com SET http_status=%s, is_checked=1 WHERE id=%s""",
(output.get('http_status', ""), id))
for _ in xrange(10000):
cursor.execute("SELECT domain, id FROM com WHERE is_checked IS NULL LIMIT 100")
result = cursor.fetchall()
check_urls(result)
db.commit()
cursor.close()
db.close()
My program get stuck at db.commit(). No values updated in the database. Can someone tell me whats wrong?.
Please note:
My check_urls function has a for loop and each loop has one mysql update query.

It seems like you should move your db.commit() into fetch function after cursor.execute. Another way you may use "global" keyword with cursor. Or, third way, just make cursor as parameter of functions.

Context manager for Python's MySQLdb

I am used to (spoiled by?) python's SQLite interface to deal with SQL databases. One nice feature in python's SQLite's API the "context manager," i.e., python's with statement. I usually execute queries in the following way:
import as sqlite
with sqlite.connect(db_filename) as conn:
query = "INSERT OR IGNORE INTO shapes VALUES (?,?);"
results = conn.execute(query, ("ID1","triangle"))
With the code above, if my query modifies the database and I forget to run conn.commit(),the context manager runs it for me automatically upon exiting the with statement. It also handles exceptions nicely: if an exception occurs before I commit anything, then the database is rolled back.
I am now using the MySQLdb interface, which doesn't seem to support a similar context manager out of the box. How do I create my own? There is a related question here, but it doesn't offer a complete solution.

Previously, MySQLdb connections were context managers.
As of this commit on 2018-12-04, however, MySQLdb connections are no longer context managers,
and users must explicitly call conn.commit() or conn.rollback(), or write their own context manager, such as the one below.
You could use something like this:
import config
import MySQLdb
import MySQLdb.cursors as mc
import _mysql_exceptions
import contextlib
DictCursor = mc.DictCursor
SSCursor = mc.SSCursor
SSDictCursor = mc.SSDictCursor
Cursor = mc.Cursor
#contextlib.contextmanager
def connection(cursorclass=Cursor,
host=config.HOST, user=config.USER,
passwd=config.PASS, dbname=config.MYDB,
driver=MySQLdb):
connection = driver.connect(
host=host, user=user, passwd=passwd, db=dbname,
cursorclass=cursorclass)
try:
yield connection
except Exception:
connection.rollback()
raise
else:
connection.commit()
finally:
connection.close()
#contextlib.contextmanager
def cursor(cursorclass=Cursor, host=config.HOST, user=config.USER,
passwd=config.PASS, dbname=config.MYDB):
with connection(cursorclass, host, user, passwd, dbname) as conn:
cursor = conn.cursor()
try:
yield cursor
finally:
cursor.close()
with cursor(SSDictCursor) as cur:
print(cur)
connection = cur.connection
print(connection)
sql = 'select * from table'
cur.execute(sql)
for row in cur:
print(row)
To use it you would place config.py in your PYTHONPATH and define the HOST, USER, PASS, MYDB variables there.

Think things have changed since this question was originally asked. Somewhat confusingly (from my point of view at least), for recent versions of MySQLdb, if you use a connection in a context you get a cursor (as per the oursql example), not something that closes automatically (as you would if you opened a file for instance).
Here's what I do:
from contextlib import closing
with closing(getConnection()) as conn: #ensure that the connection is closed
with conn as cursor: #cursor will now auto-commit
cursor.execute('SELECT * FROM tablename')