We Keep Coding

When does sqlite3 roll back transactions in python, and when doesn't it?

I am creating a little workshop to teach how to use python and SQL and came across this oddity. I wanted to show how to use the with statement to create a transaction with sqlite:
import sqlite3
filename = 'data/transaction.db'
print("_________________________")
print("Create Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
sqls = [
'DROP TABLE IF EXISTS test',
'CREATE TABLE test (i integer)',
'INSERT INTO "test" VALUES(99)',
'SELECT * FROM test']
for sql in sqls:
cursor.execute(sql)
print(cursor.fetchall())
print("_________________________")
print("Create Error with 'with'")
try:
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
sqls = [
'update test set i = 1',
'SELECT * FROM test',
'fnord', # <-- trigger error
'update test set i = 0',]
for sql in sqls:
cursor.execute(sql)
print(cursor.fetchall())
except sqlite3.OperationalError as err:
print(err)
# near "fnord": syntax error
print("_________________________")
print("Show Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM test')
for row in cursor:
print(row)
# (99,)
This works exactly as expected. However to prove that without the with block the executions would be done halfway I tried the following:
print("_________________________")
print("Create Error without 'with'")
conn = sqlite3.connect(filename)
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
cursor.execute( 'UPDATE test SET i = 1 WHERE i = 99')
print(cursor.fetchall())
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
cursor.execute( 'update test set i = 0')
print(cursor.fetchall())
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
conn.close()
print("_________________________")
print("Show Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM test')
for row in cursor:
print(row)
# (99,)`
The whole output is:
_________________________
Create Table
[]
[]
[]
[(99,)]
_________________________
Create Error with 'with'
[]
[(1,)]
near "fnord": syntax error
_________________________
Show Table
(99,)
_________________________
Create Error without 'with'
[(99,)]
[]
[(1,)]
[]
[(0,)]
_________________________
Show Table
(99,) # Why is this not (0,)???
I am very confused as to why the last Block shows a 99 again. Eventually the plan is to add a try,except block with an exception, such that the SQL code mimics the first block - however I am confused without this already :).
Thanks for clarifying

From the python sqlite3 API doc:
The underlying sqlite3 library operates in autocommit mode by default,
but the Python sqlite3 module by default does not.
autocommit mode means that statements that modify the database take
effect immediately. A BEGIN or SAVEPOINT statement disables autocommit
mode, and a COMMIT, a ROLLBACK, or a RELEASE that ends the outermost
transaction, turns autocommit mode back on.
The Python sqlite3 module by default issues a BEGIN statement
implicitly before a Data Modification Language (DML) statement (i.e.
INSERT/UPDATE/DELETE/REPLACE).
Python will rollback transactions if the connection is closed without a commit (or an explicit ROLLBACK is issued). No transactions are committed in this program.
FYI a new connection is created in the "Create error without 'with'" block, but no new cursor is instantiated.

The Python with statement works with context managers. Whereas some context managers will release resources and possibly close an object, it seems that at least with the sqlite3.connection object, it merely commits or rolls back transactions but does not close the connection. This can be confirmed for the DB-API 2.0 interface:
Connection objects can be used as context managers that automatically commit or rollback transactions. In the event of an exception, the transaction is rolled back; otherwise, the transaction is committed:
...
# Successful, con.commit() is called automatically afterwards
...
# con.rollback() is called after the with block finishes with an exception, the
# exception is still raised and must be caught
...
# Connection object used as context manager only commits or rollbacks transactions,
# so the connection object should be closed manually
For the non-with statement block, you are never committing the transactions. When the connection is closed, all changes are automatically rolled back.
You need to call
conn.commit();
See Why the need to commit explicitly when doing an UPDATE? for more details.
As a side note, the section of your code titled "Create Error without 'with'" does not actually cause an error/exception.

Does PyMySQL support SELECT...FOR UPDATE?

PyMySQL, a python package to access MySQL database, seems not support SELECT ... FOR UPDATE.
In the code below, I used SELECT...FOR UPDATE to read some_table in function f(), used UPDATE to modify the table in g(), and spawned 50 threads for each function.
I expected deadlock to happen since SELECT...FOR UPDATE should block the threads spawned by g. But actually no deadlock happened. Can some one explain why?
from threading import Thread
import pymysql
def f():
db = pymysql.connect("localhost", "tester","pwd", "testDB")
cur = db.cursor()
sql = "SELECT * FROM some_table FOR UPDATE"
try:
cur.execute(sql)
except:
print("Exception in select")
def g():
db = pymysql.connect("localhost", "tester", "pwd","testDB")
cur = db.cursor()
sql = "UPDATE some_table SET val=20 WHERE id=2"
try:
cur.execute(sql)
db.commit()
except:
print("Exception in update")
db.rollback()
for _ in range(50):
Thread(target=f, args=()).start()
Thread(target=g, args=()).start()
I am using Python 3.4 and PyMySQL 0.6.6. Thank you in advance.

PyMySQL does support SELECT ... FOR UPDATE
But you need start the transaction using connection.begin()
Here's an example:
connection= pymysql.connect("localhost", "tester", "pwd","testDB")
connection.begin()
cursor = db.cursor()
cursor.execute("SELECT * FROM some_table FOR UPDATE")
Table/row (depending upon your select query) is now in locked state. Only the current connection can make changes.
To release the lock. You can,
commit the changes (if any)
connection.commit()
close the connection
connection.close()

Python not inserting data's into mysql

I am testing Python and Mysql in that i am able to create and delete table's but i am unable to insert data in them.I searched stackoverflow and mostly they suggest to use
commit()
So i used it and even after i used the data is not inserted into the database.Please help me.
This is the code i use it creates the table but not inserting data
import MySQLdb
db = MySQLdb.connect("localhost","user","password")
cxn = MySQLdb.connect(db='test')
cursor = cxn.cursor()
cursor.execute("CREATE TABLE users(name VARCHAR(40),id VARCHAR(40))")
cursor.execute("INSERT INTO users(name,id) VALUES('John','1')")
db.commit()
print "Opertion completed successfully"

Are db and cxn connections to the same database?
You should establish your connection using following:
db = MySQLdb.connect(host="localhost",
db="test",
user="user",
passwd="password")
The cursor should then be derived from this connection via:
cursor = db.cursor()
I would hazard that your issue is coming from the ambiguity between db and cxn.

Context manager for Python's MySQLdb

I am used to (spoiled by?) python's SQLite interface to deal with SQL databases. One nice feature in python's SQLite's API the "context manager," i.e., python's with statement. I usually execute queries in the following way:
import as sqlite
with sqlite.connect(db_filename) as conn:
query = "INSERT OR IGNORE INTO shapes VALUES (?,?);"
results = conn.execute(query, ("ID1","triangle"))
With the code above, if my query modifies the database and I forget to run conn.commit(),the context manager runs it for me automatically upon exiting the with statement. It also handles exceptions nicely: if an exception occurs before I commit anything, then the database is rolled back.
I am now using the MySQLdb interface, which doesn't seem to support a similar context manager out of the box. How do I create my own? There is a related question here, but it doesn't offer a complete solution.

Previously, MySQLdb connections were context managers.
As of this commit on 2018-12-04, however, MySQLdb connections are no longer context managers,
and users must explicitly call conn.commit() or conn.rollback(), or write their own context manager, such as the one below.
You could use something like this:
import config
import MySQLdb
import MySQLdb.cursors as mc
import _mysql_exceptions
import contextlib
DictCursor = mc.DictCursor
SSCursor = mc.SSCursor
SSDictCursor = mc.SSDictCursor
Cursor = mc.Cursor
#contextlib.contextmanager
def connection(cursorclass=Cursor,
host=config.HOST, user=config.USER,
passwd=config.PASS, dbname=config.MYDB,
driver=MySQLdb):
connection = driver.connect(
host=host, user=user, passwd=passwd, db=dbname,
cursorclass=cursorclass)
try:
yield connection
except Exception:
connection.rollback()
raise
else:
connection.commit()
finally:
connection.close()
#contextlib.contextmanager
def cursor(cursorclass=Cursor, host=config.HOST, user=config.USER,
passwd=config.PASS, dbname=config.MYDB):
with connection(cursorclass, host, user, passwd, dbname) as conn:
cursor = conn.cursor()
try:
yield cursor
finally:
cursor.close()
with cursor(SSDictCursor) as cur:
print(cur)
connection = cur.connection
print(connection)
sql = 'select * from table'
cur.execute(sql)
for row in cur:
print(row)
To use it you would place config.py in your PYTHONPATH and define the HOST, USER, PASS, MYDB variables there.

Think things have changed since this question was originally asked. Somewhat confusingly (from my point of view at least), for recent versions of MySQLdb, if you use a connection in a context you get a cursor (as per the oursql example), not something that closes automatically (as you would if you opened a file for instance).
Here's what I do:
from contextlib import closing
with closing(getConnection()) as conn: #ensure that the connection is closed
with conn as cursor: #cursor will now auto-commit
cursor.execute('SELECT * FROM tablename')

Postgres raises a "ACTIVE SQL TRANSACTION" (Errcode: 25001)

I use psycopg2 for accessing my postgres database in python. My function should create a new database, the code looks like this:
def createDB(host, username, dbname):
adminuser = settings.DB_ADMIN_USER
adminpass = settings.DB_ADMIN_PASS
try:
conn=psycopg2.connect(user=adminuser, password=adminpass, host=host)
cur = conn.cursor()
cur.execute("CREATE DATABASE %s OWNER %s" % (nospecial(dbname), nospecial(username)))
conn.commit()
except Exception, e:
raise e
finally:
cur.close()
conn.close()
def nospecial(s):
pattern = re.compile('[^a-zA-Z0-9_]+')
return pattern.sub('', s)
When I call createDB my postgres server throws an error:
CREATE DATABASE cannot run inside a transaction block
with the errorcode 25001 which stands for "ACTIVE SQL TRANSACTION".
I'm pretty sure that there is no other connection running at the same time and every connection I used before calling createDB is shut down.

It looks like your cursor() is actually a transaction:
http://initd.org/psycopg/docs/cursor.html#cursor
Cursors created from the same
connection are not isolated, i.e., any
changes done to the database by a
cursor are immediately visible by the
other cursors. Cursors created from
different connections can or can not
be isolated, depending on the
connections’ isolation level. See also
rollback() and commit() methods.
Skip the cursor and just execute your query. Drop commit() as well, you can't commit when you don't have a transaction open.