We Keep Coding

When does sqlite3 roll back transactions in python, and when doesn't it?

I am creating a little workshop to teach how to use python and SQL and came across this oddity. I wanted to show how to use the with statement to create a transaction with sqlite:
import sqlite3
filename = 'data/transaction.db'
print("_________________________")
print("Create Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
sqls = [
'DROP TABLE IF EXISTS test',
'CREATE TABLE test (i integer)',
'INSERT INTO "test" VALUES(99)',
'SELECT * FROM test']
for sql in sqls:
cursor.execute(sql)
print(cursor.fetchall())
print("_________________________")
print("Create Error with 'with'")
try:
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
sqls = [
'update test set i = 1',
'SELECT * FROM test',
'fnord', # <-- trigger error
'update test set i = 0',]
for sql in sqls:
cursor.execute(sql)
print(cursor.fetchall())
except sqlite3.OperationalError as err:
print(err)
# near "fnord": syntax error
print("_________________________")
print("Show Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM test')
for row in cursor:
print(row)
# (99,)
This works exactly as expected. However to prove that without the with block the executions would be done halfway I tried the following:
print("_________________________")
print("Create Error without 'with'")
conn = sqlite3.connect(filename)
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
cursor.execute( 'UPDATE test SET i = 1 WHERE i = 99')
print(cursor.fetchall())
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
cursor.execute( 'update test set i = 0')
print(cursor.fetchall())
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
conn.close()
print("_________________________")
print("Show Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM test')
for row in cursor:
print(row)
# (99,)`
The whole output is:
_________________________
Create Table
[]
[]
[]
[(99,)]
_________________________
Create Error with 'with'
[]
[(1,)]
near "fnord": syntax error
_________________________
Show Table
(99,)
_________________________
Create Error without 'with'
[(99,)]
[]
[(1,)]
[]
[(0,)]
_________________________
Show Table
(99,) # Why is this not (0,)???
I am very confused as to why the last Block shows a 99 again. Eventually the plan is to add a try,except block with an exception, such that the SQL code mimics the first block - however I am confused without this already :).
Thanks for clarifying

From the python sqlite3 API doc:
The underlying sqlite3 library operates in autocommit mode by default,
but the Python sqlite3 module by default does not.
autocommit mode means that statements that modify the database take
effect immediately. A BEGIN or SAVEPOINT statement disables autocommit
mode, and a COMMIT, a ROLLBACK, or a RELEASE that ends the outermost
transaction, turns autocommit mode back on.
The Python sqlite3 module by default issues a BEGIN statement
implicitly before a Data Modification Language (DML) statement (i.e.
INSERT/UPDATE/DELETE/REPLACE).
Python will rollback transactions if the connection is closed without a commit (or an explicit ROLLBACK is issued). No transactions are committed in this program.
FYI a new connection is created in the "Create error without 'with'" block, but no new cursor is instantiated.

The Python with statement works with context managers. Whereas some context managers will release resources and possibly close an object, it seems that at least with the sqlite3.connection object, it merely commits or rolls back transactions but does not close the connection. This can be confirmed for the DB-API 2.0 interface:
Connection objects can be used as context managers that automatically commit or rollback transactions. In the event of an exception, the transaction is rolled back; otherwise, the transaction is committed:
...
# Successful, con.commit() is called automatically afterwards
...
# con.rollback() is called after the with block finishes with an exception, the
# exception is still raised and must be caught
...
# Connection object used as context manager only commits or rollbacks transactions,
# so the connection object should be closed manually
For the non-with statement block, you are never committing the transactions. When the connection is closed, all changes are automatically rolled back.
You need to call
conn.commit();
See Why the need to commit explicitly when doing an UPDATE? for more details.
As a side note, the section of your code titled "Create Error without 'with'" does not actually cause an error/exception.

Print Data from MySQL Database to Console from Python

I'm using Visual Studio 2017 with a Python Console environment. I have a MySQL database set up which I can connect to successfully. I can also Insert data into the DB. Now I'm trying to display/fetch data from it.
I connect fine, and it seems I'm fetching data from my database, but nothing is actually printing to the console. I want to be able to fetch and display data, but nothing is displaying at all.
How do I actually display the data I select?
#importing module Like Namespace in .Net
import pypyodbc
#creating connection Object which will contain SQL Server Connection
connection = pypyodbc.connect('Driver={SQL Server};Server=DESKTOP-NJR6F8V\SQLEXPRESS;Data Source=DESKTOP-NJR6F8V\SQLEXPRESS;Integrated Security=True;Connect Timeout=30;Encrypt=False;TrustServerCertificate=True;ApplicationIntent=ReadWrite;MultiSubnetFailover=False')
cursor = connection.cursor()
SQLCommand = ("SELECT ID FROM MyAI_DB.dbo.WordDefinitions WHERE ID > 117000")
#Processing Query
cursor.execute(SQLCommand)
#Commiting any pending transaction to the database.
connection.commit()
#closing connection
#connection.close()

I figured it out. I failed to include the right Print statement. Which was:
print(cursor.fetchone())
I also had the connection.commit statement in the wrong place (it was inserted even executing the Print statement). The final code that worked was this:
#importing module Like Namespace in .Net
import pypyodbc
#creating connection Object which will contain SQL Server Connection
connection = pypyodbc.connect('Driver={SQL Server};Server=DESKTOP-NJR6F8V\SQLEXPRESS;Data Source=DESKTOP-NJR6F8V\SQLEXPRESS;Integrated Security=True;Connect Timeout=30;Encrypt=False;TrustServerCertificate=True;ApplicationIntent=ReadWrite;MultiSubnetFailover=False')
cursor = connection.cursor()
SQLCommand = ("SELECT * FROM MyAI_DB.dbo.WordDefinitions")
#Processing Query
cursor.execute(SQLCommand)
#Commiting any pending transaction to the database.
print(cursor.fetchone())
connection.commit()
#closing connection
#connection.close()

Python psycopg2 Execute Hangs

I am using Python 3.x within an app hosted on Heroku with Basic PostgreSQL instance and Im using the psycopg2 library (as "lite")
Recently started hanging when I call Execute on the cursor object.
I am not sure how to troubleshoot this and would appreciate any thoughts.
Here is how I instantiate Connection:
def getConnection():
urlparse.uses_netloc.append("postgres")
url = urlparse.urlparse(os.environ["HEROKU_STUFF_HERE"])
con = lite.connect(
database=url.path[1:],
user=url.username,
password=url.password,
host=url.hostname,
port=url.port
)
return con
Here is how I generally execute non-return type statements:
def ExecuteSQL(sql):
con = DB.getConnection()
with con:
cur = con.cursor()
print("In")
cur.execute(sql)
print("out")
The app NEVER sees the light after printing the word "In" to the console.
I've left try except blocks out intentionally so as to blow the thing up.....no dice.
pretty much doesnt matter what the SQL Statement is, I get the same result
Running the same sql in a sql client tool executes instantly.
Im also not sure how to detect if this statement even makes it into Postgresql.....
Thanks for any help you can offer

psycopg2 opened the transaction which is not closed until a commit() or rollback(). Therefore your changes were not persistent (docs).
def ExecuteSQL(sql):
con = DB.getConnection()
with con:
cur = con.cursor()
cur.execute(sql)
con.commit()
or you could just do the following:
def ExecuteSQL(sql):
con = DB.getConnection()
with con:
con.autocommit = True
cur = con.cursor()
cur.execute(sql)

"Invalid cursor state" error when executing a batch that includes a USE statement

I tried retrieving data from a Microsoft SQL database using pypyodbc 1.3.3 with Python 3.5 on Windows but got a pypyodbc.ProgrammingError '[24000] [Microsoft] [SQL Server Native Client 11.0] Invalid cursor state' using the following code:
import pypyodbc
conn = pypyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=database;UID=uid;PWD=pwd')
cursor = conn.cursor()
sql = '''USE database;
SELECT R0
FROM table;'''
cursor.execute(sql)
results = cursor.fetchone()
print(results)
The SQL works in Microsoft SQL Server Management Studio, the connection and executing worked in another script i wrote to insert into the same database and also works if i remove
results = cursor.fetchone()
So far I tried cursor.fetchone(), cursor.fetchall() and list(cursor) but all produced the same result which leads me to believe that the command itself isn't the problem.
According to this microsoft site it means that there isn't an open cursor, but I can get it's description, so from my understanding there has to be.

It's not a matter of being unable to execute a USE ... statement at all, it's just that we cannot do that as part of a multi-statement batch. So, this will not work ...
crsr.execute("""\
USE master;
SELECT TOP 2 name FROM sys.tables ORDER BY name;
""")
rows = crsr.fetchall() # error
... but this will work fine
crsr.execute("USE master")
crsr.execute("SELECT TOP 2 name FROM sys.tables ORDER BY name")
rows = crsr.fetchall()
(Tested with both pypyodbc 1.3.4 and pyodbc 4.0.21)

I had a similar issue. I was able to resolve this by removing the "USE Database" statement.
You already connected to your db here:
conn = pypyodbc.connect(r'DRIVER={SQL Server Native Client 11.0};SERVER=server;DATABASE=database;UID=uid;PWD=pwd')

Postgres raises a "ACTIVE SQL TRANSACTION" (Errcode: 25001)

I use psycopg2 for accessing my postgres database in python. My function should create a new database, the code looks like this:
def createDB(host, username, dbname):
adminuser = settings.DB_ADMIN_USER
adminpass = settings.DB_ADMIN_PASS
try:
conn=psycopg2.connect(user=adminuser, password=adminpass, host=host)
cur = conn.cursor()
cur.execute("CREATE DATABASE %s OWNER %s" % (nospecial(dbname), nospecial(username)))
conn.commit()
except Exception, e:
raise e
finally:
cur.close()
conn.close()
def nospecial(s):
pattern = re.compile('[^a-zA-Z0-9_]+')
return pattern.sub('', s)
When I call createDB my postgres server throws an error:
CREATE DATABASE cannot run inside a transaction block
with the errorcode 25001 which stands for "ACTIVE SQL TRANSACTION".
I'm pretty sure that there is no other connection running at the same time and every connection I used before calling createDB is shut down.

It looks like your cursor() is actually a transaction:
http://initd.org/psycopg/docs/cursor.html#cursor
Cursors created from the same
connection are not isolated, i.e., any
changes done to the database by a
cursor are immediately visible by the
other cursors. Cursors created from
different connections can or can not
be isolated, depending on the
connections’ isolation level. See also
rollback() and commit() methods.
Skip the cursor and just execute your query. Drop commit() as well, you can't commit when you don't have a transaction open.