We Keep Coding

MariaDB - Inserting Values doesn't affect any rows

I want to insert given values from my docker app-service to the MariaDB-service.
The connection has been established because I can execute SELECT * FROM via the MariaDB.connection.cursor.
First of all I create the connection:
def get_conn() -> mariadb.connection:
try:
conn = mariadb.connect(
user="XXX",
database="XXX",
password="XXX",
host="db",
port=33030,
)
except mariadb.Error as e:
print(f'Error connecting to MariaDB Platform: {e}')
sys.exit(1)
return conn
Then I create a mariadb.connection.cursor-Object:
def get_cur() -> mariadb.connection.cursor:
conn = get_conn()
cur = conn.cursor()
return cur
Finally I want to insert new values in the table testing:
def write_data():
cursor = get_cur()
conn = get_conn()
cursor.execute('INSERT INTO testing (title) VALUE ("2nd automatic entry");')
print("Executed Query")
conn.commit()
cursor.close()
conn.close()
print("Closed Connection")
return True
To test, if the entries are inserted, I started with 1 manual entry, then executed the write_data()-function and to finish of I inserted a 2nd manual entry via the console.
After the procedure the table looks like:
Note that the ìd is on AUTO_INCREMENT. So the function write_data() was not skipped entirely, because the 2nd manual entry got the id 3 and not 2.

You're committing a transaction in a different connection than the one your cursor belongs to.
get_conn() creates a new database connection and returns it.
get_cur() calls get_conn, that gets it a new connection, retrieves a cursor object that belongs to it, and returns it.
In your main code, you call get_conn - that gives you connection A.
Then you obtain a cursor by calling get_cur - that creates a connection B and returns a cursor belonging to it.
You run execute on the cursor object (Connection B) but commit the connection you got in the first call (Connection A).
PS: This was a really fun problem to debug, thanks :)

It's really easy, in a new table with new code, to unintentionally do an INSERT without a COMMIT. That is especially true using the Python connector, which doesn't use autocommit. A dropped connection with an open transaction rolls back the transaction. And, a rolled-back INSERT does not release the autoincremented ID value for reuse.
This kind of thing happens, and it's no cause for alarm.
A wise database programmer won't rely on a set of autoincrementing IDs with no gaps in it.

need to commit (or rollback) after encountering an error in SELECT statements

when encountering an error in SELECT statements, apparently TRANSACTION stays open.
make connection with psycopg:
# ...
connection.autocommit = False
cur = connection.cursor(cursor_factory=DictCursor)
select the non-exists table:
cur.execute('SELECT * FROM "non_exists_table"') # a incorrect query
psycopg2.errors.UndefinedTable: relation "non_exists_table" does not exist
then execute another query:
cur.execute('SELECT * FROM "exists_table"') # this is a correct query
psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block
why?
Does psycopg2 open a TRANSACTION for all execute()? even SELECT?!!
Is it possible to avoid start TRANSACTION in select query?

tanks from #snakecharmerb for comment
by this reference:
By default, the first time a command is sent to the database (using one of the cursors created by the connection), a new transaction is created.
Should any command fail, the transaction will be aborted and no further command will be executed until a call to the rollback() method.
solution1:
connection.autocommit = True
solution2:
with psycopg2.connect(DSN) as conn:
with conn.cursor() as curs:
curs.execute(SQL)
if no exception has been raised by the block, the transaction is committed. In case of exception the transaction is rolled back.
Unlike file objects or other resources, exiting the connection’s with block doesn’t close the connection, but only the transaction associated to it.

psycopg2 why is the schema not created?

im trying to create a schema in postgres database using psycopg2.
For some reason the schema is not created and later on the code crashes because it tries to refer to the missing schema. The connection is set to auto commit mode, which definetly works because i can create a database with this specific connection.
For debugging purposes i have wrapped every step in it's own try/except statement.
Code is below, as it is right there, it does not raise any exceptions, just the follow up crashes because the schema is missing.
def createDB(dbName, connString):
conn = psycopg2.connect(connString)
conn.set_session(autocommit =True) # autocommit must be True sein, else CREATE DATABASE will fail https://www.psycopg.org/docs/usage.html#transactions-control
cursor = conn.cursor()
createDB = sql.SQL('CREATE DATABASE {};').format(
sql.Identifier(dbName)
)
createSchema = sql.SQL('CREATE SCHEMA IF NOT EXISTS schema2;')
searchpath = sql.SQL('ALTER DATABASE {} SET search_path TO public, schema2;').format(
sql.Identifier(dbName)
)
dropDB = sql.SQL('DROP DATABASE IF EXISTS {};').format(
sql.Identifier(dbName)
)
try:
cursor.execute(dropDB)
except Exception as e:
print('drop DB failed')
logging.error(e)
conn.close()
exit()
try:
cursor.execute(createDB)
except Exception as e:
print('create DB failed')
logging.error(e)
conn.close()
exit()
try:
cursor.execute(createSchema)
print('schema created')
except Exception as e:
print('create schema failed')
logging.error(e)
conn.close()
exit()
try:
cursor.execute(searchpath)
except Exception as e:
print('set searchpath failed')
logging.error(e)
conn.close()
exit()
conn.close()
Adding an explicit commit does not do the trick either.
What am i missing?
EDIT
I have added a small screenshot with the console logs. As you can see, the code below gets executed.
EDIT 2
Out of sheer curiosity, i have tried to execute this very SQL statement in pgadmin:
CREATE SCHEMA IF NOT EXISTS schema2
and it works just fine, which shows, that my SQL is not wrong, so back to square one.
EDIT 3 -- Solution
So i have come up with a solution, thank to you #jjanes for pointing me in the right direction. This function does not connect to a specific database, but the server as a whole, since im using it to create new databases, hence the connection string looks something like this :
user=postgres password=12345 host=localhost port=5432
Which allows me to perform server level operations like create and drop database. But schemas are a Database level operation. Moving the exact same logic to the part of the code which is connected to the newly created database works like a charm.

You create the schema in the original database specified by the connect string. Once you create the new database, you need to connect to it in order to work in it. Otherwise, you are just working in the old database.

When does sqlite3 roll back transactions in python, and when doesn't it?

I am creating a little workshop to teach how to use python and SQL and came across this oddity. I wanted to show how to use the with statement to create a transaction with sqlite:
import sqlite3
filename = 'data/transaction.db'
print("_________________________")
print("Create Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
sqls = [
'DROP TABLE IF EXISTS test',
'CREATE TABLE test (i integer)',
'INSERT INTO "test" VALUES(99)',
'SELECT * FROM test']
for sql in sqls:
cursor.execute(sql)
print(cursor.fetchall())
print("_________________________")
print("Create Error with 'with'")
try:
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
sqls = [
'update test set i = 1',
'SELECT * FROM test',
'fnord', # <-- trigger error
'update test set i = 0',]
for sql in sqls:
cursor.execute(sql)
print(cursor.fetchall())
except sqlite3.OperationalError as err:
print(err)
# near "fnord": syntax error
print("_________________________")
print("Show Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM test')
for row in cursor:
print(row)
# (99,)
This works exactly as expected. However to prove that without the with block the executions would be done halfway I tried the following:
print("_________________________")
print("Create Error without 'with'")
conn = sqlite3.connect(filename)
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
cursor.execute( 'UPDATE test SET i = 1 WHERE i = 99')
print(cursor.fetchall())
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
cursor.execute( 'update test set i = 0')
print(cursor.fetchall())
cursor.execute( 'SELECT * FROM test')
print(cursor.fetchall())
conn.close()
print("_________________________")
print("Show Table")
with sqlite3.connect(filename) as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM test')
for row in cursor:
print(row)
# (99,)`
The whole output is:
_________________________
Create Table
[]
[]
[]
[(99,)]
_________________________
Create Error with 'with'
[]
[(1,)]
near "fnord": syntax error
_________________________
Show Table
(99,)
_________________________
Create Error without 'with'
[(99,)]
[]
[(1,)]
[]
[(0,)]
_________________________
Show Table
(99,) # Why is this not (0,)???
I am very confused as to why the last Block shows a 99 again. Eventually the plan is to add a try,except block with an exception, such that the SQL code mimics the first block - however I am confused without this already :).
Thanks for clarifying

From the python sqlite3 API doc:
The underlying sqlite3 library operates in autocommit mode by default,
but the Python sqlite3 module by default does not.
autocommit mode means that statements that modify the database take
effect immediately. A BEGIN or SAVEPOINT statement disables autocommit
mode, and a COMMIT, a ROLLBACK, or a RELEASE that ends the outermost
transaction, turns autocommit mode back on.
The Python sqlite3 module by default issues a BEGIN statement
implicitly before a Data Modification Language (DML) statement (i.e.
INSERT/UPDATE/DELETE/REPLACE).
Python will rollback transactions if the connection is closed without a commit (or an explicit ROLLBACK is issued). No transactions are committed in this program.
FYI a new connection is created in the "Create error without 'with'" block, but no new cursor is instantiated.

The Python with statement works with context managers. Whereas some context managers will release resources and possibly close an object, it seems that at least with the sqlite3.connection object, it merely commits or rolls back transactions but does not close the connection. This can be confirmed for the DB-API 2.0 interface:
Connection objects can be used as context managers that automatically commit or rollback transactions. In the event of an exception, the transaction is rolled back; otherwise, the transaction is committed:
...
# Successful, con.commit() is called automatically afterwards
...
# con.rollback() is called after the with block finishes with an exception, the
# exception is still raised and must be caught
...
# Connection object used as context manager only commits or rollbacks transactions,
# so the connection object should be closed manually
For the non-with statement block, you are never committing the transactions. When the connection is closed, all changes are automatically rolled back.
You need to call
conn.commit();
See Why the need to commit explicitly when doing an UPDATE? for more details.
As a side note, the section of your code titled "Create Error without 'with'" does not actually cause an error/exception.

Print Data from MySQL Database to Console from Python

I'm using Visual Studio 2017 with a Python Console environment. I have a MySQL database set up which I can connect to successfully. I can also Insert data into the DB. Now I'm trying to display/fetch data from it.
I connect fine, and it seems I'm fetching data from my database, but nothing is actually printing to the console. I want to be able to fetch and display data, but nothing is displaying at all.
How do I actually display the data I select?
#importing module Like Namespace in .Net
import pypyodbc
#creating connection Object which will contain SQL Server Connection
connection = pypyodbc.connect('Driver={SQL Server};Server=DESKTOP-NJR6F8V\SQLEXPRESS;Data Source=DESKTOP-NJR6F8V\SQLEXPRESS;Integrated Security=True;Connect Timeout=30;Encrypt=False;TrustServerCertificate=True;ApplicationIntent=ReadWrite;MultiSubnetFailover=False')
cursor = connection.cursor()
SQLCommand = ("SELECT ID FROM MyAI_DB.dbo.WordDefinitions WHERE ID > 117000")
#Processing Query
cursor.execute(SQLCommand)
#Commiting any pending transaction to the database.
connection.commit()
#closing connection
#connection.close()

I figured it out. I failed to include the right Print statement. Which was:
print(cursor.fetchone())
I also had the connection.commit statement in the wrong place (it was inserted even executing the Print statement). The final code that worked was this:
#importing module Like Namespace in .Net
import pypyodbc
#creating connection Object which will contain SQL Server Connection
connection = pypyodbc.connect('Driver={SQL Server};Server=DESKTOP-NJR6F8V\SQLEXPRESS;Data Source=DESKTOP-NJR6F8V\SQLEXPRESS;Integrated Security=True;Connect Timeout=30;Encrypt=False;TrustServerCertificate=True;ApplicationIntent=ReadWrite;MultiSubnetFailover=False')
cursor = connection.cursor()
SQLCommand = ("SELECT * FROM MyAI_DB.dbo.WordDefinitions")
#Processing Query
cursor.execute(SQLCommand)
#Commiting any pending transaction to the database.
print(cursor.fetchone())
connection.commit()
#closing connection
#connection.close()