I want to insert given values from my docker app-service to the MariaDB-service.
The connection has been established because I can execute SELECT * FROM via the MariaDB.connection.cursor.
First of all I create the connection:
def get_conn() -> mariadb.connection:
try:
conn = mariadb.connect(
user="XXX",
database="XXX",
password="XXX",
host="db",
port=33030,
)
except mariadb.Error as e:
print(f'Error connecting to MariaDB Platform: {e}')
sys.exit(1)
return conn
Then I create a mariadb.connection.cursor-Object:
def get_cur() -> mariadb.connection.cursor:
conn = get_conn()
cur = conn.cursor()
return cur
Finally I want to insert new values in the table testing:
def write_data():
cursor = get_cur()
conn = get_conn()
cursor.execute('INSERT INTO testing (title) VALUE ("2nd automatic entry");')
print("Executed Query")
conn.commit()
cursor.close()
conn.close()
print("Closed Connection")
return True
To test, if the entries are inserted, I started with 1 manual entry, then executed the write_data()-function and to finish of I inserted a 2nd manual entry via the console.
After the procedure the table looks like:
Note that the ìd is on AUTO_INCREMENT. So the function write_data() was not skipped entirely, because the 2nd manual entry got the id 3 and not 2.
You're committing a transaction in a different connection than the one your cursor belongs to.
get_conn() creates a new database connection and returns it.
get_cur() calls get_conn, that gets it a new connection, retrieves a cursor object that belongs to it, and returns it.
In your main code, you call get_conn - that gives you connection A.
Then you obtain a cursor by calling get_cur - that creates a connection B and returns a cursor belonging to it.
You run execute on the cursor object (Connection B) but commit the connection you got in the first call (Connection A).
PS: This was a really fun problem to debug, thanks :)
It's really easy, in a new table with new code, to unintentionally do an INSERT without a COMMIT. That is especially true using the Python connector, which doesn't use autocommit. A dropped connection with an open transaction rolls back the transaction. And, a rolled-back INSERT does not release the autoincremented ID value for reuse.
This kind of thing happens, and it's no cause for alarm.
A wise database programmer won't rely on a set of autoincrementing IDs with no gaps in it.
Related
when encountering an error in SELECT statements, apparently TRANSACTION stays open.
make connection with psycopg:
# ...
connection.autocommit = False
cur = connection.cursor(cursor_factory=DictCursor)
select the non-exists table:
cur.execute('SELECT * FROM "non_exists_table"') # a incorrect query
psycopg2.errors.UndefinedTable: relation "non_exists_table" does not exist
then execute another query:
cur.execute('SELECT * FROM "exists_table"') # this is a correct query
psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block
why?
Does psycopg2 open a TRANSACTION for all execute()? even SELECT?!!
Is it possible to avoid start TRANSACTION in select query?
tanks from #snakecharmerb for comment
by this reference:
By default, the first time a command is sent to the database (using one of the cursors created by the connection), a new transaction is created.
Should any command fail, the transaction will be aborted and no further command will be executed until a call to the rollback() method.
solution1:
connection.autocommit = True
solution2:
with psycopg2.connect(DSN) as conn:
with conn.cursor() as curs:
curs.execute(SQL)
if no exception has been raised by the block, the transaction is committed. In case of exception the transaction is rolled back.
Unlike file objects or other resources, exiting the connection’s with block doesn’t close the connection, but only the transaction associated to it.
im trying to create a schema in postgres database using psycopg2.
For some reason the schema is not created and later on the code crashes because it tries to refer to the missing schema. The connection is set to auto commit mode, which definetly works because i can create a database with this specific connection.
For debugging purposes i have wrapped every step in it's own try/except statement.
Code is below, as it is right there, it does not raise any exceptions, just the follow up crashes because the schema is missing.
def createDB(dbName, connString):
conn = psycopg2.connect(connString)
conn.set_session(autocommit =True) # autocommit must be True sein, else CREATE DATABASE will fail https://www.psycopg.org/docs/usage.html#transactions-control
cursor = conn.cursor()
createDB = sql.SQL('CREATE DATABASE {};').format(
sql.Identifier(dbName)
)
createSchema = sql.SQL('CREATE SCHEMA IF NOT EXISTS schema2;')
searchpath = sql.SQL('ALTER DATABASE {} SET search_path TO public, schema2;').format(
sql.Identifier(dbName)
)
dropDB = sql.SQL('DROP DATABASE IF EXISTS {};').format(
sql.Identifier(dbName)
)
try:
cursor.execute(dropDB)
except Exception as e:
print('drop DB failed')
logging.error(e)
conn.close()
exit()
try:
cursor.execute(createDB)
except Exception as e:
print('create DB failed')
logging.error(e)
conn.close()
exit()
try:
cursor.execute(createSchema)
print('schema created')
except Exception as e:
print('create schema failed')
logging.error(e)
conn.close()
exit()
try:
cursor.execute(searchpath)
except Exception as e:
print('set searchpath failed')
logging.error(e)
conn.close()
exit()
conn.close()
Adding an explicit commit does not do the trick either.
What am i missing?
EDIT
I have added a small screenshot with the console logs. As you can see, the code below gets executed.
EDIT 2
Out of sheer curiosity, i have tried to execute this very SQL statement in pgadmin:
CREATE SCHEMA IF NOT EXISTS schema2
and it works just fine, which shows, that my SQL is not wrong, so back to square one.
EDIT 3 -- Solution
So i have come up with a solution, thank to you #jjanes for pointing me in the right direction. This function does not connect to a specific database, but the server as a whole, since im using it to create new databases, hence the connection string looks something like this :
user=postgres password=12345 host=localhost port=5432
Which allows me to perform server level operations like create and drop database. But schemas are a Database level operation. Moving the exact same logic to the part of the code which is connected to the newly created database works like a charm.
You create the schema in the original database specified by the connect string. Once you create the new database, you need to connect to it in order to work in it. Otherwise, you are just working in the old database.
I'm using Visual Studio 2017 with a Python Console environment. I have a MySQL database set up which I can connect to successfully. I can also Insert data into the DB. Now I'm trying to display/fetch data from it.
I connect fine, and it seems I'm fetching data from my database, but nothing is actually printing to the console. I want to be able to fetch and display data, but nothing is displaying at all.
How do I actually display the data I select?
#importing module Like Namespace in .Net
import pypyodbc
#creating connection Object which will contain SQL Server Connection
connection = pypyodbc.connect('Driver={SQL Server};Server=DESKTOP-NJR6F8V\SQLEXPRESS;Data Source=DESKTOP-NJR6F8V\SQLEXPRESS;Integrated Security=True;Connect Timeout=30;Encrypt=False;TrustServerCertificate=True;ApplicationIntent=ReadWrite;MultiSubnetFailover=False')
cursor = connection.cursor()
SQLCommand = ("SELECT ID FROM MyAI_DB.dbo.WordDefinitions WHERE ID > 117000")
#Processing Query
cursor.execute(SQLCommand)
#Commiting any pending transaction to the database.
connection.commit()
#closing connection
#connection.close()
I figured it out. I failed to include the right Print statement. Which was:
print(cursor.fetchone())
I also had the connection.commit statement in the wrong place (it was inserted even executing the Print statement). The final code that worked was this:
#importing module Like Namespace in .Net
import pypyodbc
#creating connection Object which will contain SQL Server Connection
connection = pypyodbc.connect('Driver={SQL Server};Server=DESKTOP-NJR6F8V\SQLEXPRESS;Data Source=DESKTOP-NJR6F8V\SQLEXPRESS;Integrated Security=True;Connect Timeout=30;Encrypt=False;TrustServerCertificate=True;ApplicationIntent=ReadWrite;MultiSubnetFailover=False')
cursor = connection.cursor()
SQLCommand = ("SELECT * FROM MyAI_DB.dbo.WordDefinitions")
#Processing Query
cursor.execute(SQLCommand)
#Commiting any pending transaction to the database.
print(cursor.fetchone())
connection.commit()
#closing connection
#connection.close()
Can someone please explain to me why the defined test().commit() does not work as varcon.commit()? Everything else seem to work fine. (using vagrant virtualbox of ubuntu-trusty-32)
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import psycopg2
varcon = psycopg2.connect('dbname=tournament')
def test():
try:
psycopg2.connect("dbname=tournament")
except:
print("Connection to Tournament Database Failed")
else:
return psycopg2.connect('dbname=tournament')
def writer():
#db = psycopg2.connect('dbname=tournament')
c =varcon.cursor()
c.execute('select * from players')
data = c.fetchall()
c.execute("insert into players (name) values ('Joe Smith')")
varcon.commit()
varcon.close
print(data)
def writer2():
#db = psycopg2.connect('dbname=tournament')
c =test().cursor()
c.execute('select * from players')
data = c.fetchall()
c.execute("insert into players (name) values ('Joe Smith')")
test().commit()
test().close
print(data)
writer2() #this seem not commited, but database registers the insert by observing the serial promotion
#writer() # this works as expected
maybe this is because the return statement in a function block (def) is not equal to an assignment like =
The psycopg2 connection function returns a connection object and this is assigned to conn or varcon
conn = psycopg2.connect("dbname=test user=postgres password=secret")
http://initd.org/psycopg/docs/module.html
the test() function also returns the psycopg2 connection object but in writer2 it is not assigned to a variable (memory place) meaning that there is no reference
this also explains why the database connection is established (initialized in the test function) but the commit does not work (broken reference)
(http://www.icu-project.org/docs/papers/cpp_report/the_anatomy_of_the_assignment_operator.html)
maybe try
ami=test()
ami.commit()
to establish a reference
Every time you call psycopg2.connect() you open new connection to database.
So effectively your code executes SQL in one connection, commits another, and then closes third connection. Even in your test() function you are opening two different connections.
I use the following pattern to access PostgreSQL:
conn = psycopg2.connect(DSN)
with conn:
with conn.cursor() as curs:
...
curs.execute(SQL1)
with conn:
with conn.cursor() as curs:
...
curs.execute(SQL2)
conn.close()
with statement ensures transaction is opened and properly committed around your SQL. It also automatically rolls transaction back in case your code inside with raises an exception.
Reference: http://initd.org/psycopg/docs/usage.html#with-statement
I use psycopg2 for accessing my postgres database in python. My function should create a new database, the code looks like this:
def createDB(host, username, dbname):
adminuser = settings.DB_ADMIN_USER
adminpass = settings.DB_ADMIN_PASS
try:
conn=psycopg2.connect(user=adminuser, password=adminpass, host=host)
cur = conn.cursor()
cur.execute("CREATE DATABASE %s OWNER %s" % (nospecial(dbname), nospecial(username)))
conn.commit()
except Exception, e:
raise e
finally:
cur.close()
conn.close()
def nospecial(s):
pattern = re.compile('[^a-zA-Z0-9_]+')
return pattern.sub('', s)
When I call createDB my postgres server throws an error:
CREATE DATABASE cannot run inside a transaction block
with the errorcode 25001 which stands for "ACTIVE SQL TRANSACTION".
I'm pretty sure that there is no other connection running at the same time and every connection I used before calling createDB is shut down.
It looks like your cursor() is actually a transaction:
http://initd.org/psycopg/docs/cursor.html#cursor
Cursors created from the same
connection are not isolated, i.e., any
changes done to the database by a
cursor are immediately visible by the
other cursors. Cursors created from
different connections can or can not
be isolated, depending on the
connections’ isolation level. See also
rollback() and commit() methods.
Skip the cursor and just execute your query. Drop commit() as well, you can't commit when you don't have a transaction open.