On my Windows machine, I have a very simple database on MariaDB (10.3.7) to which I connect with the mysql-connector-python-rf (2.2.2).
I also connect to the database with 2 instances of HeidiSQL workbench.
When I add or delete a line in a data table using one of the work benches, I can immediately access the changed data with a SELECT statement in the other work bench. My conclusion: the first work bench has already committed the change to the data base.
However, seeing the change in Python seems more complicated. I have to add a commit() before the query to see the changes:
config = {'user' : 'some_user',
'password': 'some_password',
'host' : '127.0.0.1',
'database': 'some_database',
'raise_on_warnings': True,
}
db = mysql.connector.connect(**config)
# wait some to make changes to the database using the HeidiSQL workbenches
db.commit() # even though Python has not changed anything which needs to be
# committed, this seems necessary to re-read the db to catch
# the changes that were committed by the other clients
cursor = db.cursor()
cursor.execute('some_SQL_query')
for result in cursor:
do_something_with(result)
cursor.close()
So far I thought commit() is used to commit changes that Python wants to make to the database.
Is it correct to say that commit() also reads changes into Python that were done by other clients since the last connect()? Is this a bug/inconvenience or a feature?
Or is something else going on here that I am missing?
The thread writing issues COMMIT after writing. Doing the COMMIT in the reading thread has no effect.
I would not change the "isolation level" unless you need for the reader to see unfinished changes while they are happening. This is not normally required.
So, the writer should issue COMMIT as soon as it has finished some unit of work. That might be a single INSERT; it might be a long, complicated, combination of operations. A simple example is the classic 'transfer of funds:
BEGIN;
UPDATE accounts SET balance = balance + 100 WHERE id = 123; -- my account
UPDATE accounts SET balance = balance - 100 WHERE id = 432; -- your account
COMMIT;
For the integrity of accounts you want both UPDATEs to either happen nor not, even if the system crashes in the middle. And you don't want any other thread to see an inconsistency in balance if it reads the data in the middle.
Another way to phrase it: The writer is responsible for saying "I'm done" (by calling commit).
As #brunodesthuilliers has pointed out, the answer seems to be in the isolation levels. The default for Python seems to be REPEATABLE READ. To always read the latest commits it is necessary to change the transactions' isolation level, e.g. to READ COMMITTED.
config = {'user' : 'some_user',
'password': 'some_password',
'host' : '127.0.0.1',
'database': 'some_database',
'raise_on_warnings': True,
}
db = mysql.connector.connect(**config)
cursor = db.cursor()
cursor.execute('SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;')
cursor.close()
# wait some to make changes to the database using the HeidiSQL workbenches
cursor = db.cursor()
cursor.execute('some_SQL_query') # will now read the last committed data
for result in cursor:
do_something_with(result)
cursor.close()
Related
I am currently working on a project and I use the MariaDB connector to run the queries.
I can't use ORM so I have to use raw queries.
In general, the system works fine, as expected, but when I make a bit 'big' queries I get a Too many connections error message.
This has happened to me for both MySQL and MariaDB connectors, but I mainly use MariaDB.
Example of my code (truncated / simplified):
import mariadb
def get_cursor():
conn = mariadb.connect(
user="user",
password="pass",
host="localhost",
database="db")
return conn, conn.cursor(named_tuple=True)
def get_duplicated_variants():
results = []
conn_cursor = get_cursor()
cursor = conn_cursor[1]
conn = conn_cursor[0]
try:
cursor.execute("SELECT * FROM `db`.s_data;")
columns = [column[0] for column in cursor.description]
results = []
for row in cursor.fetchall():
results.append(dict(zip(columns, row)))
cursor.close()
conn.close()
return results
except mariadb.Error as e:
print(f"Error: {e}")
What I've tried:
show status like '%onn%';
And also: show variables like 'max_connections';
So the max_used_connections = 152 and I have 2503 Connections.
I also tried to execute the following query:
SELECT
CONCAT('KILL ', id, ';')
FROM INFORMATION_SCHEMA.PROCESSLIST
WHERE `User` = 'user'
AND `Host` = 'localhost'
AND `db` = 'db';
As seen in this question.
But the number of connections is the same after running the query, it does not work.
How could I close the connections properly?
I don't understand why the connections are still active since I use both cursor.close() to close the cursor and conn.close() to close the connection, but the connection is still active apparently.
I know I can increase max_connections with something like: set global max_connections = 500; but I would like to close the connections from the backend after the queries are done.
Any idea?
The API located here for connection close() certainly is clear that the connection is closed.
I realize you said you truncated the code, but seeing your comment on 2,503 connections in a single program certainly makes it seem like you aren't sharing that connection and are creating new connections for each query. I would suggest you inspect the code that you did not include to ensure you are properly storing and reusing that connection which will be expensive to keep recreating.
Finally, I would instead be looking at the state tables with something like netstat to see which connections are really going and where they are coming/going from. It isn't entirely clear to me that you are excluding connections which may be from other entities to/from the DB or that the connection isn't actually getting destroyed. In short, I am somewhat unsure if you are chasing a red herring here. I still think the >2000 connections is something which seems unexpected and you should be chasing that down first as to why so many connections are getting created in the first place, based on the code you provided.
Hello everyone I have the following issue,
I am trying to run a simple UPDATE query using sqlalchemy and psycopg2 for Postgres.
the query is
update = f"""
UPDATE {requests_table_name}
SET status = '{status}', {column_db_table} = '{dt.datetime.now()}'
WHERE request_id = '{request_id}'
"""
and then commit the changes using
cursor.execute(update).commit()
But it throws an error that AttributeError: 'NoneType' object has no attribute 'commit'
My connection string is
engine = create_engine(
f'postgresql://{self.params["user"]}:{self.params["password"]}#{self.params["host"]}:{self.params["port"]}/{self.params["database"]}')
conn = engine.connect().connection
cursor = conn.cursor()
The other thing is that cursor is always closed <cursor object at 0x00000253827D6D60; closed: 0>
The connection with the database is ok, I can etch tables and update them using pandas pd_to_sql method, but with commmiting using cursor it does not work. It works perfect with sql server but not with postgres.
In postgres, however, it creates a PID with the status "idle in transaction" and Client: ClientRead, every time I run cursor.execute(update).commit().
I connot get where is the problem, in the code or in the database.
I tried to use different methods to initiate a cursor, like raw_connection(), but without a result.
I checked for Client: ClientRead with idle in transaction but am not sure how to overcome it.
You have to call commit() on the connection object.
According to the documentation, execute() returns None.
Note that even if you use a context manager like this:
with my_connection.cursor() as cur:
cur.execute('INSERT INTO ..')
You may find your database processes still getting stuck in the idle in transaction state. The COMMIT is handled at the connection level, like #laurenz-albe said, so you need to wrap that too:
with my_connection as conn:
with conn.cursor() as cur:
cur.execute('INSERT INTO ..')
It's spelled out clearly in the documentation, but I still managed to overlook it.
I have a question regarding MySQL and transactions. I work with MySQL 5.7.18, python 3 and the Oracle mysql connector v2.1.4
I do not understand the difference between
a) having a transaction and –in case of error – rollback and
b) not having a transaction and – in case of error – simply not commiting the changes.
Both seem to leave me with exactly the same results (i.e. no entries in table, see code example below). Does this have to do with using InnoDB – would the results differ otherwise?
What is the advantage of using a transaction if
1) I cannot rollback commited changes and
2) I could just as well not commit changes (until I am done with my task or sure that some query didn’t raise any exceptions)?
I have tried to find the answers to those questions in https://downloads.mysql.com/docs/connector-python-en.a4.pdf but failed to find the essential difference.
Somebody asked an almost identical question and received some replies but I don’t think those actually contain an answer: Mysql transaction : commit and rollback Replies focused on having multiple connections open and visibility of changes. Is that all there is to it?
import mysql.connector
# Connect to MySQL-Server
conn = mysql.connector.connect(user='test', password='blub',
host='127.0.0.1', db='my_test')
cursor = conn.cursor(buffered=True)
# This is anyway the default in mysql.connector
# cursor.autocommit = False
sql = """CREATE TABLE IF NOT EXISTS `my_test`.`employees` (
`emp_no` int(11) NOT NULL AUTO_INCREMENT,
`first_name` varchar(14) NOT NULL,
PRIMARY KEY (`emp_no`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8"""
try:
cursor.execute(sql)
conn.commit()
except:
print("error")
# Arguments on default values
# conn.start_transaction(consistent_snapshot=False,
# isolation_level=None, readonly=False)
sql = """INSERT INTO `my_test`.`employees`
(`first_name`)
VALUES
(%s);"""
employees = {}
employees["1"] = ["Peter"]
employees["2"] = ["Bruce"]
for employee, value in employees.items():
cursor.execute(sql, (value[0],))
print(conn.in_transaction)
# If I do not commit the changes, table is left empty (whether I write
# start_transaction or not)
# If I rollback the changes (without commit first), table is left empty
# (whether I write start_transaction or not)
# If I commit and then rollback, the rollback had no effect (i.e. there are
# values in the table (whether I write start_transaction or not)
conn.commit()
conn.rollback()
Thank you very much for your help in advance! I appreciate it.
I think having not committed nor rolled back leaves the transaction in a running state, in which it may still hold resources like locks etc
Well it doesn't matter which db you are using when you call a transaction ,it will lock the resource (I.e any table) until the transaction is completed or rolled back for example if i write a transaction to insert something to a table test the test table will be locked until the transaction is completed this may leads to deadlock since others may need that table...You can try it on yourself just open two instances of your mysql in the first instance run transaction without commit and in the second try to insert something on the same table ...it will clear your doubt
Transactions prevent other queries from modifying the data while your query is running. Furthermore, a transaction scope can contain multiple queries - so you can rollback ALL of them in the event of an error, whereas that is not the case if some of them run successfully and only one query results in error, in which case you may end up with partially committed results, like JLH said.
Your decision to have a transaction should take into account the numerous reasons for having one, including having multiple statements each of which commits writes the database.
In your example I don't think it makes a difference, but in more complicated scenarios you need a transaction to ensure ACID.
Context
So I am trying to figure out how to properly override the auto-transaction when using SQLite in Python. When I try and run
cursor.execute("BEGIN;")
.....an assortment of insert statements...
cursor.execute("END;")
I get the following error:
OperationalError: cannot commit - no transaction is active
Which I understand is because SQLite in Python automatically opens a transaction on each modifying statement, which in this case is an INSERT.
Question:
I am trying to speed my insertion by doing one transaction per several thousand records.
How can I overcome the automatic opening of transactions?
As #CL. said you have to set isolation level to None. Code example:
s = sqlite3.connect("./data.db")
s.isolation_level = None
try:
c = s.cursor()
c.execute("begin")
...
c.execute("commit")
except:
c.execute("rollback")
The documentaton says:
You can control which kind of BEGIN statements sqlite3 implicitly executes (or none at all) via the isolation_level parameter to the connect() call, or via the isolation_level property of connections.
If you want autocommit mode, then set isolation_level to None.
I have a script that waits until some row in a db is updated:
con = MySQLdb.connect(server, user, pwd, db)
When the script starts the row's value is "running", and it waits for the value to become "finished"
while(True):
sql = '''select value from table where some_condition'''
cur = self.getCursor()
cur.execute(sql)
r = cur.fetchone()
cur.close()
res = r['value']
if res == 'finished':
break
print res
time.sleep(5)
When I run this script it hangs forever. Even though I see the value of the row has changed to "finished" when I query the table, the printout of the script is still "running".
Is there some setting I didn't set?
EDIT: The python script only queries the table. The update to the table is carried out by a tomcat webapp, using JDBC, that is set on autocommit.
This is an InnoDB table, right? InnoDB is transactional storage engine. Setting autocommit to true will probably fix this behavior for you.
conn.autocommit(True)
Alternatively, you could change the transaction isolation level. You can read more about this here:
http://dev.mysql.com/doc/refman/5.0/en/set-transaction.html
The reason for this behavior is that inside a single transaction the reads need to be consistent. All consistent reads within the same transaction read the snapshot established by the first read. Even if you script only reads the table this is considered a transaction too. This is the default behavior in InnoDB and you need to change that or run conn.commit() after each read.
This page explains this in more details: http://dev.mysql.com/doc/refman/5.0/en/innodb-consistent-read.html
I worked around this by running
c.execute("""set session transaction isolation level READ COMMITTED""")
early on in my reading session. Updates from other threads do come through now.
In my instance I was keeping connections open for a long time (inside mod_python) and so updates by other processes weren't being seen at all.