I have a small issue(for lack of a better word) with MySQL db. I am using Python.
So I have this table in which rows are inserted regularly. As regularly as 1 row /sec.
I run two Python scripts together. One that simulates the insertion at 1 row/sec. I have also turned autocommit off and explicitly commit after some number of rows, say 10.
The other script is a simple "SELECT count(*) ..." query on the table. This query doesn't show me the number of rows the table currently has. It is stubbornly stuck at whatever number of rows the table had initially when the script started running. I have even tried "SELECT SQL_NO_CACHE count(*) ..." to no effect.
Any help would be appreciated.
My guess is you're using INNODB with REPEATABLE READ isolation mode. Try setting the isolation mode to READ COMMITTED:
SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED
Another way is starting a new transaction every time you perform a select query. Read more here
If autocommit is turned off in the reader as well, then it will be doing the reads inside a transaction and thus not seeing the writes the other script is doing.
My guess is that either the reader or writer (most likely the writer) is operating inside a transaction which hasn't been committed. Try ensuring that the writer is committing after each write, and try a ROLLBACK from the reader to make sure that it isn't inside a transaction either.
Related
I have a python script to execute a stored procedure to purge the tables in database. This SP further calls another SP which has delete statements for each table in database. Something like below -
Python calls - Stored procedure Purge_DB
Purge_DB calls - Stored procedure Purge_Table
Purge_Table has definition to delete data from each table.
When I run this python script, the transaction logs increase exponentially and on running this script 2-3 times, I get the transaction log full error.
Please note that the deletion happens in transaction.
BEGIN TRAN
EXEC (#DEL_SQL)
COMMIT TRAN
Earlier I was executing the same SP using VB script and never got any issue related to transaction log.
Is there a different way that Python uses to create transaction log?
Why is the log size much bigger with Python than VB script?
This is resolved now.
Python starts a transaction when execute method is called and that transaction remains open until we explicitly call commit() method. Since, this purge SP was called for more than 100 tables, the transaction log was populated until transaction was closed in the python code and hence, it was getting full because of this job.
I have set the autocommit property of pyodbc to true which will now automatically commit each SQL statement as and when it is executed as part of that connection. Please refer to the documentation here -
https://github.com/mkleehammer/pyodbc/wiki/Database-Transaction-Management
I'm trying to build some autotest scripts using Kenneth Reitz's records library. I need to check the MySQL database for updates here and there.
db_url_mysql = 'mysql://user:pass#mysql.localhost:3306/DB'
db = records.Database(db_url_mysql)
rows = db.query("SELECT * FROM Users").all()
However, I encountered that sometimes simple queries return an empty dataset, ignoring updates in tables. I could totally see the updates in GUI clients or even using records in alternative python console.
I found that records use SQLAlchemy with default isolation level "REPEATABLE READ". Repeatable reads ignore all updates until an end of the transaction, so I could not see them.
Is there any way I can change that? I probably need to close a transaction and open a new one, or perhaps change isolation level for this connection, but how can I do it?
Now I can answer to my question.
With the release of records 0.5.0 (Novemmber 2016) there is now a possibility to pass arguments to SQLAlchemy driver. There was no such possibility in 0.4.3 which I used.
So, to set up isolation level on connection we should use
db_url_mysql = 'mysql://user:pass#mysql.localhost:3306/DB'
db = records.Database(db_url_mysql,isolation_level="READ_UNCOMMITTED")
I making a simple python script which checks a mysql table every x seconds and print the result to the console. I use the MySQL Connector Driver.
However, running the script only prints the initalial values. By that I mean, that if I change the values in the database while my script is running, it's not registered by the script and it's keeps on writing the initial values.
The code which retrieves the values in a while loop is as follows:
def get_fans():
global cnx
query = 'SELECT * FROM config'
while True:
cursor = cnx.cursor()
cursor.execute(query)
for (config_name, config_value) in cursor:
print config_name, config_value
print "\n"
cursor.close()
time.sleep(3)
Why is this happening?
Most likely, it's an autocommit issue. MySQL Connector Driver documentation states, that it has autocommit turned off. Make sure you commit your implicit transactions while changing your table. Also because default isolation is REPEATABLE READ you have:
All consistent reads within the same transaction read the snapshot
established by the first read.
So I guess you have to manage transaction even for your polling script. Or change isolation level to READ COMMITTED.
Though, the better way is to restore to MySQL client default autocommit-on mode. Even though PEP 249 guides to have it initially disabled, it's mere a proposal and most likely a serious design mistake. Not only it makes novices wonder about uncommited changes, makes even your read-only workload slower, it complicates data-layer design and breaks explicit is better than implicit Python zen. Even sluggish things like Django have it rethought.
I am writing a program on python which interacts with MySQL database.
For sql queries I use MySQLdb.
The problem is that fetchone() returns None but with the database browser I can see that that row exists.
This piece of code:
query = "SELECT * FROM revision WHERE rev_id=%s;"
cursor.execute(query % revision_id)
row = cursor.fetchone()
if row == None:
raise Exception("there isn't revision with id %s" % revision_id)
I have no idea what is going on here. Any ideas?
EDIT: okay, in some cases it works in some cases it doesn't but anyway when it
does not work the row exists in the table. I am passing a cursor object to a function and the code above is in the function. The problem is connected with this cursor object. Could the problem be that I pass the cursor as an argument to the function? How can I test it?
EDIT2: yes, the problem is that cursor does not work after I use it several times. Wether because other program connects to the DB or I am doing something wrong.
I have while loop in which I call a function to get info from the DB. After some iterations it does not work again. There is another program which writes to
the DB while while loop works.
Okay, db.autocommit(True) solved my problem.
This is related to transaction isolation level on your MySQL server. In the case of REPEATABLE_READ which is the default level for InnoDb, a snapshot is created at the time of first read, and subsequent read by the same cursor are made from this snapshot. Read more about isolation levels here
What we usually require while reusing the same cursor to run multiple queries, is READ_COMMITTED. Thankfully, if you can not change this on your SQL server, you can set your cursor to a particular isolation level.
cur = conn.cursor()
cur.execute("SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED")
This makes sure that every query you make, there is a fresh latest committed snapshot is used.
Best Practice is to commit db, after all query executed db.commit()
In Python, is there a way to get notified that a specific table in a MySQL database has changed?
It's theoretically possible but I wouldn't recommend it:
Essentially you have a trigger on the the table the calls a UDF which communicates with your Python app in some way.
Pitfalls include what happens if there's an error?
What if it blocks? Anything that happens inside a trigger should ideally be near-instant.
What if it's inside a transaction that gets rolled back?
I'm sure there are many other problems that I haven't thought of as well.
A better way if possible is to have your data access layer notify the rest of your app. If you're looking for when a program outside your control modifies the database, then you may be out of luck.
Another way that's less ideal but imo better than calling an another program from within a trigger is to set some kind of "LastModified" table that gets updated by triggers with triggers. Then in your app just check whether that datetime is greater than when you last checked.
If by changed you mean if a row has been updated, deleted or inserted then there is a workaround.
You can create a trigger in MySQL
DELIMITER $$
CREATE TRIGGER ai_tablename_each AFTER INSERT ON tablename FOR EACH ROW
BEGIN
DECLARE exec_result integer;
SET exec_result = sys_exec(CONCAT('my_cmd '
,'insert on table tablename '
,',id=',new.id));
IF exec_result = 0 THEN BEGIN
INSERT INTO table_external_result (id, tablename, result)
VALUES (null, 'tablename', 0)
END; END IF;
END$$
DELIMITER ;
This will call executable script my_cmd on the server. (see sys_exec fro more info) with some parameters.
my_cmd can be a Python program or anything you can execute from the commandline using the user account that MySQL uses.
You'd have to create a trigger for every change (INSERT/UPDATE/DELETE) that you'd want your program to be notified of, and for each table.
Also you'd need to find some way of linking your running Python program to the command-line util that you call via sys_exec().
Not recommended
This sort of behaviour is not recommend because it is likely to:
slow MySQL down;
make it hang/timeout if my_cmd does not return;
if you are using transaction, you will be notified before the transaction ends;
I'm not sure if you'll get notified of a delete if the transaction rolls back;
It's an ugly design
Links
sys_exec: http://www.mysqludf.org/lib_mysqludf_sys/index.php
Yes, may not be SQL standard. But PostgreSQL supports this with LISTEN and NOTIFY since around Version 9.x
http://www.postgresql.org/docs/9.0/static/sql-notify.html
Not possible with standard SQL functionality.
It might not be a bad idea to try using a network monitor instead of a MySQL trigger. Extending a network monitor like this:
http://sourceforge.net/projects/pynetmontool/
And then writing a script that waits for activity on port 3306 (or whatever port your MySQL server listens on), and then checks the database when the network activity meets certain filter conditions.
It's a very high level idea that you'll have to research further, but you don't run into the DB trigger problems and you won't have to write a cron job that runs every second.