I'm using MySQLdb in Python.
I have an update that may succeed or fail:
UPDATE table
SET reserved_by = PID
state = "unavailable"
WHERE state = "available"
AND id = REQUESTED_ROW_ID
LIMIT 1;
As you may be able to infer, multiple processes are using the database, and I need processes to be able to securely grab rows for themselves, without race conditions causing problems.
My theory (perhaps incorrect) is that only one process will be able to succeed with this query (.rowcount=1) -- the others will fail (.rowcount=0) or get a different row (.rowcount=1).
The problem is, it appears that everything that happens through MySQLdb happens in a virtual world -- .rowcount reads =1, but you can't really know whether anything really happened, until you perform a .commit().
My questions:
In MySQL, is a single UPDATE atomic within itself? That is, if the same UPDATE above (with different PID values, but the same REQUESTED_ROW_ID) were sent to the same MySQL server at "once," am I guaranteed that one will succeed and the other will fail?
Is there a way to know, after calling "conn.commit()", whether there was a meaningful change or not?
** Can I get a reliable .rowcount for the actual commit operation?
Does the .commit operation send the actual query (SET's and WHERE conditions intact,) or does it just perform the SETs on affected rows, independent the WHERE clauses that inspired them?
Is my problem solved neatly by .autocommit?
Turn autocommit on.
The commit operation just "confirms" updates already done. The alternative is rollback, which "undoes" any updates already made.
Related
I am using a postgres-DB and a pyton script, which should be notified for DB-Changes with the postgres NOTIFY-functionality.
In all examples i can find for this topic the trigger in postgres is implemented with BEFORE like in this example:
CREATE TRIGGER notify_on_changes
BEFORE UPDATE OR INSERT OR DELETE
ON table_bla_bla
FOR EACH ROW
EXECUTE PROCEDURE notify_changes();
what is the reason for using BEFORE and not AFTER? I do not want to change anything before inserting/updating or deleting a row.
Shouldn't it be better to use AFTER?
AFTER triggers have to be queued up in memory for later execution, so are less efficient.
BEFORE triggers carry the risk that some other BEFORE trigger will modify the row after you have seen it but before it is written.
I'm trying to call a stored procedure in my MSSQL database from a python script, but it does not run completely when called via python. This procedure consolidates transaction data into hour/daily blocks in a single table which is later grabbed by the python script. If I run the procedure in SQL studio, it completes just fine.
When I run it via my script, it gets cut short about 2/3's of the way through. Currently I found a work around, by making the program sleep for 10 seconds before moving on to the next SQL statement, however this is not time efficient and unreliable as some procedures may not finish in that time. I'm looking for a more elegant way to implement this.
Current Code:
cursor.execute("execute mySP")
time.sleep(10)
cursor.commit()
The most related article I can find to my issue is here:
make python wait for stored procedure to finish executing
I tried the solution using Tornado and I/O generators, but ran into the same issue as listed in the article, that was never resolved. I also tried the accepted solution to set a runningstatus field in the database by my stored procedures. At the beginnning of my SP Status is updated to 1 in RunningStatus, and when the SP finished Status is updated to 0 in RunningStatus. Then I implemented the following python code:
conn=pyodbc_connect(conn_str)
cursor=conn.cursor()
sconn=pyodbc_connect(conn_str)
scursor=sconn.cursor()
cursor.execute("execute mySP")
cursor.commit()
while 1:
q=scursor.execute("SELECT Status FROM RunningStatus").fetchone()
if(q[0]==0):
break
When I implement this, the same problem happens as before with my storedprocedure finishing executing prior to it actually being complete. If I eliminate my cursor.commit(), as follows, I end up with the connection just hanging indefinitely until I kill the python process.
conn=pyodbc_connect(conn_str)
cursor=conn.cursor()
sconn=pyodbc_connect(conn_str)
scursor=sconn.cursor()
cursor.execute("execute mySP")
while 1:
q=scursor.execute("SELECT Status FROM RunningStatus").fetchone()
if(q[0]==0):
break
Any assistance in finding a more efficient and reliable way to implement this, as opposed to time.sleep(10) would be appreciated.
As OP found out, inconsistent or imcomplete processing of stored procedures from application layer like Python may be due to straying from best practices of TSQL scripting.
As #AaronBetrand highlights in this Stored Procedures Best Practices Checklist blog, consider the following among other items:
Explicitly and liberally use BEGIN ... END blocks;
Use SET NOCOUNT ON to avoid messages sent to client for every row affected action, possibly interrupting workflow;
Use semicolons for statement terminators.
Example
CREATE PROCEDURE dbo.myStoredProc
AS
BEGIN
SET NOCOUNT ON;
SELECT * FROM foo;
SELECT * FROM bar;
END
GO
While using cx_Oracle(Python), the code goes into waiting when the the following statement is executed:
some_connection.execute(some_sql)
What could be the reason?
Without seeing the actual SQL in question it is hard to know for sure. Some possible answers include:
1) the SQL actually takes a long time to execute (and you just have to be patient)
2) the SQL is blocked by another transaction (and that transaction needs to be committed or rolled back first)
You can find out by examining the contents of dba_locks, specifically looking at the blocking_others column. You can also attempt to issue the same SQL in SQL*Plus and see if it exhibits the same behaviour.
This question already has answers here:
Closed 11 years ago.
Possible Duplicate:
Django cache.set() causing duplicate key error
I ran into this problem using django core's database cache:
ERROR: duplicate key value violates unique constraint "cache_pkey"
STATEMENT: INSERT INTO "cache" (cache_key, value, expires) VALUES (E':1:cms-menu_nodes_en-us_1', E'gAJdcQEoY21lbnVzLmJhc2UKTmF2aW
LOG: server process (PID 8453) was terminated by signal 9: Killed
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
FATAL: could not create shared memory segment: Cannot allocate memory
DETAIL: Failed system call was shmget(key=5432001, size=29278208, 03600).
I looked in the table and sure enough, there is an entry for the key ':1:cms-menu_nodes_en-us_1'. I found a similar issue here, but was unable to exactly understand what the issue is.
Anyone have any ideas or suggestions? Sounds like a bug in django core, since if a key exist, it should update the record.
edit: I should have clarified that the DB was PostgreSQL 8.4.7. Thanks lazerscience.
edit # Jack M: I haven't been able to replicate this error, but believe the code is in django.core.cache.backends.db.DatabaseCache in a method called set() that calls _base_set().
Sounds like a bug in django core, since if a key exist, it should update the record.
Indeed, but I'd suggest that said bug is related to a concurrency issue, in which case it could be fixed at the app level. As in two neighboring calls to the same asset/page/whatever run an exist() statement, find no row, and proceed to insert as a result -- without issuing a lock of any kind, and without wrapping the thing in a transaction to discard the offending call and (since it's just a cache) continuing.
It also begs one question: are you sure that you should be caching in your database in the first place? The database typically is a bottleneck in a web application (especially when using an ORM), and the whole point of caching is to avoid that bottleneck. Shouldn't you be using memcache instead?
In Python, is there a way to get notified that a specific table in a MySQL database has changed?
It's theoretically possible but I wouldn't recommend it:
Essentially you have a trigger on the the table the calls a UDF which communicates with your Python app in some way.
Pitfalls include what happens if there's an error?
What if it blocks? Anything that happens inside a trigger should ideally be near-instant.
What if it's inside a transaction that gets rolled back?
I'm sure there are many other problems that I haven't thought of as well.
A better way if possible is to have your data access layer notify the rest of your app. If you're looking for when a program outside your control modifies the database, then you may be out of luck.
Another way that's less ideal but imo better than calling an another program from within a trigger is to set some kind of "LastModified" table that gets updated by triggers with triggers. Then in your app just check whether that datetime is greater than when you last checked.
If by changed you mean if a row has been updated, deleted or inserted then there is a workaround.
You can create a trigger in MySQL
DELIMITER $$
CREATE TRIGGER ai_tablename_each AFTER INSERT ON tablename FOR EACH ROW
BEGIN
DECLARE exec_result integer;
SET exec_result = sys_exec(CONCAT('my_cmd '
,'insert on table tablename '
,',id=',new.id));
IF exec_result = 0 THEN BEGIN
INSERT INTO table_external_result (id, tablename, result)
VALUES (null, 'tablename', 0)
END; END IF;
END$$
DELIMITER ;
This will call executable script my_cmd on the server. (see sys_exec fro more info) with some parameters.
my_cmd can be a Python program or anything you can execute from the commandline using the user account that MySQL uses.
You'd have to create a trigger for every change (INSERT/UPDATE/DELETE) that you'd want your program to be notified of, and for each table.
Also you'd need to find some way of linking your running Python program to the command-line util that you call via sys_exec().
Not recommended
This sort of behaviour is not recommend because it is likely to:
slow MySQL down;
make it hang/timeout if my_cmd does not return;
if you are using transaction, you will be notified before the transaction ends;
I'm not sure if you'll get notified of a delete if the transaction rolls back;
It's an ugly design
Links
sys_exec: http://www.mysqludf.org/lib_mysqludf_sys/index.php
Yes, may not be SQL standard. But PostgreSQL supports this with LISTEN and NOTIFY since around Version 9.x
http://www.postgresql.org/docs/9.0/static/sql-notify.html
Not possible with standard SQL functionality.
It might not be a bad idea to try using a network monitor instead of a MySQL trigger. Extending a network monitor like this:
http://sourceforge.net/projects/pynetmontool/
And then writing a script that waits for activity on port 3306 (or whatever port your MySQL server listens on), and then checks the database when the network activity meets certain filter conditions.
It's a very high level idea that you'll have to research further, but you don't run into the DB trigger problems and you won't have to write a cron job that runs every second.