While using cx_Oracle(Python), the code goes into waiting when the the following statement is executed:
some_connection.execute(some_sql)
What could be the reason?
Without seeing the actual SQL in question it is hard to know for sure. Some possible answers include:
1) the SQL actually takes a long time to execute (and you just have to be patient)
2) the SQL is blocked by another transaction (and that transaction needs to be committed or rolled back first)
You can find out by examining the contents of dba_locks, specifically looking at the blocking_others column. You can also attempt to issue the same SQL in SQL*Plus and see if it exhibits the same behaviour.
Related
I'm trying to call a stored procedure in my MSSQL database from a python script, but it does not run completely when called via python. This procedure consolidates transaction data into hour/daily blocks in a single table which is later grabbed by the python script. If I run the procedure in SQL studio, it completes just fine.
When I run it via my script, it gets cut short about 2/3's of the way through. Currently I found a work around, by making the program sleep for 10 seconds before moving on to the next SQL statement, however this is not time efficient and unreliable as some procedures may not finish in that time. I'm looking for a more elegant way to implement this.
Current Code:
cursor.execute("execute mySP")
time.sleep(10)
cursor.commit()
The most related article I can find to my issue is here:
make python wait for stored procedure to finish executing
I tried the solution using Tornado and I/O generators, but ran into the same issue as listed in the article, that was never resolved. I also tried the accepted solution to set a runningstatus field in the database by my stored procedures. At the beginnning of my SP Status is updated to 1 in RunningStatus, and when the SP finished Status is updated to 0 in RunningStatus. Then I implemented the following python code:
conn=pyodbc_connect(conn_str)
cursor=conn.cursor()
sconn=pyodbc_connect(conn_str)
scursor=sconn.cursor()
cursor.execute("execute mySP")
cursor.commit()
while 1:
q=scursor.execute("SELECT Status FROM RunningStatus").fetchone()
if(q[0]==0):
break
When I implement this, the same problem happens as before with my storedprocedure finishing executing prior to it actually being complete. If I eliminate my cursor.commit(), as follows, I end up with the connection just hanging indefinitely until I kill the python process.
conn=pyodbc_connect(conn_str)
cursor=conn.cursor()
sconn=pyodbc_connect(conn_str)
scursor=sconn.cursor()
cursor.execute("execute mySP")
while 1:
q=scursor.execute("SELECT Status FROM RunningStatus").fetchone()
if(q[0]==0):
break
Any assistance in finding a more efficient and reliable way to implement this, as opposed to time.sleep(10) would be appreciated.
As OP found out, inconsistent or imcomplete processing of stored procedures from application layer like Python may be due to straying from best practices of TSQL scripting.
As #AaronBetrand highlights in this Stored Procedures Best Practices Checklist blog, consider the following among other items:
Explicitly and liberally use BEGIN ... END blocks;
Use SET NOCOUNT ON to avoid messages sent to client for every row affected action, possibly interrupting workflow;
Use semicolons for statement terminators.
Example
CREATE PROCEDURE dbo.myStoredProc
AS
BEGIN
SET NOCOUNT ON;
SELECT * FROM foo;
SELECT * FROM bar;
END
GO
I've tried to see other similar problems, but they did not quite give me the answer I was looking for. My main goal is to store a large dataset into a google cloud. I tried to store around 1000 insert statement to google cloud and it went well.
In the other hand, storing over 200.000 insert statement proved to be challenge than I thought. In my code, I have this function:
def insert_to_gcloud(information):
db = psycopg2.connect(database="database",
user="user",
password="password",
host="host",
port=1234)
cur = db.cursor()
cur.execute("".join(information))
db.commit()
db.close()
I use batch insert (suppose to be faster) where the first index contain the insert statement and the rest values. Then I use "".join() to make it into a string. To make a simple example:
INSERT INTO tbl_name (a,b,c) VALUES (1,2,3),(4,5,6),(7,8,9);
["INSERT INTO tbl_name (a,b,c) VALUES",
"(1,2,3)," ,
"(4,5,6)," ,
"(7,8,9);" ]
At first, I tried to execute 200.000 insert statement, but I got an error about EOF after about 1 min. I guess it was to big to send, so I made a chunk function. Basically, it divides the array into proper size as I want it. (reference)
Then used a simple for-loop to store each one block at a time:
for i in range(len(chunk_block)):
insert_to_gcloudd(chunk_block[i])
It seems like it was working, but I let it run over the night, used 237 min, I got this message:
psycopg2.OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
My next test is to store the chunk block into files, read and log the files. If the server gets closed unexpectedly again, I can check the log. Thought, this is not a good way to do it in my opinion, but I'm out of ideas here.
So to my question, is there any option I could try? Maybe there are other tools I can use? 🙂
I have the following code, and the stored procedure is used to call several stored procedures. I can run the stored procedure and it will complete without issues in SQL 2012. I am using Python 3.3.
cnxn = pyodbc.connect('DRIVER={SQL Server};Server=.\SQLEXPRESS;Database=MyDatabase;Trusted_Connection=yes;')
cursor = cnxn.cursor()
cnxn.timeout = 0
cnxn.autocommit = True
cursor.execute("""exec my_SP""")
The python code is executing, I have determined this from inserting numerous prints.
I did see the other question regarding python waiting for the SP to finish. I tried adding a 'time.sleep()' after the execute, and varying the time (up to 120 seconds) no change.
The stored procedure appears to be partially executing, based on the results. The data suggests that it is even interrupting one of the sub-stored procedures, yet it is fine when the SP is run from query analyzer.
My best guess would be that this is something SQL config related, but I am lost in where to look.
Any thoughts?
Adding SET NOCOUNT OFF to my proc worked for me.
I had the same issue and solved it with a combination of setting a locking variable (see answer from Ben Caine in this thread: make python wait for stored procedure to finish executing) and adding
"SET NOCOUNT ON"
after "CREATE PROCEDURE ... AS"
Just a follow up; I have had limited success using the time features located at the link below, and reducing the level of nesting stored procedures.
At the level that I was calling in the above, there were 4 layers of nested SP's; pyodbc seems to behave a little better when you have 3 layers or less. Doesn't make a lot of sense to me, but it works.
make python wait for stored procedure to finish executing
Any input on the rationale behind this would be greatly appreciated.
The situation is detailed in my previous question:
MySQLdb is caching SELECT results?
In short:
python 2.7 + MySQLdb
the "issue" happens inside a Python script (but not from the mysql client prompt)
when querying a SELECT inside a loop, the first result is repeated for all subsequent iterations of the loop
this happens even though another program updates the DB (and commits).
I can see the changes from mysql clients, but not from my python loop.
SQL_NO_CACHE didn't fix it
recreating a cursor didn't help!
autocommit(True) worked --> each query reflects DB change.
So why MySQLdb thinks it's inside a transaction, when it's clearly irrelevant?
I'm using MySQLdb in Python.
I have an update that may succeed or fail:
UPDATE table
SET reserved_by = PID
state = "unavailable"
WHERE state = "available"
AND id = REQUESTED_ROW_ID
LIMIT 1;
As you may be able to infer, multiple processes are using the database, and I need processes to be able to securely grab rows for themselves, without race conditions causing problems.
My theory (perhaps incorrect) is that only one process will be able to succeed with this query (.rowcount=1) -- the others will fail (.rowcount=0) or get a different row (.rowcount=1).
The problem is, it appears that everything that happens through MySQLdb happens in a virtual world -- .rowcount reads =1, but you can't really know whether anything really happened, until you perform a .commit().
My questions:
In MySQL, is a single UPDATE atomic within itself? That is, if the same UPDATE above (with different PID values, but the same REQUESTED_ROW_ID) were sent to the same MySQL server at "once," am I guaranteed that one will succeed and the other will fail?
Is there a way to know, after calling "conn.commit()", whether there was a meaningful change or not?
** Can I get a reliable .rowcount for the actual commit operation?
Does the .commit operation send the actual query (SET's and WHERE conditions intact,) or does it just perform the SETs on affected rows, independent the WHERE clauses that inspired them?
Is my problem solved neatly by .autocommit?
Turn autocommit on.
The commit operation just "confirms" updates already done. The alternative is rollback, which "undoes" any updates already made.