I'd like to manage OpenERP user's activity by installing the audittrail module.
After creating some rules ( define which user, which object and which activity (create, update..) will be monitored). I update a product to see it works.
When I've tried to update a product i got the system error. Seeing the log, I get
[2010-08-31 12:53:35,042] Cursor not closed explicitly
[2010-08-31 12:53:35,043] Cursor was created at /home/pilgrim/working/sources/addons/audittrail/audittrail.py:204
Here the line that causes error
cr = pooler.get_db(db).cursor()
Looking at sql_db.py, I get the comment
def __del__(self):
if not self.__closed:
# Oops. 'self' has not been closed explicitly.
# The cursor will be deleted by the garbage collector,
# but the database connection is not put back into the connection
# pool, preventing some operation on the database like dropping it.
# This can also lead to a server overload.
msg = "Cursor not closed explicitly\n" \
"Cursor was created at %s:%s" % self.__caller
log(msg, netsvc.LOG_WARNING)
self.close()
Since I'm new to Python, I don't know how to overcome this issue?
Any hint to get over this?
Thank
t would be important to see the source code to understand whats going on.
But from what you have posted it looks like the previous cursor was not closed explicitly.
cr = sqldb.db_connect(dbname).cursor()
.........
cr.close()
cr = None
I would suggest that you hack audittrail.py to find where ever you are creating the cursor and where ever you close them. A typical issue arises in incorrect handling of exceptions, causing code to jump over normal closure.
Try placing a try, except and finally clause around the questionable cursor operation. That should help you to get around the problem.
I think I find the answer.
See an example
def a():
try:
print 'before return '
return 1
finally:
print 'in finally'
call a()
before return
in finally
1
It's normal. OK.
Try another example ( code extract from audittrail.py)
def do_something_with_db(db):
// open cusror again
cr = db.cursor()
// do somethign
// close cursor internally
cr.close()
def execute(db)
// 1, open connection and open cursor
cr = db.cursor
try:
//2, do something with db, seeing that this method will open cursor again
return do_something_with_db(db)
finally:
cr.close()
Seeing that the implementation of do_something_with_db trying to open the cursor ( can be called connection) but the current one is not explicitly closed.
So the solution is simple: Pass the current cr around
Before
**do_something_with_db(db)**
after
**do_something_with_db(cr)**
Now the error's gone.
#Don Kirkby: Yes, we should experiment with try...finally
Can you run OpenERP in a debugger like the PyDev plug in for Eclipse? I find that the most effective way to track down problems. I haven't used the audit trail module, but I took a quick look at the source code, and it appears that the cursor is being opened near the beginning of log_fct(). (I would have expected it to report line 207, which version are you running?) Here's what I think is the relevant code:
def log_fct(self, db, uid, passwd, object, method, fct_src, *args):
logged_uids = []
pool = pooler.get_pool(db)
cr = pooler.get_db(db).cursor() # line 207 in version 5.0.12
# ...
if method in ('create'):
# ...
cr.close()
return res_id
# ...
cr.close()
It looks like there are several return statements in the method, but each one seems to call cr.close() first, so I don't see any obvious problems. Try running it in the debugger with a break point in this method. If that's not possible, you can try writing to the log with something like this:
logger = netsvc.Logger()
logger.notifyChannel('audittrail', netsvc.LOG_INFO, 'something happened')
Update:
You commented that this happens under heavy load. Perhaps an exception is being thrown and the cursor is not being closed. You could use a try ... finally statement to make sure that the cursor is always closed. Here's how the sample above would look after converting it:
def log_fct(self, db, uid, passwd, object, method, fct_src, *args):
logged_uids = []
pool = pooler.get_pool(db)
cr = pooler.get_db(db).cursor() # line 207 in version 5.0.12
try:
# ...
if method in ('create'):
# ...
return res_id
# ...
finally:
cr.close()
Related
Is there a simple example of working with the neo4j python driver?
How do I just pass cypher query to the driver to run and return a cursor?
If I'm reading for example this it seems the demo has a class wrapper, with a private member func I pass to the session.write,
session.write_transaction(self._create_and_return_greeting, ...
That then gets called it with a transaction as a first parameter...
def _create_and_return_greeting(tx, message):
that in turn runs the cypher
result = tx.run("CREATE (a:Greeting) "
This seems 10X more complicated than it needs to be.
I did just try a simpler:
def raw_query(query, **kwargs):
neodriver = neo_connect() # cached dbconn
with neodriver.session() as session:
try:
result = session.run(query, **kwargs)
return result.data()
But this results in a socket error on the query, probably because the session goes out of scope?
[dfcx/__init__] ERROR | Underlying socket connection gone (_ssl.c:2396)
[dfcx/__init__] ERROR | Failed to write data to connection IPv4Address(('neo4j-core-8afc8558-3.production-orch-0042.neo4j.io', 7687)) (IPv4Address(('34.82.120.138', 7687)))
Also I can't return a cursor/iterator, just the data()
When the session goes out of scope, the query result seems to die with it.
If I manually open and close a session, then I'd have the same problems?
Python must be the most popular language this DB is used with, does everyone use a different driver?
Py2neo seems cute, but completely lacking in ORM wrapper function for most of the cypher language features, so you have to drop down to raw cypher anyway. And I'm not sure it supports **kwargs argument interpolation in the same way.
I guess that big raise should help iron out some kinks :D
Slightly longer version trying to get a working DB wrapper:
def neo_connect() -> Union[neo4j.BoltDriver, neo4j.Neo4jDriver]:
global raw_driver
if raw_driver:
# print('reuse driver')
return raw_driver
neoconfig = NEOCONFIG
raw_driver = neo4j.GraphDatabase.driver(
neoconfig['url'], auth=(
neoconfig['user'], neoconfig['pass']))
if raw_driver is None:
raise BaseException("cannot connect to neo4j")
else:
return raw_driver
def raw_query(query, **kwargs):
# just get data, no cursor
neodriver = neo_connect()
session = neodriver.session()
# logging.info('neoquery %s', query)
# with neodriver.session() as session:
try:
result = session.run(query, **kwargs)
data = result.data()
return data
except neo4j.exceptions.CypherSyntaxError as err:
logging.error('neo error %s', err)
logging.error('failed query: %s', query)
raise err
# finally:
# logging.info('close session')
# session.close()
update: someone pointed me to this example which is another way to use the tx wrapper.
https://github.com/neo4j-graph-examples/northwind/blob/main/code/python/example.py#L16-L21
def raw_query(query, **kwargs):
neodriver = neo_connect() # cached dbconn
with neodriver.session() as session:
try:
result = session.run(query, **kwargs)
return result.data()
This is perfectly fine and works as intended on my end.
The error you're seeing is stating that there is a connection problem. So there must be something going on between the server and the driver that's outside of its influence.
Also, please note, that there is a difference between all of these ways to run a query:
with driver.session():
result = session.run("<SOME CYPHER>")
def work(tx):
result = tx.run("<SOME CYPHER>")
with driver.session():
session.write_transaction(work)
The latter one might be 3 lines longer and the team working on the drivers collected some feedback regarding this. However, there are more things to consider here. Firstly, changing the API surface is something that needs careful planning and cannot be done in say a patch release. Secondly, there are technical hurdles to overcome. Here are the semantics, anyway:
Auto-commit transaction. Runs only that query as one unit of work.
If you run a new auto-commit transaction within the same session, the previous result will buffer all available records for you (depending on the query, this will consume a lot of memory). This can be avoided by calling result.consume(). However, if the session goes out of scope, the result will be consumed automatically. This means you cannot extract further records from it. Lastly, any error will be raised and needs handling in the application code.
Managed transaction. Runs whatever unit of work you want within that function. A transaction is implicitly started and committed (unless you rollback explicitly) around the function.
If the transaction ends (end of function or rollback), the result will be consumed and become invalid. You'll have to extract all records you need before that.
This is the recommended way of using the driver because it will not raise all errors but handle some internally (where appropriate) and retry the work function (e.g. if the server is only temporarily unavailable). Since the function might be executed multiple time, you must make sure it's idempotent.
Closing thoughts:
Please remember that stackoverlfow is monitored on a best-effort basis and what can be perceived as hasty comments may get in the way of getting helpful answers to your questions
I currently have a process running that should call a method every 10 seconds. I see that it actually calls the method at that interval, but it seems to not execute something in the code. Weird thing is, is that when I cancel the loop, and start it new it does actually do it the first time. Then when I keep it running it does not do anything.
def main():
try:
while True:
read()
time.sleep(10)
except KeyboardInterrupt:
pass
Above is the loop, and the code here is actually the beginning of the method that is being called, and I found out that it does not actually get results in the results, while the file has changed. In this case it gets data from a .json file
def read():
message = Query()
results = DB.search(message.pushed == False)
Am I overlooking something?
Solved. I had the DB declared globally and that did not go so well. It is being fixed by declaring it just before the statement.
I am constructing a model that does large parts of its calculations in a Postgresql database (for performance reasons). It looks somewhat like this:
def sql_func1(conn):
# prepare some data, crunch some number, etc.
curs = conn.cursor()
curs.execute("SOME SQL COMMAND")
curs.commit()
curs.close()
if __name__ == "__main__":
connection = psycopg2.connect(dbname='name', user='user', password='pass', host='localhost', port=1234)
sql_func1(conn)
sql_func2(conn)
sql_func3(conn)
connection.close()
The script uses around 30 individual functions like sql_func1. Obviously it is a little awkward to manage the connection and cursor in each function all the time. Thus I started using a decorator as described here. Now I can simply wrap sql_func1 with a decorator #db_connect and pass the connection from there. However, that means I am opening and closing the connection all the time, which is not good practice either. The psycopg2 FAQ says:
Creating a connection can be slow (think of SSL over TCP) so the best
practice is to create a single connection and keep it open as long as
required. It is also good practice to rollback or commit frequently
(even after a single SELECT statement) to make sure the backend is
never left “idle in transaction”. See also psycopg2.pool for
lightweight connection pooling.
Could you please give me some insights which would be an ideal practice im my case. Should I rather use a decorator that passes the cursor object instead of the connection? If so, please provide a code sample for the decorator. As I am rather new to programming, please let me also know in case you think my overall approach is wrong.
What about storing the connection in a global variable without closing it in the finally block? Something like this (according to the example you linked):
cnn = None
def with_connection(f):
def with_connection_(*args, **kwargs):
global cnn
if not cnn:
cnn = psycopg.connect(DSN)
try:
rv = f(cnn, *args, **kwargs)
except Exception, e:
cnn.rollback()
raise
else:
cnn.commit() # or maybe not
return rv
return with_connection_
I have an extremely strange problem that has stumped me for about the past day.
import wx, gui, threading, time, pyodbc
from OrderNumEntry import entryID, plyID, stationID
from main import strPlyDB
from Op1Login import eID1
global eID1
class MainFrame1(gui.MainFrame1Op):
def __init__(self, parent):
gui.MainFrame1Op.__init__(self, parent)
self.m_textCtrl3.SetValue(eID1)
# print eID1
def checkClick1(self, event):
if threading.activeCount() < 2:
t1 = threading.Thread(target = self.checkAdvance1)
t1.setDaemon(True)
t1.start()
self.m_checkBox110.SetValue(True)
def checkAdvance1(self):
strODBC_conn = 'DRIVER={Microsoft Access Driver (*.mdb)};DBQ=%s' % strPlyDB
conn = pyodbc.connect(strODBC_conn)
cursor = conn.cursor()
# eID1 = '00839530'
print 'test1'
print eID1
print 'test2'
SQL = 'INSERT INTO tblRecord(EntryID, PlyID, StationID, EmployeeNum, BuyoffTime) values(?,?,?,?,?)'
cursor.execute(SQL, (entryID, plyID, stationID, eID1, time.strftime("%c")))
cursor.commit()
cursor.close()
conn.close()
What's going on is there's a primary Python file main.py that initializes my mainframe1 class, which references gui.py for all of its wxpython code to bring up the window. This file is simply the event handler. checkClick1 runs when I click on a check box in my window (creative name I know), which in turn starts a thread for checkAdvance1.
Everything works fine in the program except for one part. My eID1 variable works just fine in __init__, and textCtrl3 displays it. However, when I try and use it again in checkAdvance1, the program hangs.
To test it, I surrounded the print statement for eID1 with two test print statements. Only test1 is printed, and no error code is given, the program simply hangs. To make matters even more confusing, when I put a print eID1 statement in the __init__ function (it's commented out right now), the program runs just fine!
I'm not sure how variable functionality could be affected by a print statement, but it's happening. However, putting a print statement in to fix it seems like a shoddy workaround that ignores the actual problem. Also, the commented-out eID1 declaration is just an example to show you what the content of the variable looks like when the program is running.
EDIT:
I have come up with a work around that I'm not entirely satisfied with, but it's better than adding print. If I create another variable, I can set it to the string translation of empID1, like so: empID1 = str(eID1). I can then reference this value later in place of eID1. However, this still doesn't sit well with me, as according to everything I know and have researched, my original code SHOULD work. It makes me feel like there's something bad going on behind the scenes that could come back to bite me later.
I have searched for quite some time now and can't found a solution to my problem. We are using SQLAlchemy in conjunction with MySQL for our project and we encounter several time the dreaded error:
1213, 'Deadlock found when trying to get lock; try restarting transaction'.
We would like to try to restart the transaction at most three times in this case.
I have started to write a decorator that does this but i don't know how to save the session state before the fail and retry the same transaction after it ? (As SQLAlchemy requires a rollback whenever an exception is raised)
My work so far,
def retry_on_deadlock_decorator(func):
lock_messages_error = ['Deadlock found', 'Lock wait timeout exceeded']
#wraps(func)
def wrapper(*args, **kwargs):
attempt_count = 0
while attempt_count < settings.MAXIMUM_RETRY_ON_DEADLOCK:
try:
return func(*args, **kwargs)
except OperationalError as e:
if any(msg in e.message for msg in lock_messages_error) \
and attempt_count <= settings.MAXIMUM_RETRY_ON_DEADLOCK:
logger.error('Deadlock detected. Trying sql transaction once more. Attempts count: %s'
% (attempt_count + 1))
else:
raise
attempt_count += 1
return wrapper
You can't really do that with the Session from the outside. Session would have to support this internally. It would involve saving a lot of private state, so this may not be worth your time.
I completely ditched most ORM stuff in favour of the lower level SQLAlchemy Core interface. Using that (or even any dbapi interface) you can trivially use your retry_on_deadlock_decorator decorator (see question above) to make a retry-aware db.execute wrapper.
#retry_on_deadlock_decorator
def deadlock_safe_execute(db, stmt, *args, **kw):
return db.execute(stmt, *args, **kw)
And instead of
db.execute("UPDATE users SET active=0")
you do
deadlock_safe_execute(db, "UPDATE users SET active=0")
which will retry automatically if a deadlock happens.
Did you use code like this?
try:
Perform table transaction
break
except:
rollback
delay
try again to perform table transaction
The only way to truly handle deadlocks is to write your code to expect
them. This generally isn't very difficult if your database code is
well written. Often you can just put a try/catch around the query
execution logic and look for a deadlock when errors occur. If you
catch one, the normal thing to do is just attempt to execute the
failed query again.
Usefull links:
How to Cope with Deadlocks
Understanding Autocommit
Property MySQLConnection.autocommit