We Keep Coding

Turbogears2 with MySQL db not update tables in multi threads

I am using turbogears2 with MySQL db. With the same code, single thread case can update/write to the tables. But thread thread has no error, however, no write is successful.
Outside turbogears2, multi threads can write to the tables no problems.
No error or complaints with multi thread with tg2. Just no successful write to the table.
I will be very grateful if anyone using tg2 can advise.

With default configuration settings, in a regular request/response cycle, TuborGears2 enables a transaction manager to automatically commit changes to the database when a controller has finished processing a request.
This is introduced in the Wiki in 20 Minutes tutorial:
[...] you would usually need to flush the SQLAlchemy Unit of Work and commit the currently running transaction, those are operations that TurboGears2 transaction management will automatically do for us.
You don’t have to do anything to use this transaction management system, it should just work.
However, for everything that is outside a regular request/response cycle, for example a stream, or a different thread like a scheduler, manually flushing the session and committing the transaction is required. This is performed with DBSession.flush() and transaction.commit().

Is it possible to check for db updates before performing db.rollback() within a worker transaction?

I'm using a queue to queue transactions for a MySQL database in python. I'm using autocommit=False so that I can rollback a transaction if not all of the data queries are executed properly.
I'm wondering whether it's possible to check whether a worker has performed an action on the database before performing db.rollback()? Or can you perform db.rollback even if a worker hasn't done anything to the database without any errors occurring?
Thanks!

If I've understood the question correctly, it is not an error to rollback a transaction where nothing else has happened, although it is an error to attempt to rollback when no transaction exists. To tell which is the case you can test the MySQLConnection.in_transaction property.

If you issue a
START TRANSACTION;
(which brings the connection into the same as autocommit = false), sending the command
ROLLBACK WORK;
(see also https://dev.mysql.com/doc/refman/5.5/en/commit.html )
returns with
0 row(s) affected
So, this means, you may rollback an empty transaction safely (tested on MySQL 5.1.49-3).
Alternatively, you always may store in a worker a local/static variable, which is initialized with false. If at least a single update command has been issued by the worker, you set this variable to true.
If you have to decide later on whether to ROLLBACK or not, you just look at the variable: if it true, you know that you have sent a least one update command through the DB channel.

Python (PostgreSQL) wrap local and remote calls in one transaction

I am doing synchronization between two databases in Odoo. If it goes without any issues on remote, then it synchronizes on both sides. But if something goes wrong on remote, then local database changes are committed, but remote is not.
In other words, databases go out of sync.
Is there a way to make changes in local database and if something goes wrong trying to synchronize remote database, rollback local database to previous state.
There is this method:
#api.one
def order_process_now(self):
servers = self._synchro_before_sale()
# Process local order
inv_id = self.action_invoice_create()
if inv_id:
inv = self.env['account.invoice'].search([('id', '=', inv_id)])
inv.signal_workflow('invoice_open')
for picking in self.picking_ids:
picking.force_assign()
picking.action_done()
# Process remote orders
self._remote_order_action('order_process_now', servers)
As you can see it is divided into two parts. First it makes changes to local database, then makes changes on remote (using xmlrpclib with erppeek wrapper).
How can I make this method as one transaction, so if anything goes wrong executing method, any changes to databases would rollback?

What you need for this is two-phase commit.
The general idea is:
Begin your local and remote changes
Do the required work on each
On the remote side PREPARE TRANSACTION and take note of the returned ID in persistent storage
On the local side COMMIT the changes
On the remote side COMMIT PREPARED with the returned ID, or if the local commit failed for some reason, ROLLBACK PREPARED instead.
If your app restarts it must look at its record of of prepared-but-not-committed remote transactions and:
if the local transaction was committed) issue a COMMIT PREPARED; or
if the local transaction was NOT committed issue a ROLLBACK PREPARED
This is not simple to get right. The naïve approach that fails to record the local commit ID doesn't really fix anything, it just replaces inconsistent database state with leaked prepared transactions. You must actually keep a record of the prepared transactions and resolve them after a crash or restart. Remember that the ROLLBACK PREPARED or COMMIT PREPARED can fail due to connectivity issues, DB restarts, etc.
For this reason many people use a separate transaction manager that takes care of this part for them. MSDTC is an option on Windows systems. For Java you can supposedly use JTC. On C/UNIX systems you could use XA. Unfortunately distributed transaction managers appear to attract horrible, baroque and ill-defined API design (can you say javax.transaction.HeuristicMixedException?)

You'll need to look at two phase commits. Basically this lets you do a trial commit on each separate system and then only if both succeed do a final "real" commit.
You still need to deal with the case where e.g. the client crashes. Then you'll have prepared commits hanging about and you'll want to roll them back and start again.

SSL syscall error bad file descriptor using sqlalchemy and postgres

So I have a daemon process that talks to Postgres via sqlalchemy. The daemon does something like this:
while True:
oEngine = setup_new_engine()
with oEngine.connect() as conn:
Logger.debug("connection established")
DBSession = sessionmaker(bind=conn)()
Logger.debug('DBSession created. id={0}'.format(id(DBSession)))
#do a bunch of stuff with DBSession
DBSession.commit()
Logger.debug('DBSession committed. id={0}'.format(id(DBSession)))
On the first iteration of the forever loop everything works great. For a while. The DBSession successfully makes a few queries to the database. But then one query fails with the error:
OperationalError: (OperationalError) SSL SYSCALL error: Bad file descriptor
This speaks to me of a closed connection or file descriptor being used. But the connections are created and maintained by the daemon so I don't know what this means.
In other words what happens is:
create engine
open connection
setup dbsession
query dbsession => works great
query dbsession => ERROR
The query in question looks like:
DBSession.query(Login)
.filter(Login.LFTime == oLineTime)
.filter(Login.success == self.success)
.count()
which seems perfectly reasonable to me.
My question is: What kinds of reasons could there be for this kind of behaviour and how can I fix it or isolate the problem?
Let me know if you need more code. There is a heck of a lot of it so I went for the minimalist approach here...

I fixed this by thinking about the session scope instead of the transaction scope.
while True:
do_stuff()
def do_stuff():
oEngine = setup_new_engine()
with oEngine.connect() as conn:
Logger.debug("connection established")
DBSession = sessionmaker(bind=conn)()
#do a bunch of stuff with DBSession
DBSession.commit()
DBSession.close()
I would still like to know why this fixed things though...

You are creating the session inside your while loop, which is very ill-advised. With the code the way you had it the first time, you would spawn off a new connection at every iteration and leave it open. Before too long, you would be bound to hit some kind of limit and be unable to open yet another new session. (What kind of limit? Hard to say, but it could be a memory condition since DB connections are pretty weighty; it could be a DB-server limit where it will only accept a certain number of simultaneous user connections for performance reasons; hard to know and it doesn't really matter, because whatever the limit was, it has prevented you from using a very wasteful approach and hence has worked as intended!)
The solution you have hit upon fixes the problem because, as you open a new connection with each loop, so you also close it with each loop, freeing up the resources and allowing additional loops to create sessions of their own and succeed. However, this is still a lot of unnecessary busyness and a waste of processing resources on both the server and the client. I suspect it would work just as well-- and potentially be a lot faster-- if you move the sessionmaker outside the while loop.
def main():
oEngine = setup_new_engine()
with oEngine.connect() as conn:
Logger.debug("connection established")
DBSession = sessionmaker(bind=conn)()
apparently_infinite_loop(DBSession)
# close only after we are done and have somehow exited the infinite loop
DBSession.close()
def apparently_infinite_loop(DBSession):
while True:
#do a bunch of stuff with DBSession
DBSession.commit()
I don't currently have a working sqlalchemy setup, so you likely have some syntax errors in there, but anyway I hope it makes the point about the fundamental underlying issue.
More detail is available here: http://docs.sqlalchemy.org/en/rel_0_9/orm/session.html#session-faq-whentocreate
Some points from the docs to note:
"The Session will begin a new transaction if it is used again". So this is why you don't need to be constantly opening new sessions in order to get transaction scope; a commit is all it takes.
"As a general rule, the application should manage the lifecycle of the session externally to functions that deal with specific data." So your fundamental problem originally (and still) is all of that session management going on right down there inside the while loop right alongside your data processing code.

Mysql Connection, one or many?

I'm writing a script in python which basically queries WMI and updates the information in a mysql database. One of those "write something you need" to learn to program exercises.
In case something breaks in the middle of the script, for example, the remote computer turns off, it's separated out into functions.
Query Some WMI data
Update that to the database
Query Other WMI data
Update that to the database
Is it better to open one mysql connection at the beginning and leave it open or close the connection after each update?
It seems as though one connection would use less resources. (Although I'm just learning, so this is a complete guess.) However, opening and closing the connection with each update seems more 'neat'. Functions would be more stand alone, rather than depend on code outside that function.

"However, opening and closing the connection with each update seems more 'neat'. "
It's also a huge amount of overhead -- and there's no actual benefit.
Creating and disposing of connections is relatively expensive. More importantly, what's the actual reason? How does it improve, simplify, clarify?
Generally, most applications have one connection that they use from when they start to when they stop.

I don't think that there is "better" solution. Its too early to think about resources. And since wmi is quite slow ( in comparison to sql connection ) the db is not an issue.
Just make it work. And then make it better.
The good thing about working with open connection here, is that the "natural" solution is to use objects and not just functions. So it will be a learning experience( In case you are learning python and not mysql).

Think for a moment about the following scenario:
for dataItem in dataSet:
update(dataItem)
If you open and close your connection inside of the update function and your dataSet contains a thousand items then you will destroy the performance of your application and ruin any transactional capabilities.
A better way would be to open a connection and pass it to the update function. You could even have your update function call a connection manager of sorts. If you intend to perform single updates periodically then open and close your connection around your update function calls.
In this way you will be able to use functions to encapsulate your data operations and be able to share a connection between them.
However, this approach is not great for performing bulk inserts or updates.

Useful clues in S.Lott's and Igal Serban's answers. I think you should first find out your actual requirements and code accordingly.
Just to mention a different strategy; some applications keep a pool of database (or whatever) connections and in case of a transaction just pull one from that pool. It seems rather obvious you just need one connection for this kind of application. But you can still keep a pool of one connection and apply following;
Whenever database transaction is needed the connection is pulled from the pool and returned back at the end.
(optional) The connection is expired (and of replaced by a new one) after a certain amount of time.
(optional) The connection is expired after a certain amount of usage.
(optional) The pool can check (by sending an inexpensive query) if the connection is alive before handing it over the program.
This is somewhat in between single connection and connection per transaction strategies.