We Keep Coding

Front-end application database context with Peewee supporting pooling and automatic reconnection?

I have a data model defined with Peewee and it is all good. Now I need to bookkeep some form of database context in an xlwings front end application but I'm not 100% sure how to proceed. I see couple of bits and pieces of information here and there but not 100% sure how to:
https://docs.peewee-orm.com/en/2.10.2/peewee/database.html#connection-pooling
https://docs.peewee-orm.com/en/2.10.2/peewee/database.html#automatic-reconnect
I'm already using a DatabaseProxy to deffer tying the model to a specific connection until runtime as I can target different vendors e.g. PROD -> Postgres and UT -> SQLite:
database_proxy = DatabaseProxy()
Furthermore, I use playhouse.db_url's connect way with a fully qualified URL. I need a "Database Context" that:
Works with DatabaseProxy and playhouse.db_url.connect and can support at least SQLite and Postgres.
Keeps a pool of N connections which are managed internally and if the connection is lost then a retry is attempted until a timeout.
Indeed, I need a connect timeout instead of keep waiting forever to fail.
The connection pooling part: automatic recycling and closing of idle connections.
Robust automatic retrial on OperationalError due to short database outages.
I struggle to put all the pieces together here, for example, the playhouse.db_url.connect doesn't have retry or timeout arguments, it is also not clear how to decorate such connection with pooling and so on.

Difference between connecting to the database in app.on_event('startup') vs in a dependency in FastAPI

In Tiangolo's FastAPI, it states that you can create a persistent database connection by using a dependency
https://fastapi.tiangolo.com/tutorial/sql-databases/#create-a-dependency
However, in the Async database docs, the database is connected to in app startup
https://fastapi.tiangolo.com/advanced/async-sql-databases/#connect-and-disconnect
This same pattern is followed in the encode/databases docs
https://www.encode.io/databases/connections_and_transactions/
Which is the right pattern? It seems to me that using dependencies, one database connection would be created per API call, while connecting the the database during startup would establish one database connection per worker. If this is correct, connecting to the database on startup would be far superior.
What the difference between the two and is one better?

I won't go into the details of each database library. Let me just say that most modern tools use a connection pool. They do it explicitly or implicitly, hiding behind certain abstractions like Session in your first link.
In all of your examples, the connection pool is created when the application starts. And when creating a session, there is no heavy operation of establishing a new connection, but only a connection is fetched from the pool, and when the session is closed, the connection is returned to the pool.

Unable scale up cassandra using latest datastax python driver

We are using multiple cassandra datastax cluster instances(6) to connect to cassandra using python. We are pooling these multiple connections to do some operations. Each operation is independent of other.
It works fine on a small number of operations, but once I try to scale up I get the following errors :
NoHostAvailable: ('Unable to complete the operation against any hosts', {<Host: 127.ption('Pool is shutdown',)})
and sometimes the following warning:
WARNING Heartbeat failed for connection (140414695068880) to 127.0.0.1
I tried changing some cluster object parameters but it did not help.
Following is the configuration of key space in cassandra I am using :
'class': 'SimpleStrategy',
'replication_factor': '1'
I am using lastest versions of cassandra and datastax driver for python. There is only one node is cassandra.
EDIT: More details:
The multiple cluster instances are in different processes (processes are created using the Python multiprocessing module) - one cluster instance per process. Lets call the proccesses Cassandra-Processes (CP). There are a bunch of other process that do some computation and need to look up a Cassandra DB, and write to it, occassionally. The current design is that each of these processes is mapped to one CP, and all DB reads/writes to be done by the process is done via this mapped CP. 'what' exactly is to be read/written is passed into a queue (again from the multiprocessing library) which the mapped CP reads.
We observe that this setup runs for quite sometime - and then suddenly Cassandra begins erroring out.

It's unclear why you're using six cluster instances against a single Cassandra node. Generally, you should use one Cluster instance per application (per remote cluster). You can read about general design considerations for Cassandra drivers here
If you're looking to "scale" with regards to throughput, you might consider using multiprocessing. I discuss this in a blog post here.
Follow-on:
Two things can be inferred from the information we have so far:
The application is pushing more concurrent requests than your connection pool is configured to handle. I say this because the "Pool is shutdown" only occurs when a request is waiting for a connection/stream to become available. You can tune connection pooling to make more available initially using cluster settings. However, if your "cluster" (server node) is overwhelmed, you won't gain much there.
Your connection is being shutdown. This exception only happens when the node is suddenly marked down. In a single node setup this is most likely because of a connection error. Look for clues in the server log, or driver debug log if you're capturing that.
We probably need to know more about your execution model to help more. Is it possible you're running unfettered async requests without occasionally waiting for them to complete?

Remote diagnosis is hard to do without knowing anything on your specific topology, setup and system configuration. This however looks much like a configuration problem or even the python driver. If you google your error message you will find multiple topics on Datastax's Jira describing this or similar problems, I would check that the Python Driver is up to date.
What would help in the first place is to see in detail what you try to do, how your cluster is configured aso.

SQLAlchemy session reconnect

How can I force my engine to reconnect if a query returns an OperationalError like user does not have access to the database or something like that?
engine = create_engine(url, pool_recycle=3600)
Session = sessionmaker(bind=engine)
try:
sesh = Session()
sesh.query....
sesh.close()
except OperationalError:
# force engine to reconnect here somehow?

If you catch an error that indicates the connection was closed during an operation, SQLAlchemy automatically reconnects on the next access. However, when a database disconnects, your transaction is gone, so SQLAlchemy requires that you emit rollback() on the Session in order to establish within your application that a new transaction is to take place. you then need to start your whole transaction all over again.
Dealing with that issue has a few angles. You should read through the Dealing with Disconnects section of the documentation which illustrates two ways to work with disconnects. Beyond that, if you truly wanted to pick up your transaction from where you left off, you'd need to "replay" the whole thing back, assuming you've done more than one thing in your transaction. This is best suited by application code that packages what it needs to do in a function that can be called again. Note that a future version of SQLAlchemy may introduce an extension called the Transaction Replay Extension that provides another way of doing this, however it will have lots of caveats, as replaying a lost transaction in a generic way is not a trivial affair.

A lot have happened since this question was first answered.
By taking a pessimistic error handling approach you get the most bang for the buck - easy implementation and very effective.
Apply the pool_pre_ping=Truewhen you create the engine, like this:
engine = create_engine("mysql+pymysql://user:pw#host/db", pool_pre_ping=True)
See more: docs.sqlalchemy.org/en/latest/core/pooling.html#pool-disconnects-pessimistic
Another approach is to deal with errors in an optimistic way - as they happen. In that case you can wrap the execute statement in a try and except and the invalidate the connection if an exception is raised. Once the connection is invalidated you and re-instantiate it.
See more: docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-optimistic
Both approaches works great in situations where your connection otherwise would timeout e.g. overnight / weekend. It also makes it much easier for IT operations to take a database down and not have to worry too much about downstream applications relying on a restart. How ever this is not a silver bullet, it's worth thinking about secure transaction handling (as mentioned by zzzeek) if you deal with very critical transactions.

If you are using sqlalchemy in Flask, when you query like
MyModel.query.all()
and you get errors like
File "./lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 427, in _revalidate_connection
"Can't reconnect until invalid "
StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
You can simply reconnect DB by
MyModel.query.session.close()
MyModel.query.all()

How to manage db connections using webpy with SQLObject?

Web.py has its own database API, web.db. It's possible to use SQLObject instead, but I haven't been able to find documentation describing how to do this properly. I'm especially interested in managing database connections. It would be best to establish a connection at the wsgi entry point, and reuse it. Webpy cookbook contains an example how to do this with SQLAlchemy. I'd be interested to see how to properly do a similar thing using SQLObject.
This is how I currently do it:
class MyPage(object):
def GET(self):
ConnectToDatabase()
....
return render.MyPage(...)
This is obviously inefficient, because it establishes a new database connection on each query. I'm sure there's a better way.

As far as I understand the SQLAlchemy example given, a processor is used, that is, a session is created for each connection and committed when the handler is complete (or rolled back if an error has occurred).
I don't see any simple way to do what you propose, i.e. open a connection at the WSGI entry point. You will probably need a connection pool to serve multiple clients at the same time. (I have no idea what are the requirements for efficiency, code simplicity and so on, though. Please comment.)
Inserting ConnectToDatabase calls into each handler is of course ugly. I suggest that you adapt the cookbook example replacing the SQLAlchemy session with a SQLObject connection.