Instead of:
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
# an Engine, which the Session will use for connection
# resources
engine = create_engine('sqlite:///...')
# create session and add objects
with Session(engine) as session:
session.add(some_object)
session.add(some_other_object)
session.commit()
I create a sessionmaker (according to example in documentation, see bellow):
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# an Engine, which the Session will use for connection
# resources, typically in module scope
engine = create_engine('postgresql://scott:tiger#localhost/')
# a sessionmaker(), also in the same scope as the engine
Session = sessionmaker(engine)
# we can now construct a Session() without needing to pass the
# engine each time
with Session() as session:
session.add(some_object)
session.add(some_other_object)
session.commit()
Can I use the sessions from session maker in different threads (spawning multiple sessions at the same time)? In other words, is session maker thread safe object? If yes, can multiple sessions exists and read/write into same tables at the same time?
Furthermore, what is the advantage of of using 'scoped_session' - is it realated to problem of multiple sessions (one per thread)?:
# set up a scoped_session
from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker
session_factory = sessionmaker(bind=some_engine)
Session = scoped_session(session_factory)
# now all calls to Session() will create a thread-local session
some_session = Session()
# you can now use some_session to run multiple queries, etc.
# remember to close it when you're finished!
Session.remove()
Session objects are not thread-safe, but are thread-local. What I recommend using is sessionmaker instead of Session. It will yield a Session object every time you need it, thus not idling the database connection. I'd use the approach below.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, Session
DB_ENGINE = create_engine('sqlite:///...')
DB_SES_MAKER = sessionmaker(bind=DB_ENGINE)
def get_db():
db = DB_SES_MAKER()
try:
yield db
finally:
db.close()
Then call get_db whenever needed:
db = next(get_db())
Related
I'm working on an async FastAPI project and I want to connect to the database during tests. Coming from Django, my instinct was to create pytest fixtures that take care of creating/dropping the test database. However, I couldn't find much documentation on how to do this. The most complete instructions I could find were in this tutorial, but they don't work for me because they are all synchronous. I'm somewhat new to async development so I'm having trouble adapting the code to work async. This is what I have so far:
import pytest
from sqlalchemy.ext.asyncio import create_async_engine, session
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy_utils import database_exists, create_database
from fastapi.testclient import TestClient
from app.core.db import get_session
from app.main import app
Base = declarative_base()
#pytest.fixture(scope="session")
def db_engine():
default_db = (
"postgresql+asyncpg://postgres:postgres#postgres:5432/postgres"
)
test_db = "postgresql+asyncpg://postgres:postgres#postgres:5432/test"
engine = create_async_engine(default_db)
if not database_exists(test_db): # <- Getting error on this line
create_database(test_db)
Base.metadata.create_all(bind=engine)
yield engine
#pytest.fixture(scope="function")
def db(db_engine):
connection = db_engine.connect()
# begin a non-ORM transaction
connection.begin()
# bind an individual Session to the connection
Session = sessionmaker(bind=connection)
db = Session()
# db = Session(db_engine)
yield db
db.rollback()
connection.close()
#pytest.fixture(scope="function")
def client(db):
app.dependency_overrides[get_session] = lambda: db
PREFIX = "/api/v1/my-endpoint"
with TestClient(PREFIX, app) as c:
yield c
And this is the error I'm getting:
E sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/14/xd2s)
/usr/local/lib/python3.9/site-packages/sqlalchemy/util/_concurrency_py3k.py:67: MissingGreenlet
Any idea what I have to do to fix it?
You try to use sync engine with async session. Try to use:
from sqlalchemy.ext.asyncio import AsyncSession
Session = sessionmaker(bind= connection, class_=AsyncSession)
https://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.html
I am trying to convert my single threaded application to multi threaded application which uses database using SQLAlchemy. And I found that SQLAlchemy session is not thread safe. So we need to use scoped_session factory for thread safe db access.
Below is my input dataset
input_list = [data1, data2, data3, data4, data5]
Single thread application
from sqlalchemy.orm import sessionmaker, scoped_session
Session = sessionmaker(bind=engine_url)
for data in input_list:
def myfunction(data):
db_session = Session()
print(db_session)
# use db_session to query/store the data
When I try to convert it to multithreaded application
from sqlalchemy.orm import sessionmaker, scoped_session
Session = scoped_session(sessionmaker(bind=engine_url))
def myfunction(data):
db_session = Session()
print(db_session)
# use db_session to query/store the data
def myfunction_parallel():
with ThreadPool(4) as pool:
output = pool.map(myfunction, input_list)
In multithread variant, I am getting db_session as same object, but my expectation is that there should be a new session object created for each thread and the session should be different?
The scoped session registry registers session for each thread that requests one. This enables code to call db_session = Session() and get the expected session for the thread.
However it's application's responsibility to inform the session registry when a session is no longer required. The application does this by calling Session.remove(), as documented here:
The scoped_session.remove() method first calls Session.close() on the current Session, which has the effect of releasing any connection/transactional resources owned by the Session first, then discarding the Session itself. “Releasing” here means that connections are returned to their connection pool and any transactional state is rolled back, ultimately using the rollback() method of the underlying DBAPI connection.
At this point, the scoped_session object is “empty”, and will create a new Session when called again.
This code should work as expected:
def myfunction(data):
db_session = Session()
print(db_session)
# use db_session to query/store the data
Session.remove()
I'm utilizing Flask and SqlAlchemy. The database I've created for SqlAlchemy seems to mess up when I try to run my website and will pop up with the error stating that there's a thread error. I'm wondering if it's because I haven't dropped my table from my previous schema. I'm using a linux server to try and run the "python3" and the file to set up my database.
I've tried to physically delete the table from my local drive and the re run it but I still up this error.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import scoped_session
from database_setup import Base, Category, Item
engine = create_engine('sqlite:///database_tables.db')
Base.metadata.bind = engine
Session = sessionmaker()
Session.bind = engine
session = Session()
brushes = Category(id = 1, category_name = 'Brushes')
session.add(brushes)
session.commit()
pencils = Category(id = 2, category_name = 'Pencils')
session.add(pencils)
session.commit()
When I am in debug mode using Flask, I click the links I've made using these rows, but after three clicks I get the error
"(sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 140244909291264 and this is thread id 140244900898560 [SQL: SELECT category.id AS category_id, category.category_name AS category_category_name FROM category] [parameters: [{}]] (Background on this error at: http://sqlalche.me/e/f405)"
you can use for each thread a session, by indexing them using the thread id _thread.get_ident():
import _thread
engine = create_engine('sqlite:///history.db', connect_args={'check_same_thread': False})
...
Base.metadata.create_all(engine)
sessions = {}
def get_session():
thread_id = _thread.get_ident() # get thread id
if thread_id in sessions:
return sessions[thread_id]
session_factory = sessionmaker(bind=engine)
Session = scoped_session(session_factory)
sessions[thread_id] = Session()
return sessions[thread_id]
then use get_session() where it is needed, in your case:
get_session().add(brushes)
get_session().commit()
I have a code that runs a query from a query list. These query are long and take quite a long time to execute. Since I am executing these query in a loop, the session seems to expire and I get a error telling me that the connection to the server was lost.
Then I created the session as well as engine inside the loop (I closed the session and disposed the engine at the end of the loop.) I have understood that creating new connection is an expensive operation.
How can I re-use the connection in this case so that I do not have to create the session and engine each time?
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# an Engine, which the Session will use for connection
# resources
some_engine = create_engine('mysql://user:password#localhost/')
# create a configured "Session" class
Session = sessionmaker(bind=some_engine)
# create a Session
session = Session()
for long_query in long_query_list:
# work with sess
session.execute(long_query)
session.commit()
I'm getting this error sometime (sometime is ok, sometime is wrong):
sqlalchemy.exc.OperationalError: (OperationalError) MySQL Connection not available.
while using session.query
I'm writing a simple server with Flask and SQLAlchemy (MySQL). My app.py like this:
Session = sessionmaker(bind=engine)
session = Session()
#app.route('/foo')
def foo():
try:
session.query(Foo).all()
except Exception:
session.rollback()
Update
I also create new session in another file and call it in app.py
Session = sessionmaker(bind=engine)
session = Session()
def foo_helper(): #call in app.py
session.query(Something).all()
Update 2
My engine:
engine = create_engine('path')
How can I avoid that error?
Thank you!
Make sure the value of ‘pool_recycle option’ is less than your MYSQLs wait_timeout value when using SQLAlchemy ‘create_engine’ function.
engine = create_engine("mysql://username:password#localhost/myDatabase", pool_recycle=3600)
Try to use scoped_session to make your session:
from sqlalchemy.orm import scoped_session, sessionmaker
session = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=engine))
and close/remove your session after retrieving your data.
session.query(Foo).all()
session.close()