I'm getting this error sometime (sometime is ok, sometime is wrong):
sqlalchemy.exc.OperationalError: (OperationalError) MySQL Connection not available.
while using session.query
I'm writing a simple server with Flask and SQLAlchemy (MySQL). My app.py like this:
Session = sessionmaker(bind=engine)
session = Session()
#app.route('/foo')
def foo():
try:
session.query(Foo).all()
except Exception:
session.rollback()
Update
I also create new session in another file and call it in app.py
Session = sessionmaker(bind=engine)
session = Session()
def foo_helper(): #call in app.py
session.query(Something).all()
Update 2
My engine:
engine = create_engine('path')
How can I avoid that error?
Thank you!
Make sure the value of ‘pool_recycle option’ is less than your MYSQLs wait_timeout value when using SQLAlchemy ‘create_engine’ function.
engine = create_engine("mysql://username:password#localhost/myDatabase", pool_recycle=3600)
Try to use scoped_session to make your session:
from sqlalchemy.orm import scoped_session, sessionmaker
session = scoped_session(sessionmaker(autocommit=False, autoflush=False, bind=engine))
and close/remove your session after retrieving your data.
session.query(Foo).all()
session.close()
Related
Instead of:
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
# an Engine, which the Session will use for connection
# resources
engine = create_engine('sqlite:///...')
# create session and add objects
with Session(engine) as session:
session.add(some_object)
session.add(some_other_object)
session.commit()
I create a sessionmaker (according to example in documentation, see bellow):
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# an Engine, which the Session will use for connection
# resources, typically in module scope
engine = create_engine('postgresql://scott:tiger#localhost/')
# a sessionmaker(), also in the same scope as the engine
Session = sessionmaker(engine)
# we can now construct a Session() without needing to pass the
# engine each time
with Session() as session:
session.add(some_object)
session.add(some_other_object)
session.commit()
Can I use the sessions from session maker in different threads (spawning multiple sessions at the same time)? In other words, is session maker thread safe object? If yes, can multiple sessions exists and read/write into same tables at the same time?
Furthermore, what is the advantage of of using 'scoped_session' - is it realated to problem of multiple sessions (one per thread)?:
# set up a scoped_session
from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker
session_factory = sessionmaker(bind=some_engine)
Session = scoped_session(session_factory)
# now all calls to Session() will create a thread-local session
some_session = Session()
# you can now use some_session to run multiple queries, etc.
# remember to close it when you're finished!
Session.remove()
Session objects are not thread-safe, but are thread-local. What I recommend using is sessionmaker instead of Session. It will yield a Session object every time you need it, thus not idling the database connection. I'd use the approach below.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, Session
DB_ENGINE = create_engine('sqlite:///...')
DB_SES_MAKER = sessionmaker(bind=DB_ENGINE)
def get_db():
db = DB_SES_MAKER()
try:
yield db
finally:
db.close()
Then call get_db whenever needed:
db = next(get_db())
I have this database connection in a fast API app.
I am using multiple binds to the same session, I followed the documentation here:
Query the db:
method1: working
db.query(Model).all()
method2: not working and throwing the following error:
db.execute("SELECT * from ...");
Exception has occurred: UnboundExecutionError
Could not locate a bind configured on SQL expression or this Session.
This is the database connection code...
Can you help me get method2 working?
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.automap import automap_base
from sqlalchemy import MetaData
from app.settings import (
db1_DATABASE_URL,
db2_DATABASE_URL
)
engine = create_engine(
db1_DATABASE_URL,
echo=False,
pool_recycle=1800,
pool_timeout=20,
pool_pre_ping=True,
pool_size=50,
max_overflow=100,
)
engine2 = create_engine(db2_DATABASE_URL)
Base = declarative_base()
mroi_Base = automap_base()
mroi_Base.prepare(engine2, reflect=True)
SessionLocal = sessionmaker(autocommit=False, autoflush=False)
SessionLocal.configure(binds={Base: engine, mroi_Base: engine2})
def get_db():
db = None
try:
db = SessionLocal()
yield db
finally:
db.close()
def get_db2():
db2 = None
try:
db2 = SessionLocal()
yield db2
finally:
db2.close()
Remove the below mentioned line as we would be configuring the engine while creating the respective session instances.
SessionLocal.configure(binds={Base: engine, mroi_Base: engine2})
Initialize the session class with respective engine values.
def get_db():
db = None
try:
db = SessionLocal(engine=engine)
yield db
finally:
db.close()
def get_db2():
db2 = None
try:
db2 = SessionLocal(engine=engine2)
yield db2
finally:
db2.close()
I'm utilizing Flask and SqlAlchemy. The database I've created for SqlAlchemy seems to mess up when I try to run my website and will pop up with the error stating that there's a thread error. I'm wondering if it's because I haven't dropped my table from my previous schema. I'm using a linux server to try and run the "python3" and the file to set up my database.
I've tried to physically delete the table from my local drive and the re run it but I still up this error.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import scoped_session
from database_setup import Base, Category, Item
engine = create_engine('sqlite:///database_tables.db')
Base.metadata.bind = engine
Session = sessionmaker()
Session.bind = engine
session = Session()
brushes = Category(id = 1, category_name = 'Brushes')
session.add(brushes)
session.commit()
pencils = Category(id = 2, category_name = 'Pencils')
session.add(pencils)
session.commit()
When I am in debug mode using Flask, I click the links I've made using these rows, but after three clicks I get the error
"(sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 140244909291264 and this is thread id 140244900898560 [SQL: SELECT category.id AS category_id, category.category_name AS category_category_name FROM category] [parameters: [{}]] (Background on this error at: http://sqlalche.me/e/f405)"
you can use for each thread a session, by indexing them using the thread id _thread.get_ident():
import _thread
engine = create_engine('sqlite:///history.db', connect_args={'check_same_thread': False})
...
Base.metadata.create_all(engine)
sessions = {}
def get_session():
thread_id = _thread.get_ident() # get thread id
if thread_id in sessions:
return sessions[thread_id]
session_factory = sessionmaker(bind=engine)
Session = scoped_session(session_factory)
sessions[thread_id] = Session()
return sessions[thread_id]
then use get_session() where it is needed, in your case:
get_session().add(brushes)
get_session().commit()
I'm totally new using sqlalchemy and postgresql. I read this tutorial to build the following piece of code :
import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy import engine
def connect(user, password, db, host='localhost', port=5432):
'''Returns a connection and a metadata object'''
# We connect with the help of the PostgreSQL URL
# postgresql://federer:grandestslam#localhost:5432/tennis
url = 'postgresql://{}:{}#{}:{}/{}'
url = url.format(user, password, host, port, db)
# The return value of create_engine() is our connection object
con = sqlalchemy.create_engine(url, client_encoding='utf8')
# We then bind the connection to MetaData()
meta = sqlalchemy.MetaData(bind=con, reflect=True)
return con, meta
con, meta = connect('federer', 'grandestslam', 'tennis')
con
engine('postgresql://federer:***#localhost:5432/tennis')
meta
MetaData(bind=Engine('postgresql://federer:***#localhost:5432/tennis'))
When running it I have this error :
File "test.py", line 22, in <module>
engine('postgresql://federer:***#localhost:5432/tennis')
TypeError: 'module' object is not callable
what should I do ? thanks !
So, your problem is happening because you've made this call:
from sqlalchemy import engine
And then you've used this later in the file:
engine('postgresql://federer:***#localhost:5432/tennis')
Strangely, in that section, you have some statements that are just con and meta with no assignments or calls or anything. I'm not sure what you're doing there. I would suggest that you check out SQLalchemy's page on engine and connection use to help get you sorted.
It will of course depend on exactly how you've set up your database. I used the declarative_base module in one of my projects, so my process of setting up a session to connect to my DB looks like this:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# Connect to Database and create database session
engine = create_engine('postgresql://catalog:catalog#localhost/menus')
Base.metadata.bind = engine
DBSession = sessionmaker(bind=engine)
session = DBSession()
And in my database setup file, I've assigned:
Base = declarative_base()
But you'll have to customize it a bit to your particular setup. I hope that helps.
Edit: I see now where those calls to con and meta were coming from, as well as your other confusing lines, it's part of the tutorial you linked to. What he was doing in that tutorial was using the Python interpreter in command line. I'll explain a few of the things he did there in the hope that it helps you some more. Lines beginning with >>> are what he enters in as commands. The other lines are the output he receives back.
>>> con, meta = connect('federer', 'grandestslam', 'tennis') # he creates the connection and meta objects
>>> con # now he calls the connection by itself to have it show that it's connected to his DB
Engine(postgresql://federer:***#localhost:5432/tennis)
>>> meta # here he calls his meta object to show how it, too, is connected
MetaData(bind=Engine(postgresql://federer:***#localhost:5432/tennis))
Im writting a flask api using flaskrestful,sqlalchemy, Postgres, nginx,uwsgi. Im a newbie to python.These are my configuration
database.py
from cuewords import app
from flask.ext.sqlalchemy import SQLAlchemy
from sqlalchemy.pool import NullPool
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
from sqlalchemy import Column, Integer, String , Text , Boolean , DateTime, MetaData, Table ,Sequence
from sqlalchemy.dialects.postgresql import JSON
Base = declarative_base()
db_name="postgresql+psycopg2://admin:password#localhost:5434/database_name"
from sqlalchemy.orm import sessionmaker
engine = create_engine(db_name,poolclass=NullPool ,echo=True)
Session = sessionmaker(autocommit=False ,bind=engine)
connection = engine.connect()
metadata = MetaData()
api.py
class Webcontent(Resource):
def post(self):
session=Session()
...assign some params...
try:
insert_data=Websitecontent(site_url,publisher_id)
session.add(insert_data)
session.commit()
Websitecontent.update_url(insert_data.id,session)
except:
session.rollback()
raise
finally:
return "Data created "
session.close()
else:
return "some value"
Here im first saving the just the url then saving all the content of the site using boilerpipe later .Idea is to move to queue later
model.py
class Websitecontent(Base):
#classmethod
def update_url(self,id,session):
existing_record=session.query(Websitecontent).filter_by(id=int(id)).first()
data=Processing.processingContent(str(existing_record.url))
#boilerpipe processing the content here
#assigning some data to existing record in session
session.add(existing_record)
session.commit()
Websitecontent.processingWords(existing_record,session)
#classmethod
def processingWords(self,record,session)
...processing
Websitecontent.saveKeywordMapping(session,keyword_url,record)
#classmethod
def saveKeywordMapping(session,keyword_url,record)
session.commit()
session.close()
So this code works perfectly in locally but its doesnt work in production .So when i check pag_stat_activity it show the state "idle in transaction". The app hangs then i have to restart the servers. i dont get it why session.close() does not close the pool connection why its keeping psql transaction state busy . Guys any help would be really appreciated.
You are returning before closing the session:
return "Data created "
session.close()
I think returning inside finally might swallow the exception, as well.