We Keep Coding

Refresh the data within an sqlalchemy table model quicker

Due to complications with how Django and SQLAlchemy deal with JSON data, I've had to create a system that uses models from both of them as well as using standard SQLAlchemy.
The problem I'm having is that when I update information within a table via the table.update() method there's quite a considerable delay until my SQLAlchemy table model picks up the change.
Is there a way to force the model to update?
My code is along these lines:
# Database Connection
engine = create_engine('mysql+pymysql://'+dbusername+':'+dbuserpass+dbserver+dbname,
pool_recycle=3600, echo=False)
con = scoped_session(sessionmaker(autocommit=True,autoflush=False,bind=engine))
Session = sessionmaker(bind=engine)
sess = Session()
meta = MetaData(engine)
insp = inspect(engine)
Base = declarative_base()
con.close()
engine.dispose()
# sqlalchemy table model
class ContactsTable(Base):
__tablename__ = 'contacts_tbl'
db_id = Column(Integer, primary_key=True)
per_contact_id = Column(JSON)
createdDateTime = Column(JSON)
lastModifiedDateTime = Column(JSON)
distlists = Column(JSON)
# Theres a lot of code missing here you can see the basics of what I'm doing, adding data and then reading
def add_to_dist(contact,dist,tbl=contacts_tbl):
con.execute(tbl.update().values(distributionLists=dists).where(tbl.c.per_contact_id==contact))
def get_dist_members(name):
data = sess.query(ContactsTable).filter(ContactsTable.distributionLists.contains(name)).all()
Everything works. It's just that the query data is out of date and seems to take anywhere up to 10 minutes to refresh. This is annoying as it's running through a web page that displays the data. It really needs to reflect the changes instantly.
If in fact I'm doing this whole thing incorrectly then feel free to school me!

Solved it by adding isolation_level="READ UNCOMMITTED" to the engine:
engine = create_engine('mysql+pymysql://'+dbusername+':'+dbuserpass+dbserver+dbname, pool_recycle=3600, echo=False, isolation_level="READ UNCOMMITTED")
Thanks to Ilja for pointing me in the right direction.

Sqlalchemy with Flask - deleted data is still shown

Description
I have an Flask application with original SQLalchemy. Application is intended to be used internally in a company for easier saving of measurement data with MySQL
On one page I have a table with all devices used for measurement and a form that is used to add, remove or modify measurement devices.
Problem
The problem is that when I enter a new device in the database, the page is automatically refreshed to fetch new data from DB and new device is sometimes shown and sometimes it is not when I refresh the page. In other words, added row in table is appearing and dissapearing even though the row is visible on database. Same goes when i try to delete the device from database. The row is sometimes shown, sometimes not when refreshing the page with row being deleted from DB.
The same problem appears for all examples similar to this one (adding, deleting and modifying data).
What i have tried
Bellow is the code for table model:
class DvDevice(Base):
__tablename__ = "dvdevice"
id = Column("device_id", Integer, primary_key=True, autoincrement=True)
name = Column("device_name", String(50), nullable=True)
code = Column("device_code", String(10), nullable=True, unique=True)
hw_ver = Column("hw_ver", String(10), nullable=True)
fw_ver = Column("fw_ver", String(10), nullable=True)
sw_ver = Column("sw_ver", String(10), nullable=True)
And here is the code that inserts/deletes data from table.
#Insertion
device = DvDevice()
device.code = self.device_code
device.name = self.device_name
device.hw_ver = self.hw_ver
device.fw_ver = self.fw_ver
device.sw_ver = self.sw_ver
ses.add(device)
ses.commit()
ses.expire_all() #Should this be here?
# Deletion
ses.query(DvDevice).filter_by(id=self.device_id).delete()
ses.commit()
ses.expire_all() # Should this be here?
I have read from some posts on stack to include the following decorator function in models.py
#app.teardown_appcontext
def shutdown_session(exception=None):
ses.expire_all() #ses being database session object.
I tried this and it still doesn't work as it should be. Should I put the decorator function somewhere else?
Second thing i tried is to put ses.expire_all() after all commits and it still doesnt work.
What should I do to prevent this from happening?
Edit 1
from sqlalchemy import create_engine, update
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import NullPool
from config import MYSQLCONNECT
engine = create_engine(MYSQLCONNECT)
Session = sessionmaker(bind=engine)
session = Session()

I solved the problem with the use of following function from http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it:
from contextlib import contextmanager
#contextmanager
def session_scope():
"""Provide a transactional scope around a series of operations."""
session = Session()
try:
yield session
session.commit()
except:
session.rollback()
raise
finally:
session.close()
with session_scope() as session:
... # code that uses session
The problem was that I created the session object in the beggining and then never closed it.

Associate "external' class model with flask sqlalchemy

We use a central class model for a wide variety of python modules. This model is defined using SQLAlchemy. The classes all inherit from declarative_base.
For example, our model definitions look something like this:
Base = declarative_base()
class Post(Base):
__tablename__ = 'Posts'
id = Column(INT, primary_key=True, autoincrement=True)
body = Column(TEXT)
timestamp = Column(TIMESTAMP)
user_id = Column(INT, ForeignKey('Users.uid'))
We have been building a flask web-application in which we employ this same model. We have discovered a tricky problem in that flask-sqlalchemy appears to be designed in such a way that it expects all classes used in its model to have been defined by passing in an active instance of the session. Here is an example of a "proper" flask-sqalchemy class model definition:
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True)
email = db.Column(db.String(120), unique=True)
Note that the above example for flask-sqlalchemy requires an already-instantiated sql session. This has horrified us, because we are completely at a loss as to how to integrate our SqlAlchemy model into flask. We really want to use the flask-security suite in particular.
This problem has been brought up before on SO. Here, for example:
How to use flask-sqlalchemy with existing sqlalchemy model?
Our requirements are different from those of whoever accepted the response there. The response points out that one loses the ability to use User.query, but this is precisely one of the things we must retain.
It is not feasible to abandon our nice, elegant, central class model definition in favor of what flask-sqlalchemy appears to require. Is there any way for us to associate our model with the SQLAlchemy() object? Bonus points for getting us the .query() method on our classes which appears to be required by flask-security.

Solution:
As of today, the best way to do this is as follows:
Implement or import sqlalchemy base
from sqlalchemy.ext.declarative import declarative_base
base = declarative_base()
class Base(base):
__abstract__ = True
uid = Column(Integer, primary_key=True, autoincrement=True)
Register the external base:
from flask_sqlalchemy import SQLAlchemy
from model.base import Base
app = Flask(__name__)
db = SQLAlchemy(app, model_class=Base)
Archived for posterity:
I spent a lot of time looking for an answer to this. This is a lot easier to do today than it was when I originally asked the question, but it still isn't exactly simple.
For anyone who decides to do security themselves, I recommend the following excellent exposition of common design patterns which use flask, but which avoid employing unnecessary dependencies like flask-security:
https://exploreflask.com/users.html
UPDATE:
For anyone interested, a patch has been in the works for some time related to this. As of now it still isn't released, but you can check its progress here:
https://github.com/mitsuhiko/flask-sqlalchemy/pull/250#issuecomment-77504337
UPDATE:
I have taken the code from the above mentioned patch and created a local override for the SQLAlchemy object which allows one to register an external base. I think this is the best option available until such time as FSA gets around to adding this officially. Here is the code from that class for anyone interested. Tested working with Flask-SqlAlchemy 2.2
Patching in register_external_base:
import flask_sqlalchemy
'''Created by Isaac Martin 2017. Licensed insofar as it can be according to the standard terms of the MIT license: https://en.wikipedia.org/wiki/MIT_License. The author accepts no liability for consequences resulting from the use of this software. '''
class SQLAlchemy(flask_sqlalchemy.SQLAlchemy):
def __init__(self, app=None, use_native_unicode=True, session_options=None,
metadata=None, query_class=flask_sqlalchemy.BaseQuery, model_class=flask_sqlalchemy.Model):
self.use_native_unicode = use_native_unicode
self.Query = query_class
self.session = self.create_scoped_session(session_options)
self.Model = self.make_declarative_base(model_class, metadata)
self._engine_lock = flask_sqlalchemy.Lock()
self.app = app
flask_sqlalchemy._include_sqlalchemy(self, query_class)
self.external_bases = []
if app is not None:
self.init_app(app)
def get_tables_for_bind(self, bind=None):
"""Returns a list of all tables relevant for a bind."""
result = []
for Base in self.bases:
for table in flask_sqlalchemy.itervalues(Base.metadata.tables):
if table.info.get('bind_key') == bind:
result.append(table)
return result
def get_binds(self, app=None):
"""Returns a dictionary with a table->engine mapping.
This is suitable for use of sessionmaker(binds=db.get_binds(app)).
"""
app = self.get_app(app)
binds = [None] + list(app.config.get('SQLALCHEMY_BINDS') or ())
retval = {}
for bind in binds:
engine = self.get_engine(app, bind)
tables = self.get_tables_for_bind(bind)
retval.update(dict((table, engine) for table in tables))
return retval
#property
def bases(self):
return [self.Model] + self.external_bases
def register_base(self, Base):
"""Register an external raw SQLAlchemy declarative base.
Allows usage of the base with our session management and
adds convenience query property using self.Query by default."""
self.external_bases.append(Base)
for c in Base._decl_class_registry.values():
if isinstance(c, type):
if not hasattr(c, 'query') and not hasattr(c, 'query_class'):
c.query_class = self.Query
if not hasattr(c, 'query'):
c.query = flask_sqlalchemy._QueryProperty(self)
# for name in dir(c):
# attr = getattr(c, name)
# if type(attr) == orm.attributes.InstrumentedAttribute:
# if hasattr(attr.prop, 'query_class'):
# attr.prop.query_class = self.Query
# if hasattr(c , 'rel_dynamic'):
# c.rel_dynamic.prop.query_class = self.Query
To be used like so:
app = Flask(__name__)
db = SQLAlchemy(app)
db.register_base(base)

Pyramid Framework including models.py from addon to main application

I am trying to create a pyramid framework authentication addon/Plugin. The Plugin needs to have a database that stores user logins and other data, so if a user uses my addon his database must contain certain tables/models from the addon. e.g this Users table
class User(Base):
__tablename__ = 'User'
id = Column(Integer, primary_key=True)
username = Column(Text())
user_firstname = Column(Text())
user_lastname = Column(Text())
user_email = Column(Text())
user_password = Column(Text())
user_registrationdate = Column(DateTime())
user_email_key = Column(Text())
user_email_key_date_created = Column(DateTime())
user_email_approved = Column(Boolean())
user_email_sent = Column(Boolean())
user_active_account = Column(Boolean())
user_banned = Column(Boolean())
user_banned_reason = Column(Text())
I need this model to be included in the users main app, i am guessing i have to include something in my includeme shown below
def includeme(config):
config.include('pyramid_mako')
config.add_route('pyramid.admin', '/pyramid/admin')
#static views
config.add_static_view('assets', 'pyramidadmin:static/assets/')
config.scan('pyramidadmin.views')
but i have no Idea what to do. Is there a way i can merge the Base and DBSession from my addon to the base in the main application so that if a user runs ../bin/initialize_myapp_db the tables from myaddon and from the main app are all created?

I've had that same problem myself. I'm still working with trying to my stuff more pluggable, but with mixed results. The way I tackled this problem was to bind each declarative base class with the same engine. In my generic scaffolding I do this:
# create db engine
engine = engine_from_config(settings, 'sqlalchemy.')
# setup db.sessionmaker
settings['db.sessionmaker'] = DBSession
# bind session to engine
DBSession.configure(bind=engine)
# bind objects to engine
Base.metadata.bind = engine
from trumpet.models.base import Base as TrumpetBase
TrumpetBase.metadata.bind = engine
if settings.get('db.populate', 'False') == 'True':
from mslemon.models.main import make_test_data
import mslemon.models.misslemon
Base.metadata.create_all(engine)
TrumpetBase.metadata.create_all(engine)
#initialize_sql(engine)

You can use the following method:
import sqlalchemy
import sqlalchemy.orm as orm
from zope.sqlalchemy import ZopeTransactionExtension
from sqlalchemy.ext.declarative import declarative_base
DBSession = None
def get_sa_base(engine):
sabase = sqlalchemy.ext.declarative.declarative_base()
sabase.metadata.reflect(engine)
return sabase
def includeme(config):
global DBSession
engine = sqlalchemy.engine_from_config(config.registry.settings)
if DBSession is None:
DBSession = orm.scoped_session(
orm.sessionmaker(extension=ZopeTransactionExtension()))
DBSession.remove()
DBSession.configure(bind=engine)
Base = get_sa_base(engine)
# example:
Base.metadata.tables.values()
An example can be found here.

How to set up global connection to database?

I have problem with setting up database connection. I want to set connection, where I can see this connection in all my controllers.
Now I use something like this in my controller:
db = create_engine('mysql://root:password#localhost/python')
metadata = MetaData(db)
email_list = Table('email',metadata,autoload=True)
In development.ini I have:
sqlalchemy.url = mysql://root#password#localhost/python
sqlalchemy.pool_recycle = 3600
How do I set _____init_____.py?

I hope you got pylons working; for anyone else that may later read question I'll present some pointers in the right direction.
First of all, you are only creating a engine and a metadata object. While you can use the engine to create connections directly you would almost always use a Session to manage querying and updating your database.
Pylons automatically setups this for you by creating a engine from your configuration file, then passing it to yourproject.model.__init__.py:init_model() which binds it to a scoped_session object.
This scoped_session object is available from yourproject.model.meta and is the object you would use to query your database. For example:
record = meta.Session.query(model.MyTable).filter(id=42)
Because it is a scoped_session it automatically creates a Session object and associates it with the current thread if it doesn't already exists. Scoped_session passes all action (.query(), .add(), .delete()) down into the real Session object and thus allows you a simple way to interact the database with having to manage the non-thread-safe Session object explicitly.
The scoped_session, Session, object from yourproject.model.meta is automatically associated with a metadata object created as either yourproject.model.meta:metadata (in pylons 0.9.7 and below) or yourproject.model.meta:Base.metadata (in pylons 1.0). Use this metadata object to define your tables. As you can see in newer versions of pylons a metadata is associated with a declarative_base() object named Base, which allows you to use SqlAlchemy's declarative style.
Using this from the controller
from yourproject import model
from yourproject.model import Session
class MyController(..):
def resource(self):
result = Session.query(model.email_list).\
filter(model.email_list.c.id=42).one()
return str(result)
Use real connections
If you really want to get a connection object simply use
from yourproject.model import Session
connection = Session.connection()
result = connection.execute("select 3+4;")
// more connection executions
Session.commit()
However this is all good, but what you should be doing is...
This leaves out that you are not really using SqlAlchemy much. The power of SqlAlchemy really shines when you start mapping your database tables to python classes. So anyone looking into using pylons with a database should take a serious look at what you can do with SqlAlchemy. If SqlAlchemy starts out intimidating simply start out with using its declarative approach, which should be enough for almost all pylons apps.
In your model instead of defining Table constructs, do this:
from sqlalchemy import Column, Integer, Unicode, ForeignKey
from sqlalchemy.orm import relation
from yourproject.model.meta import Base
class User(Base):
__tablename__ = 'users'
# primary_key implies nullable=False
id = Column(Integer, primary_key=True, index=True)
# nullable defaults to True
name = Column(Unicode, nullable=False)
notes = relation("UserNote", backref="user")
query = Session.query_property()
class UserNote(Base):
__tablename__ = 'usernotess'
# primary_key implies nullable=False
id = Column(Integer, primary_key=True, index=True)
userid = Column(Integer, index=True, ForeignKey("User.id"))
# nullable defaults to True
text = Column(Unicode, nullable=False)
query = Session.query_property()
Note the query objects. These are smart object that live on the class and associates your classes with the scoped_session(), Session. This allows you to event more easily extract data from your database.
from sqlalchemy.orm import eagerload
def resource(self):
user = User.query.filter(User.id==42).options(eagerload("notes")).one()
return "\n".join([ x.text for x in user.notes ])

1.0 version of Pylons use declarative syntax. More about this, you can see here .
In mode/init.py you can write somthing like this:
from your_programm.model.meta import Session, Base
from sqlalchemy import *
from sqlalchemy.types import *
def init_model(engine):
Session.configure(bind=engine)
class Foo(Base) :
__tablename__ = "foo"
id = Column(Integer, primary_key=True)
name = Column(String)
...

What you want to do is modify the Globals class in your app_globals.py file to include a .engine (or whatever) attribute. Then, in your controllers, you use from pylons import app_globals and app_globals.engine to access the engine (or metadata, session, scoped_session, etc...).