SQLAlchemy allowed me to create a powerful database utility. Now I don't know how to deploy it. Let me explain how is it built with an example:
# objects.py
class Item(object):
def __init__(self, name):
self.name = name
# schema.py
from sqlalchemy import *
from objects import Item
engine=create_engine('sqlite:///mydb.db')
metadata = MetaData(engine)
item_table = Table(
'items', metadata,
Column('id', Integer, primary_key=True),
Column('name', String(100))
)
item_mapper = mapper(Item, item_table)
metadata.create_all()
# application.py
from schema import engine, Item
from sqlalchemy import *
Session = sessionmaker(bind=engine)
class Browser(object):
def __init__(self):
self.s = Session()
def get_by_name(self, name):
return self.s.query(Item).filter_by(name=name)
As you can see, what I want to make available is the last interface (Browser) where I simplify the queries for the end user.
If you simply request every user to open a Python shell and from application import Browser it seems that the advantages of connection pooling are not realized, because every user creates a different Session class (as opposed to creating a different session instance).
So, should I write a server that the users connect to? Or, how would you deploy this hypothetical application?
Thank you.
Connection pooling happens within the same python instance, so when your users connect from remote to the database, you have to write a small server anyways, if you want to use it. You can also connect directly to a database server, resulting in (at least) one connection per user. Depends on what you want to achieve.
Related
I develop an API server that interacts with MySQL DB reflecting it's schema and also get worked into multiple processes. My code for DB work looks like this:
from sqlalchemy import MetaData
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm.session import Session
my_engine = create_engine_by_info(my_config)
metadata = MetaData(bind=my_engine)
Base: type = automap_base(metadata=metadata)
class User(Base):
__tablename__ = 'auth_user'
# Relation descriptions...
# Other classes...
Base.prepare(my_engine, reflect=True)
def find_user(field):
with Session(my_engine) as session:
query = session.query(User)
query = query.filter(User.field == field)
records = query.all()
for u in records:
return u
return None
And it works fine until process gets forked: after work of the child process the original one looses connection: Lost connection to MySQL server during query.
I guess I should keep my_engine separate for each process (e.g some function with a dict of engines where key is a PID), but how can I do that if my classes definition requires an engine at the beginning? Probably I can move classes in a function too, but it would be a hell... So, what is a good solution here?
Description
I have an Flask application with original SQLalchemy. Application is intended to be used internally in a company for easier saving of measurement data with MySQL
On one page I have a table with all devices used for measurement and a form that is used to add, remove or modify measurement devices.
Problem
The problem is that when I enter a new device in the database, the page is automatically refreshed to fetch new data from DB and new device is sometimes shown and sometimes it is not when I refresh the page. In other words, added row in table is appearing and dissapearing even though the row is visible on database. Same goes when i try to delete the device from database. The row is sometimes shown, sometimes not when refreshing the page with row being deleted from DB.
The same problem appears for all examples similar to this one (adding, deleting and modifying data).
What i have tried
Bellow is the code for table model:
class DvDevice(Base):
__tablename__ = "dvdevice"
id = Column("device_id", Integer, primary_key=True, autoincrement=True)
name = Column("device_name", String(50), nullable=True)
code = Column("device_code", String(10), nullable=True, unique=True)
hw_ver = Column("hw_ver", String(10), nullable=True)
fw_ver = Column("fw_ver", String(10), nullable=True)
sw_ver = Column("sw_ver", String(10), nullable=True)
And here is the code that inserts/deletes data from table.
#Insertion
device = DvDevice()
device.code = self.device_code
device.name = self.device_name
device.hw_ver = self.hw_ver
device.fw_ver = self.fw_ver
device.sw_ver = self.sw_ver
ses.add(device)
ses.commit()
ses.expire_all() #Should this be here?
# Deletion
ses.query(DvDevice).filter_by(id=self.device_id).delete()
ses.commit()
ses.expire_all() # Should this be here?
I have read from some posts on stack to include the following decorator function in models.py
#app.teardown_appcontext
def shutdown_session(exception=None):
ses.expire_all() #ses being database session object.
I tried this and it still doesn't work as it should be. Should I put the decorator function somewhere else?
Second thing i tried is to put ses.expire_all() after all commits and it still doesnt work.
What should I do to prevent this from happening?
Edit 1
from sqlalchemy import create_engine, update
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import NullPool
from config import MYSQLCONNECT
engine = create_engine(MYSQLCONNECT)
Session = sessionmaker(bind=engine)
session = Session()
I solved the problem with the use of following function from http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it:
from contextlib import contextmanager
#contextmanager
def session_scope():
"""Provide a transactional scope around a series of operations."""
session = Session()
try:
yield session
session.commit()
except:
session.rollback()
raise
finally:
session.close()
with session_scope() as session:
... # code that uses session
The problem was that I created the session object in the beggining and then never closed it.
I am attempting to implement Flask-Session in my python application. I read in the docs that its recommended to use another interface like the SqlAlchemySessionInterface instead of the default NullSessionInterface which is used when nothing is provided to the SESSION_TYPE configuration key.
From the flask_session/init.py file under class Session it reads
By default Flask-Session will use :class:NullSessionInterface, you
really should configurate your app to use a different SessionInterface.
After setting the SESSION_TYPE configuration key to "sqlalchemy" I get an error
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "sessions" does not exist
This indicates that Flask-Session is looking to use a table with the name "sessions" in my database model but I cannot find anywhere in the Flask-Session documentation where it points out that a table should be created and what fields it should have.
Can anyone suggest a solution to this please?
I wanted to use Flask-session, but I was also using Flask-migrate and didn't want to call db.create_all() manually and break the migration path. Fortunately, #Flashspeedlife's suggestion of just importing the Interface and instantiating it worked.
app/__init__.py:
from flask_session import SqlAlchemySessionInterface
from app.extensions import db, sess, migrate # My extensions file
def create_app():
app = Flask(__name__)
with app.app_context():
db.init_app(app)
migrate.init_app(app, db)
sess.init_app(app)
SqlAlchemySessionInterface(app, db, "sessions", "sess_")
Now, flask db migrate generates an alembic script with the new sessions table.
After studying the Flask-Session/init.py code I found that class SqlAlchemySessionInterface under its __init__ contains a Flask-SQLAlchemy model
class Session(self.db.Model).
To cause this table model to be created, in the file where I create my models I imported SqlAlchemySessionInterface from flask_sessionstore and put the line
SqlAlchemySessionInterface(myApp, sqlAlchemyDbObject, "table_name", "prefix_")
and then ran db.create_all().
class SqlAlchemySessionInterface(SessionInterface):
"""Uses the Flask-SQLAlchemy from a flask app as a session backend.
.. versionadded:: 0.2
:param app: A Flask app instance.
:param db: A Flask-SQLAlchemy instance.
:param table: The table name you want to use.
:param key_prefix: A prefix that is added to all store keys.
:param use_signer: Whether to sign the session id cookie or not.
:param permanent: Whether to use permanent session or not.
"""
serializer = pickle
session_class = SqlAlchemySession
def __init__(self, app, db, table, key_prefix, use_signer=False,
permanent=True):
if db is None:
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
self.db = db
self.key_prefix = key_prefix
self.use_signer = use_signer
self.permanent = permanent
class Session(self.db.Model):
__tablename__ = table
id = self.db.Column(self.db.Integer, primary_key=True)
session_id = self.db.Column(self.db.String(255), unique=True)
data = self.db.Column(self.db.LargeBinary)
expiry = self.db.Column(self.db.DateTime)
def __init__(self, session_id, data, expiry):
self.session_id = session_id
self.data = data
self.expiry = expiry
def __repr__(self):
return '<Session data %s>' % self.data
# self.db.create_all()
self.sql_session_model = Session
I'am definitely using Django for my next project. Documentation for many Flask Extensions aren't great at all.
EDIT
Changed (imported SqlAlchemySessionInterface from flask_session) to (imported SqlAlchemySessionInterface from flask_sessionstore)
I have a python module UserManager that takes care for all things user management related - users, groups, rights, authentication. Access to these assets is provided via master class that is passed SQLAlchemy engine parameter at constructor. The engine is needed to make the table-class mappings (using mapper objects), and to emit sessions.
This is how the gobal variables are established in the app module:
class UserManager:
def __init__(self, db):
self.db = db
self._db_session = None
meta = MetaData(db)
user_table = Table(
'USR_User', meta,
Column('field1'),
Column('field3')
)
mapper(User, user_table)
#property
def db_session(self):
if self._db_session is None:
self._db_session = scoped_session(sessionmaker())
self._db_session.configure(bind=self.db)
return self._db_session
class User(object):
def init(self, um):
self.um = um
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
um = UserManager(db.engine)
This module as such is designed to be context-agnostic by purpose, so that it can be used both for locally run and web application.
But here the problems arise: time to time I get the dreaded "Can't reconnect until invalid transaction is rolled back" error, presumably caused by some failed transaction in the UserManager code.
I am now trying to identify the problem source. Maybe it is not right way how to handle the database in the dynamic context of web server? Perhaps I have to pass the db.session to the um object so that I can be sure that the db connections are not mixed up?
In web context you should consider the request for every user isolated. For this you must use the flask.g
To share data that is valid for one request only from one function to
another, a global variable is not good enough because it would break
in threaded environments.Flask provides you with a special object
that ensures it is only valid for the active request and that will
return different values for each request. In a nutshell: it does the
right thing, like it does for request and session.
You can see more about here.
I have problem with setting up database connection. I want to set connection, where I can see this connection in all my controllers.
Now I use something like this in my controller:
db = create_engine('mysql://root:password#localhost/python')
metadata = MetaData(db)
email_list = Table('email',metadata,autoload=True)
In development.ini I have:
sqlalchemy.url = mysql://root#password#localhost/python
sqlalchemy.pool_recycle = 3600
How do I set _____init_____.py?
I hope you got pylons working; for anyone else that may later read question I'll present some pointers in the right direction.
First of all, you are only creating a engine and a metadata object. While you can use the engine to create connections directly you would almost always use a Session to manage querying and updating your database.
Pylons automatically setups this for you by creating a engine from your configuration file, then passing it to yourproject.model.__init__.py:init_model() which binds it to a scoped_session object.
This scoped_session object is available from yourproject.model.meta and is the object you would use to query your database. For example:
record = meta.Session.query(model.MyTable).filter(id=42)
Because it is a scoped_session it automatically creates a Session object and associates it with the current thread if it doesn't already exists. Scoped_session passes all action (.query(), .add(), .delete()) down into the real Session object and thus allows you a simple way to interact the database with having to manage the non-thread-safe Session object explicitly.
The scoped_session, Session, object from yourproject.model.meta is automatically associated with a metadata object created as either yourproject.model.meta:metadata (in pylons 0.9.7 and below) or yourproject.model.meta:Base.metadata (in pylons 1.0). Use this metadata object to define your tables. As you can see in newer versions of pylons a metadata is associated with a declarative_base() object named Base, which allows you to use SqlAlchemy's declarative style.
Using this from the controller
from yourproject import model
from yourproject.model import Session
class MyController(..):
def resource(self):
result = Session.query(model.email_list).\
filter(model.email_list.c.id=42).one()
return str(result)
Use real connections
If you really want to get a connection object simply use
from yourproject.model import Session
connection = Session.connection()
result = connection.execute("select 3+4;")
// more connection executions
Session.commit()
However this is all good, but what you should be doing is...
This leaves out that you are not really using SqlAlchemy much. The power of SqlAlchemy really shines when you start mapping your database tables to python classes. So anyone looking into using pylons with a database should take a serious look at what you can do with SqlAlchemy. If SqlAlchemy starts out intimidating simply start out with using its declarative approach, which should be enough for almost all pylons apps.
In your model instead of defining Table constructs, do this:
from sqlalchemy import Column, Integer, Unicode, ForeignKey
from sqlalchemy.orm import relation
from yourproject.model.meta import Base
class User(Base):
__tablename__ = 'users'
# primary_key implies nullable=False
id = Column(Integer, primary_key=True, index=True)
# nullable defaults to True
name = Column(Unicode, nullable=False)
notes = relation("UserNote", backref="user")
query = Session.query_property()
class UserNote(Base):
__tablename__ = 'usernotess'
# primary_key implies nullable=False
id = Column(Integer, primary_key=True, index=True)
userid = Column(Integer, index=True, ForeignKey("User.id"))
# nullable defaults to True
text = Column(Unicode, nullable=False)
query = Session.query_property()
Note the query objects. These are smart object that live on the class and associates your classes with the scoped_session(), Session. This allows you to event more easily extract data from your database.
from sqlalchemy.orm import eagerload
def resource(self):
user = User.query.filter(User.id==42).options(eagerload("notes")).one()
return "\n".join([ x.text for x in user.notes ])
1.0 version of Pylons use declarative syntax. More about this, you can see here .
In mode/init.py you can write somthing like this:
from your_programm.model.meta import Session, Base
from sqlalchemy import *
from sqlalchemy.types import *
def init_model(engine):
Session.configure(bind=engine)
class Foo(Base) :
__tablename__ = "foo"
id = Column(Integer, primary_key=True)
name = Column(String)
...
What you want to do is modify the Globals class in your app_globals.py file to include a .engine (or whatever) attribute. Then, in your controllers, you use from pylons import app_globals and app_globals.engine to access the engine (or metadata, session, scoped_session, etc...).