I have an existing, working Flask app that uses SQLAlchemy. Several of the models/tables in this app have columns that store raw HTML, and I'd like to inject a function on a column's setter so that the incoming raw html gets 'cleansed'. I want to do this in the model so I don't have to sprinkle "clean this data" all through the form or route code.
I can currently already do this like so:
from application import db, clean_the_data
from sqlalchemy.ext.hybrid import hybrid_property
class Example(db.Model):
__tablename__ = 'example'
normal_column = db.Column(db.Integer,
primary_key=True,
autoincrement=True)
_html_column = db.Column('html_column', db.Text,
nullable=False)
#hybrid_property
def html_column(self):
return self._html_column
#html_column.setter
def html_column(self, value):
self._html_column = clean_the_data(value)
This works like a charm - except for the model definition the _html_column name is never seen, the cleaner function is called, and the cleaned data is used. Hooray.
I could of course stop there and just eat the ugly handling of the columns, but why do that when you can mess with metaclasses?
Note: the following all assumes that 'application' is the main Flask module, and that it contains two children: 'db' - the SQLAlchemy handle and 'clean_the_data', the function to clean up the incoming HTML.
So, I went about trying to make a new base Model class that spotted a column that needs cleaning when the class is being created, and juggled things around automatically, so that instead of the above code, you could do something like this:
from application import db
class Example(db.Model):
__tablename__ = 'example'
__html_columns__ = ['html_column'] # Our oh-so-subtle hint
normal_column = db.Column(db.Integer,
primary_key=True,
autoincrement=True)
html_column = db.Column(db.Text,
nullable=False)
Of course, the combination of trickery with metaclasses going on behind the scenes with SQLAlchemy and Flask made this less than straight-forward (and is also why the nearly matching question "Custom metaclass to create hybrid properties in SQLAlchemy" doesn't quite help - Flask gets in the way too). I've almost gotten there with the following in application/models/__init__.py:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.hybrid import hybrid_property
# Yes, I'm importing _X stuff...I tried other ways to avoid this
# but to no avail
from flask_sqlalchemy import (Model as BaseModel,
_BoundDeclarativeMeta,
_QueryProperty)
from application import db, clean_the_data
class _HTMLBoundDeclarativeMeta(_BoundDeclarativeMeta):
def __new__(cls, name, bases, d):
# Move any fields named in __html_columns__ to a
# _field/field pair with a hybrid_property
if '__html_columns__' in d:
for field in d['__html_columns__']:
if field not in d:
continue
hidden = '_' + field
fget = lambda self: getattr(self, hidden)
fset = lambda self, value: setattr(self, hidden,
clean_the_data(value))
d[hidden] = d[field] # clobber...
d[hidden].name = field # So we don't have to explicitly
# name the column. Should probably
# force a quote on the name too
d[field] = hybrid_property(fget, fset)
del d['__html_columns__'] # Not needed any more
return _BoundDeclarativeMeta.__new__(cls, name, bases, d)
# The following copied from how flask_sqlalchemy creates it's Model
Model = declarative_base(cls=BaseModel, name='Model',
metaclass=_HTMLBoundDeclarativeMeta)
Model.query = _QueryProperty(db)
# Need to replace the original Model in flask_sqlalchemy, otherwise it
# uses the old one, while you use the new one, and tables aren't
# shared between them
db.Model = Model
Once that's set, your model class can look like:
from application import db
from application.models import Model
class Example(Model): # Or db.Model really, since it's been replaced
__tablename__ = 'example'
__html_columns__ = ['html_column'] # Our oh-so-subtle hint
normal_column = db.Column(db.Integer,
primary_key=True,
autoincrement=True)
html_column = db.Column(db.Text,
nullable=False)
This almost works, in that there's no errors, data is read and saved correctly, etc. Except the setter for the hybrid_property is never called. The getter is (I've confirmed with print statements in both), but the setter is ignored totally and the cleaner function is thus never called. The data is set though - changes are made quite happily with the un-cleaned data.
Obviously I've not quite completely emulated the static version of the code in my dynamic version, but I honestly have no idea where the issue is. As far as I can see, the hybrid_property should be registering the setter just like it has the getter, but it's just not. In the static version, the setter is registered and used just fine.
Any ideas on how to get that final step working?
Maybe use a custom type ?
from sqlalchemy import TypeDecorator, Text
class CleanedHtml(TypeDecorator):
impl = Text
def process_bind_param(self, value, dialect):
return clean_the_data(value)
Then you can just write your models this way:
class Example(db.Model):
__tablename__ = 'example'
normal_column = db.Column(db.Integer, primary_key=True, autoincrement=True)
html_column = db.Column(CleanedHtml)
More explanations are available in the documentation here: http://docs.sqlalchemy.org/en/latest/core/custom_types.html#augmenting-existing-types
Related
I have two particular model classes that are giving me errors during my testing, upon inspecting the methods for each I was pretty certain my issues were the result of typos on my part.
I've made the changes in both classes as needed, but re-running the tests produces the same error. I even tried dropping my schema and re-creating it with Flask-SQLAlchemy's create_all() method, but I still run into issues.
In the Metrics class, the variables in the __init__ method were wrong and were missing underscores (Ie: self.name instead of self._name). I addressed that by changing them to self._name and self._metric_type
In the HostMetricMapping class, I needed to add the host_id parameter to the __init__ method, since I had forgotten it the first time. So, I added it.
class Metrics(_database.Model):
__tablename__ = 'Metrics'
_ID = _database.Column(_database.Integer, primary_key=True)
_name = _database.Column(_database.String(45), nullable=False)
_metric_type = _database.Column(_database.String(45))
_host_metric_mapping = _database.relationship('HostMetricMapping', backref='_parent_metric', lazy=True)
def __init__(self, name, metric_type):
self._name = name # This line used to say self.name, but was changed to self._name to match the column name
self._metric_type = metric_type # This line used to say self.metric_type, but was changed to self._metric_type to match the column name
def __repr__(self):
return '{0}'.format(self._ID)
class HostMetricMapping(_database.Model):
__tablename__ = 'HostMetricMapping'
_ID = _database.Column(_database.Integer, primary_key=True)
_host_id = _database.Column(_database.Integer, _database.ForeignKey('Hosts._ID'), nullable=False)
_metric_id = _database.Column(_database.Integer, _database.ForeignKey('Metrics._ID'), nullable=False)
_metric = _database.relationship('MetricData', backref='_metric_hmm', lazy=True)
_threshold = _database.relationship('ThresholdMapping', backref='_threshold_hmm', lazy=True)
def __init__(self, host_id, metric_id):
self._host_id = host_id # This line and it's corresponding parameter were missing, and were added
self._metric_id = metric_id
def __repr__(self):
return '{0}'.format(self._ID)
The issues I encounter are:
When trying to instantiate an instance of Metrics and add it into the database, SQLAlchemy raises an IntegrityError because I have the _name column set to not null, and SQLAlchemy inherits the values for both _name and _metric_type as None or NULL, even though I instantiate it with values for both parameters.
For HostMetricMapping, Python raises an exception because it still treats that class as only having the metric_id parameter, instead of also having the host_id parameter I've added.
A better way to override __init__ when using flask-sqlalchemy is to use reconstructor. Object initialization with sqlalchemy is a little tricky, and flask-sqlalchemy might be complicating it as well.. anyways, here's how we do it:
from sqlalchemy.orm import reconstructor
class MyModel(db.Model):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.init_on_load()
#reconstructor
def init_on_load(self):
# put your init stuff here
I am using flaks python and sqlalchemy to connect to a huge db, where a lot of stats are saved. I need to create some useful insights with the use of these stats, so I only need to read/get the data and never modify.
The issue I have now is the following:
Before I can access a table I need to replicate the table in my models file. For example I see the table Login_Data in the DB. So I go into my models and recreate the exact same table.
class Login_Data(Base):
__tablename__ = 'login_data'
id = Column(Integer, primary_key=True)
date = Column(Date, nullable=False)
new_users = Column(Integer, nullable=True)
def __init__(self, date=None, new_users=None):
self.date = date
self.new_users = new_users
def get(self, id):
if self.id == id:
return self
else:
return None
def __repr__(self):
return '<%s(%r, %r, %r)>' % (self.__class__.__name__, self.id, self.date, self.new_users)
I do this because otherwise I cant query it using:
some_data = Login_Data.query.limit(10)
But this feels unnecessary, there must be a better way. Whats the point in recreating the models if they are already defined. What shall I use here:
some_data = [SOMETHING HERE SO I DONT NEED TO RECREATE THE TABLE].query.limit(10)
Simple question but I have not found a solution yet.
Thanks to Tryph for the right sources.
To access the data of an existing DB with sqlalchemy you need to use automap. In your configuration file where you load/declare your DB type. You need to use the automap_base(). After that you can create your models and use the correct table names of the DB without specifying everything yourself:
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import Session
from sqlalchemy import create_engine
import stats_config
Base = automap_base()
engine = create_engine(stats_config.DB_URI, convert_unicode=True)
# reflect the tables
Base.prepare(engine, reflect=True)
# mapped classes are now created with names by default
# matching that of the table name.
LoginData = Base.classes.login_data
db_session = Session(engine)
After this is done you can now use all your known sqlalchemy functions on:
some_data = db_session.query(LoginData).limit(10)
You may be interested by reflection and automap.
Unfortunately, since I never used any of those features, I am not able to tell you more about them. I just know that they allow to use the database schema without explicitly declaring it in Python.
We use a central class model for a wide variety of python modules. This model is defined using SQLAlchemy. The classes all inherit from declarative_base.
For example, our model definitions look something like this:
Base = declarative_base()
class Post(Base):
__tablename__ = 'Posts'
id = Column(INT, primary_key=True, autoincrement=True)
body = Column(TEXT)
timestamp = Column(TIMESTAMP)
user_id = Column(INT, ForeignKey('Users.uid'))
We have been building a flask web-application in which we employ this same model. We have discovered a tricky problem in that flask-sqlalchemy appears to be designed in such a way that it expects all classes used in its model to have been defined by passing in an active instance of the session. Here is an example of a "proper" flask-sqalchemy class model definition:
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True)
email = db.Column(db.String(120), unique=True)
Note that the above example for flask-sqlalchemy requires an already-instantiated sql session. This has horrified us, because we are completely at a loss as to how to integrate our SqlAlchemy model into flask. We really want to use the flask-security suite in particular.
This problem has been brought up before on SO. Here, for example:
How to use flask-sqlalchemy with existing sqlalchemy model?
Our requirements are different from those of whoever accepted the response there. The response points out that one loses the ability to use User.query, but this is precisely one of the things we must retain.
It is not feasible to abandon our nice, elegant, central class model definition in favor of what flask-sqlalchemy appears to require. Is there any way for us to associate our model with the SQLAlchemy() object? Bonus points for getting us the .query() method on our classes which appears to be required by flask-security.
Solution:
As of today, the best way to do this is as follows:
Implement or import sqlalchemy base
from sqlalchemy.ext.declarative import declarative_base
base = declarative_base()
class Base(base):
__abstract__ = True
uid = Column(Integer, primary_key=True, autoincrement=True)
Register the external base:
from flask_sqlalchemy import SQLAlchemy
from model.base import Base
app = Flask(__name__)
db = SQLAlchemy(app, model_class=Base)
Archived for posterity:
I spent a lot of time looking for an answer to this. This is a lot easier to do today than it was when I originally asked the question, but it still isn't exactly simple.
For anyone who decides to do security themselves, I recommend the following excellent exposition of common design patterns which use flask, but which avoid employing unnecessary dependencies like flask-security:
https://exploreflask.com/users.html
UPDATE:
For anyone interested, a patch has been in the works for some time related to this. As of now it still isn't released, but you can check its progress here:
https://github.com/mitsuhiko/flask-sqlalchemy/pull/250#issuecomment-77504337
UPDATE:
I have taken the code from the above mentioned patch and created a local override for the SQLAlchemy object which allows one to register an external base. I think this is the best option available until such time as FSA gets around to adding this officially. Here is the code from that class for anyone interested. Tested working with Flask-SqlAlchemy 2.2
Patching in register_external_base:
import flask_sqlalchemy
'''Created by Isaac Martin 2017. Licensed insofar as it can be according to the standard terms of the MIT license: https://en.wikipedia.org/wiki/MIT_License. The author accepts no liability for consequences resulting from the use of this software. '''
class SQLAlchemy(flask_sqlalchemy.SQLAlchemy):
def __init__(self, app=None, use_native_unicode=True, session_options=None,
metadata=None, query_class=flask_sqlalchemy.BaseQuery, model_class=flask_sqlalchemy.Model):
self.use_native_unicode = use_native_unicode
self.Query = query_class
self.session = self.create_scoped_session(session_options)
self.Model = self.make_declarative_base(model_class, metadata)
self._engine_lock = flask_sqlalchemy.Lock()
self.app = app
flask_sqlalchemy._include_sqlalchemy(self, query_class)
self.external_bases = []
if app is not None:
self.init_app(app)
def get_tables_for_bind(self, bind=None):
"""Returns a list of all tables relevant for a bind."""
result = []
for Base in self.bases:
for table in flask_sqlalchemy.itervalues(Base.metadata.tables):
if table.info.get('bind_key') == bind:
result.append(table)
return result
def get_binds(self, app=None):
"""Returns a dictionary with a table->engine mapping.
This is suitable for use of sessionmaker(binds=db.get_binds(app)).
"""
app = self.get_app(app)
binds = [None] + list(app.config.get('SQLALCHEMY_BINDS') or ())
retval = {}
for bind in binds:
engine = self.get_engine(app, bind)
tables = self.get_tables_for_bind(bind)
retval.update(dict((table, engine) for table in tables))
return retval
#property
def bases(self):
return [self.Model] + self.external_bases
def register_base(self, Base):
"""Register an external raw SQLAlchemy declarative base.
Allows usage of the base with our session management and
adds convenience query property using self.Query by default."""
self.external_bases.append(Base)
for c in Base._decl_class_registry.values():
if isinstance(c, type):
if not hasattr(c, 'query') and not hasattr(c, 'query_class'):
c.query_class = self.Query
if not hasattr(c, 'query'):
c.query = flask_sqlalchemy._QueryProperty(self)
# for name in dir(c):
# attr = getattr(c, name)
# if type(attr) == orm.attributes.InstrumentedAttribute:
# if hasattr(attr.prop, 'query_class'):
# attr.prop.query_class = self.Query
# if hasattr(c , 'rel_dynamic'):
# c.rel_dynamic.prop.query_class = self.Query
To be used like so:
app = Flask(__name__)
db = SQLAlchemy(app)
db.register_base(base)
I have model declared as:
class SAProduct(Base):
sku = Column(PRODUCT_SKU_TYPE, primary_key=True)
i_want_to_hide = Column(String(20), nullable=False)
name = Column(Unicode(255), nullable=True)
#property
def my_property(self):
return i_calculate_property_here(self)
and Spyne model declared as:
db = create_engine('sqlite:///:memory:')
Session = sessionmaker(bind=db)
class TableModel(ComplexModelBase):
__metaclass__ = ComplexModelMeta
__metadata__ = MetaData(bind=db)
class SProduct(TableModel):
__table__ = SAProduct.__table__
How can I make attribute i_want_to_hide to be excluded from Spyne model, and property my_property to be included as Spyne model attribute?
P.S.
Now I use monkey patching Spyne to support this syntax:
class SProduct(GComplexModel):
__model__ = Product
class Attributes:
exclude_attrs = ('i_want_to_hide',)
add_attrs = {'my_property': Boolean}
But I want get rid of it.
This doesn't directly answer your question, but please consider the following code:
from spyne import *
TableModel = TTableModel()
class SomeClass(TableModel):
__tablename__ = 'some_table'
id = Integer(pk=True)
s = Unicode
i = Integer(exc_table=True)
Here, pk stands for primary key (you can use the long form primary_key if you wish) and the i attribute will just be ignored by SqlAlchemy. e.g. it won't be created in the table, it won't be instrumented by SqlAlchemy's metaclass, etc.
As for an attribute that will be hidden from the RPC Parts of Spyne, but not from SqlAlchemy, that's a new feature coming in 2.12.
You will be able to say e.g.:
i = Integer(exc_table=True, pa={JsonObject: dict(exc=true)})
where pa stands for protocol attributes. (you can use the long form prot_attrs if you wish) Here i is ignored by every protocol that inherits JsonObject.
If you don't want it on the wsdl either, you'll have to do:
i = Integer(exc_table=True, exc_interface=True)
https://github.com/arskom/spyne/blob/fa4b1eef5815d3584287d1fef66b61846f82d2f8/spyne/interface/xml_schema/model.py#L197
Spyne offers a richer object model interface compared to SqlAlchemy. Trying to replicate this functionality without adding Spyne as a dependency means you'll have to duplicate all the work done in Spyne in your project. It's your choice!
I'm new to SQLAlchemy and am using Flask-SQLAlchemy for my current project. I'm getting an error that has me stumped:
sqlalchemy.orm.exc.UnmappedInstanceError: Class 'flask_sqlalchemy._BoundDeclarativeMeta' is not mapped; was a class (app.thing.Thing) supplied where an instance was required?
I've tried changing the notation in the instatiation, and moving some things around. Could the problem be related to this question, or something else?
Here is my module (thing.py) with the class that inherits from Flask-SQLAlchemy's db.Model (like SQLAlchemy's Base class):
from app import db
class Thing(db.Model):
indiv_id = db.Column(db.INTEGER, primary_key = True)
stuff_1 = db.Column(db.INTEGER)
stuff_2 = db.Column(db.INTEGER)
some_stuff = db.Column(db.BLOB)
def __init__():
pass
And here is my call to populate the database in the thing_wrangler.py module:
import thing
from app import db
def populate_database(number_to_add):
for each_thing in range(number_to_add):
a_thing = thing.Thing
a_thing.stuff_1 = 1
a_thing.stuff_2 = 2
a_thing.some_stuff = [1,2,3,4,5]
db.session.add(a_thing)
db.session.commit()
You need to create an instance of your model. Instead of a_thing = thing.Thing, it should be a_thing = thing.Thing(). Notice the parentheses. Since you overrode __init__, you also need to fix it so it takes self as the first argument.