I am using SQLAlchemy + SQLite3 for creating multiple databases based on user input. When initializing a new database, the user defines any number of arbitrary features and their types. I wrote a DBManager class to serve as an interface between user input and database creation/access.
Dynamically "injecting" these arbitrary features in the declarative model (the Features class) is working as expected. The problem I have is when the user wants to create a second/different database: I can't figure out how to completely "clear" or "refresh" the model or the declarative_base so that the user is able to create a new database (with possibly different features).
Below is a minimal reproducible example of my situation:
src.__init__.py:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Session = sessionmaker()
Base = declarative_base()
src.features.py
from sqlalchemy import Column, ForeignKey, Integer
from sqlalchemy.orm import relationship
from src import Base
class Features(Base):
__tablename__ = "features"
features_id = Column(Integer, primary_key=True)
#classmethod
def add_feature(cls, feature_name, feature_type):
setattr(cls, feature_name, Column(feature_type))
src.db_manager.py:
from typing import Optional, Dict
from sqlalchemy import create_engine
from src import Base, Session
from src.features import Features
class DBManager:
def __init__(self, path: str, features: Optional[Dict] = None) -> None:
self.engine = create_engine(f'sqlite:///{path}')
Session.configure(bind=self.engine)
self.session = Session()
self.features = features
if self.features: # user passed in some arbitrary features
self.bind_features_to_features_table()
Base.metadata.create_all(bind=self.engine)
def bind_features_to_features_table(self):
for feature_name, feature_type in self.features.items():
Features.add_feature(feature_name=feature_name, feature_type=feature_type)
I'd like to be able to do something like this:
from sqlalchemy import String, Float, Integer
from src.db_manager import DBManager
# User wants to create a database with these features
features = {
'name': String,
'height': Float,
}
db_manager = DBManager(path='my_database.db', features=features)
# ... User does some stuff with database here ...
# Now the user wants to create another database with these features
other_features = {
'age': Integer,
'weight': Float,
'city_of_residence': String,
'name': String,
}
db_manager = DBManager(path='another_database.db', features=other_features)
After executing the last line, I'm met with: InvalidRequestError: Implicitly combining column features.name with column features.name under attribute 'name'. Please configure one or more attributes for these same-named columns explicitly. The error wouldn't occur if the feature name did not appear on both databases, but then the feature height would be brought over to the second database, which is not desired.
Things I tried but didn't work:
call Base.metadata.clear() between DBManager instances: same error
call sqlalchemy.orm.clear_mappers() between DBManager instances: results in AttributeError: 'NoneType' object has no attribute 'instrument_attribute'
call delattr(Features, feature_name): results in NotImplementedError: Can't un-map individual mapped attributes on a mapped class..
This program will be running inside a GUI, so I can't really afford to exit/restart the script in order to connect to the second database. The user should be able to load/create different databases without having to close the program.
I understand that the error stems from the fact that the underlying Base object has not been "refreshed" and is still keeping track of the features created in my first DBManager instance. However I do not know how to fix this. What's worse, any attempt to overwrite/reload a new Base object will need to be applied to all modules that imported that object from __init__.py, which sounds tricky. Does anyone have a solution for this?
My solution was to define the Features declarative class inside a function, get_features, that takes a Base (declarative base) instance as an argument. The function returns the Features class object, so that every call essentially creates a new Features class as a whole.
The class DBManager is then responsible for calling that function, and Features becomes a instance attribute of DBManager. Creating a new instance of DBManager means creating an entire new class based on Features, to which I can then add any arbitrary features I'd like.
The code looks something like this:
def get_features(declarative_base):
class Features(declarative_base):
__tablename__ = "features"
features_id = Column(Integer, primary_key=True)
#classmethod
def add_feature(cls, feature_name, feature_type):
setattr(cls, feature_name, Column(feature_type))
return Features
class DBManager:
def __init__(self, path, features):
self.engine = create_engine(f'sqlite:///{path}')
Session.configure(bind=self.engine)
self.session = Session()
base = declarative_base()
self.features_table = get_features(base=base)
if self.features: # user passed in some arbitrary features
self.bind_features_to_features_table()
Base.metadata.create_all(bind=self.engine)
def bind_features_to_features_table(self):
for feature_name, feature_type in self.features.items():
self.features_table.add_feature(feature_name=feature_name, feature_type=feature_type)
It definitely feels a bit convoluted, and I have no idea if there are any caveats I'm not aware of, but as far as I can tell this approach solved my problem.
I have a class Contract to represent my contracts:
.../mypackage/Contract.py
class Contract:
# setter and getters.
def isValid( self, contract_number=None ):
#code
def cancelTheContract( self, contract_number=None ):
# code
And my SQLAlchemy Contract class:
.../mypackage/orm.py
from sqlalchemy import create_engine
from sqlalchemy import MetaData
from sqlalchemy import Column, ForeignKey, Integer, String, Table
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
db = create_engine( 'mysql://myuser:mypasswd#localhost/mydatabase' )
contracts = Table( 'contracts', MetaData( bind = None ) )
class Connection:
def connect( self ):
Session = sessionmaker( bind = db )
session = Session()
return session
class Contract( Base ):
__tablename__ = 'contracts'
id = Column( Integer, primary_key = True )
type = Column( String )
price = Column( Float )
So...
Would be ok to merge both Contract classes in a single one?
If not, I have to instantiate a class specific for the database table and another class specific for the business logic, so when I have to deal with database data, manipulate it and put it back, I have to deal with two objects that are basically the thing.
Well... I guess I'm missing some important concept here.
What should I read to understand better about my question implications?
Thanks!
Gio
Yes, the philosophy of ORMs is to map physical tables to business entity objects, so it is best practice to combine your two classes. SQLA attributes manage the persistent fields of your entity and you can encapsulate all the business logic in that class per standard object-oriented modeling techniques.
Is there a "Pythonic" way (I mean, no "pure SQL" query) to define an SQL view with SQLAlchemy?
Update: SQLAlchemy now has a great usage recipe here on this topic, which I recommend. It covers different SQL Alchemy versions up to the latest and has ORM integration (see comments below this answer and other answers). And if you look through the version history, you can also learn why using literal_binds is iffy (in a nutshell: binding parameters should be left to the database), but still arguably any other solution would make most users of the recipe not happy. I leave the below answer mostly for historical reasons.
Original answer: Creating a (read-only non-materialized) view is not supported out of the box as far as I know. But adding this functionality in SQLAlchemy 0.7 is straightforward (similar to the example I gave here). You just have to write a compiler extension CreateView. With this extension, you can then write (assuming that t is a table object with a column id)
createview = CreateView('viewname', t.select().where(t.c.id>5))
engine.execute(createview)
v = Table('viewname', metadata, autoload=True)
for r in engine.execute(v.select()):
print r
Here is a working example:
from sqlalchemy import Table
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.sql.expression import Executable, ClauseElement
class CreateView(Executable, ClauseElement):
def __init__(self, name, select):
self.name = name
self.select = select
#compiles(CreateView)
def visit_create_view(element, compiler, **kw):
return "CREATE VIEW %s AS %s" % (
element.name,
compiler.process(element.select, literal_binds=True)
)
# test data
from sqlalchemy import MetaData, Column, Integer
from sqlalchemy.engine import create_engine
engine = create_engine('sqlite://')
metadata = MetaData(engine)
t = Table('t',
metadata,
Column('id', Integer, primary_key=True),
Column('number', Integer))
t.create()
engine.execute(t.insert().values(id=1, number=3))
engine.execute(t.insert().values(id=9, number=-3))
# create view
createview = CreateView('viewname', t.select().where(t.c.id>5))
engine.execute(createview)
# reflect view and print result
v = Table('viewname', metadata, autoload=True)
for r in engine.execute(v.select()):
print r
If you want, you can also specialize for a dialect, e.g.
#compiles(CreateView, 'sqlite')
def visit_create_view(element, compiler, **kw):
return "CREATE VIEW IF NOT EXISTS %s AS %s" % (
element.name,
compiler.process(element.select, literal_binds=True)
)
stephan's answer is a good one and covers most bases, but what left me unsatisfied was the lack of integration with the rest of SQLAlchemy (the ORM, automatic dropping etc.). After hours of experimenting and piecing together knowledge from all corners of the internet I came up with the following:
import sqlalchemy_views
from sqlalchemy import Table
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.sql.ddl import DropTable
class View(Table):
is_view = True
class CreateView(sqlalchemy_views.CreateView):
def __init__(self, view):
super().__init__(view.__view__, view.__definition__)
#compiles(DropTable, "postgresql")
def _compile_drop_table(element, compiler, **kwargs):
if hasattr(element.element, 'is_view') and element.element.is_view:
return compiler.visit_drop_view(element)
# cascade seems necessary in case SQLA tries to drop
# the table a view depends on, before dropping the view
return compiler.visit_drop_table(element) + ' CASCADE'
Note that I am utilizing the sqlalchemy_views package, just to simplify things.
Defining a view (e.g. globally like your Table models):
from sqlalchemy import MetaData, text, Text, Column
class SampleView:
__view__ = View(
'sample_view', MetaData(),
Column('bar', Text, primary_key=True),
)
__definition__ = text('''select 'foo' as bar''')
# keeping track of your defined views makes things easier
views = [SampleView]
Mapping the views (enable ORM functionality):
Do when loading up your app, before any queries and after setting up the DB.
for view in views:
if not hasattr(view, '_sa_class_manager'):
orm.mapper(view, view.__view__)
Creating the views:
Do when initializing the database, e.g. after a create_all() call.
from sqlalchemy import orm
for view in views:
db.engine.execute(CreateView(view))
How to query a view:
results = db.session.query(SomeModel, SampleView).join(
SampleView,
SomeModel.id == SampleView.some_model_id
).all()
This would return exactly what you expect (a list of objects that each has a SomeModel object and a SampleView object).
Dropping a view:
SampleView.__view__.drop(db.engine)
It will also automatically get dropped during a drop_all() call.
This is obviously a very hacky solution but in my eyes it is the best one and cleanest one out there at the moment. I have tested it these past few days and have not had any issues. I'm not sure how to add in relationships (ran into problems there) but it's not really necessary, as demonstrated above in the query.
If anyone has any input, finds any unexpected issues, or knows a better way to do things, please do leave a comment or let me know.
This was tested on SQLAlchemy 1.2.6 and Python 3.6.
These days there's a PyPI package for that: SQLAlchemy Views.
From it's PyPI Page:
>>> from sqlalchemy import Table, MetaData
>>> from sqlalchemy.sql import text
>>> from sqlalchemy_views import CreateView, DropView
>>> view = Table('my_view', metadata)
>>> definition = text("SELECT * FROM my_table")
>>> create_view = CreateView(view, definition, or_replace=True)
>>> print(str(create_view.compile()).strip())
CREATE OR REPLACE VIEW my_view AS SELECT * FROM my_table
However, you asked for a no "pure SQL" query, so you probably want the definition above to be created with SQLAlchemy query object.
Luckily, the text() in the example above makes it clear that the definition parameter to CreateView is such a query object. So something like this should work:
>>> from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
>>> from sqlalchemy.sql import select
>>> from sqlalchemy_views import CreateView, DropView
>>> metadata = MetaData()
>>> users = Table('users', metadata,
... Column('id', Integer, primary_key=True),
... Column('name', String),
... Column('fullname', String),
... )
>>> addresses = Table('addresses', metadata,
... Column('id', Integer, primary_key=True),
... Column('user_id', None, ForeignKey('users.id')),
... Column('email_address', String, nullable=False)
... )
Here is the interesting bit:
>>> view = Table('my_view', metadata)
>>> definition = select([users, addresses]).where(
... users.c.id == addresses.c.user_id
... )
>>> create_view = CreateView(view, definition, or_replace=True)
>>> print(str(create_view.compile()).strip())
CREATE OR REPLACE VIEW my_view AS SELECT users.id, users.name,
users.fullname, addresses.id, addresses.user_id, addresses.email_address
FROM users, addresses
WHERE users.id = addresses.user_id
SQLAlchemy-utils just added this functionality in 0.33.6 (available in pypi). It has views, materialized views, and it integrates with the ORM. It is not documented yet, but I am successfully using the views + ORM.
You can use their test as an example for both regular and materialized views using the ORM.
To create a view, once you install the package, use the following code from the test above as a base for your view:
class ArticleView(Base):
__table__ = create_view(
name='article_view',
selectable=sa.select(
[
Article.id,
Article.name,
User.id.label('author_id'),
User.name.label('author_name')
],
from_obj=(
Article.__table__
.join(User, Article.author_id == User.id)
)
),
metadata=Base.metadata
)
Where Base is the declarative_base, sa is the SQLAlchemy package, and create_view is a function from sqlalchemy_utils.view.
Loosely based on https://github.com/sqlalchemy/sqlalchemy/wiki/Views
Complete executable example with sqlalchemy only, hope you don't spend hours just to make it run.
import sqlalchemy as sa
import sqlalchemy.schema
import sqlalchemy.ext.compiler
engine = sa.create_engine('postgresql://localhost/postgres')
meta = sa.MetaData()
Session = sa.orm.sessionmaker(bind=engine)
session = Session()
class Drop(sa.schema.DDLElement):
def __init__(self, name, schema):
self.name = name
self.schema = schema
class Create(sa.schema.DDLElement):
def __init__(self, name, select, schema='public'):
self.name = name
self.schema = schema
self.select = select
sa.event.listen(meta, 'after_create', self)
sa.event.listen(meta, 'before_drop', Drop(name, schema))
#sa.ext.compiler.compiles(Create)
def createGen(element, compiler, **kwargs):
return 'CREATE OR REPLACE VIEW {schema}."{name}" AS {select}'.format(
name = element.name,
schema = element.schema,
select = compiler.sql_compiler.process(
element.select,
literal_binds = True
),
)
#sa.ext.compiler.compiles(Drop)
def dropGen(element, compiler, **kw):
return 'DROP VIEW {schema}."{name}"'.format(
name = element.name,
schema = element.schema,
)
if __name__ == '__main__':
view = Create(
name = 'myview',
select = sa.select(sa.literal_column('1 AS col'))
)
meta.create_all(bind=engine, checkfirst=True)
print(session.execute('SELECT * FROM myview').all())
session.close()
I couldn't find an short and handy answer.
I don't need extra functionality of View (if any), so I simply treat a view as an ordinary table as other table definitions.
So basically I have a.py where defines all tables and views, sql related stuff, and main.py where I import those class from a.py and use them.
Here's what I add in a.py and works:
class A_View_From_Your_DataBase(Base):
__tablename__ = 'View_Name'
keyword = Column(String(100), nullable=False, primary_key=True)
Notably, you need to add the primary_key property even though there's no primary key in the view.
SQL View without pure SQL?
You can create a class or function to implement a defined view.
function get_view(con):
return Table.query.filter(Table.name==con.name).first()
The docs for Camelot say that it uses Elixir models. Since SQLAlchemy has included declarative_base for a while, I had used that instead of Elixir for another app. Now I would like to use the SQLAlchemy/declarative models directly in Camelot.
There is a post on Stackoverflow that says Camelot is not tied to Elixir and that using different models would be possible but it doesn't say how.
Camelot's original model.py only has this content:
import camelot.types
from camelot.model import metadata, Entity, Field, ManyToOne, OneToMany, Unicode, Date, Integer, using_options
from camelot.view.elixir_admin import EntityAdmin
from camelot.view.forms import *
__metadata__ = metadata
I added my SQLAlchemy model and changed model.py to this:
import camelot.types
from camelot.model import metadata, Entity, Field, ManyToOne, OneToMany, Unicode, Date, using_options
from camelot.view.elixir_admin import EntityAdmin
from camelot.view.forms import *
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
__metadata__ = metadata
Base = declarative_base()
class Test(Base):
__tablename__ = "test"
id = Column(Integer, primary_key=True)
text = Column(String)
It didn't work. When I start main.py, I can see the GUI and Test in the sidebar, but can't see any rows. This is the tail of the traceback:
File "/usr/lib/python2.6/dist-packages/camelot/view/elixir_admin.py", line 52, in get_query
return self.entity.query
AttributeError: type object 'Test' has no attribute 'query'
This is the elixir_admin.py code for line 46-52:
#model_function
def get_query(self):
""":return: an sqlalchemy query for all the objects that should be
displayed in the table or the selection view. Overwrite this method to
change the default query, which selects all rows in the database.
"""
return self.entity.query
If this code is causing the problem, how do I overwrite the method to change the default query to make it work?
How can you use SQLAlchemy/declarative models in Camelot?
Here is some sample code on using Declarative to define a Movie model for Camelot, some explanation can be found here.
import sqlalchemy.types
from sqlalchemy import Column
from sqlalchemy.ext.declarative import ( declarative_base,
_declarative_constructor )
from camelot.admin.entity_admin import EntityAdmin
from camelot.model import metadata
import camelot.types
from elixir import session
class Entity( object ):
def __init__( self, **kwargs ):
_declarative_constructor( self, **kwargs )
session.add( self )
Entity = declarative_base( cls = Entity,
metadata = metadata,
constructor = None )
class Movie( Entity ):
__tablename__ = 'movie'
id = Column( sqlalchemy.types.Integer, primary_key = True )
name = Column( sqlalchemy.types.Unicode(50), nullable = False )
cover = Column( camelot.types.Image(), nullable = True )
class Admin( EntityAdmin ):
list_display = ['name']
form_display = ['name', 'cover']
Which version of Camelot are you using ?
With the current version of Camelot (11.12.30) it is possible to use Declarative through some
hacks. The upcoming version will make it much easier, while after this, the examples will be
ported to Declarative as well.
I’m working on a library where the user shall be able to simply declare a few classes which are automatically backed by the database. In short, somewhere hidden in the code, there is
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class LibraryBase(Base):
# important library stuff
and the user should then do
class MyStuff(LibraryBase):
# important personal stuff
class MyStuff_2(LibraryBase):
# important personal stuff
mystuff = MyStuff()
Library.register(mystuff)
mystuff.changeIt() # apply some changes to the instance
Library.save(mystuff) # and save it
# same for all other classes
In a static environment, e.g. the user has created one file with all personal classes and imports this file, this works pretty well. All class names are fixed and SQLAlchemy knows how to map each class.
In an interactive environment, things are different: Now, there is a chance of a class being defined twice. Both classes might have different modules; but still SQLAlchemy will complain:
SAWarning: The classname 'MyStuff' is already in the registry of this declarative base, mapped to < class 'OtherModule.MyStuff' >
Is there a way to deal with this? Can I somehow unload a class from its declarative_base so that I can exchange its definition with a new one?
You can use:
sqlalchemy.orm.instrumentation.unregister_class(cl)
del cl._decl_class_registry[cl.__name__]
The first line is to prevent accidental use of your unregisted class. The second unregisters and will prevent the warning.
It looks like, And I'm not really sure this even works, but I think what you want is
sqlalchemy.orm.instrumentation.unregister_class()
http://hg.sqlalchemy.org/sqlalchemy/file/762548ff8eef/lib/sqlalchemy/orm/instrumentation.py#l466
In my project I use this solution.
Where library specified columns defined as mixin by declared_attr and target mapper created by type call with bases, as result I have full functional mapper.
from sqlalchemy import create_engine, BigInteger, Column
from sqlalchemy.orm import sessionmaker, scoped_session
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.declarative import declared_attr
Base = declarative_base()
class LibraryBase(object):
__tablename__ = 'model'
#declared_attr
def library_field(self):
return Column(BigInteger)
class MyLibrary(object):
#classmethod
def register(cls, entity):
tablename = entity.__tablename__
Mapper = type('Entity_%s' % tablename, (Base, LibraryBase, entity), {
'__tablename__': tablename,
'id': Column(BigInteger, primary_key=True),
})
return Mapper
#classmethod
def setup(cls):
Base.metadata.create_all()
class MyStaff(object):
__tablename__ = 'sometable1'
#declared_attr
def staff_field(self):
return Column(BigInteger)
def mymethod(self):
print('My method:', self)
class MyStaff2(MyStaff):
__tablename__ = 'sometable2'
if __name__ == '__main__':
engine = create_engine('sqlite://', echo=True)
Base.metadata.bind = engine
Session = scoped_session(sessionmaker(bind=engine))
session = Session()
# register and install
MyStaffMapper = MyLibrary.register(MyStaff)
MyStaffMapper2 = MyLibrary.register(MyStaff2)
MyLibrary.setup()
MyStaffMapper().mymethod()
MyStaffMapper2().mymethod()
session.query(MyStaffMapper.library_field) \
.filter(MyStaffMapper.staff_field != None) \
.all()