I struggled writing this question. It gets into what may be a complicated and uncommon use case.
I have defined several ORM classes in one project which is responsible for maintaining a common database schema and core functionality. For example, let's say this is the model.email module.
from sqlalchemy import Column, Index, ForeignKey
from sqlalchemy import Boolean, Integer, Text
from . import Base
class CampaignDB(Base):
"""
Note: this basic database mapper class is expected to be extended.
When sub-classing, be mindful to override mappings to other extended classes.
"""
__tablename__ = 'campaigns'
audience_id = Column(Integer, ForeignKey("audiences.id"))
active = Column(Boolean)
name = Column(Text)
These ORM classes are imported into several other projects as a package. In some cases, these ORM classes are subclassed to provide additional functionality. For example, here the CampaignDB class is subclassed to provide support for sending email in a specific project context.
from model.email import CampaignDB
class Campaign(CampaignDB):
"""
Provides sending capability to the email campaign ORM class.
"""
def __init__(self, audience_id=None, active=None, name=None):
self.audience_id = audience_id
self.active = active
self.name = name
def send(self):
print("send emails to the audience")
Now I would like to refactor the CampaignDB and subclassed Campaign classes to be polymorphic bases using SQLAlchemy's Class Inheritance Hierarchies. For example, I'd like make CampaignDB a base class for EmailCampaignDB and PushCampaignDB. I would then like to extend EmailCampaignDB and PushCampaignDB separately, as say, EmailCampaign and PushCampaign in the importing project. However, I would like to still be able to query for Campaign and be returned instances of EmailCampaign and PushCampaign.
I have made several attempts to solve this but run into problems. In particular, session.query(Campaign).all() returns no results because SQLAlchemy doesn't seem to consider it as a base class. The generated SQL has the following WHERE clause: WHERE email.campaigns.type IN (NULL)
Here's the gist of what I am attempting.
class CampaignDB(Base):
"""
Note: this basic database mapper class is expected to be extended.
When sub-classing, be mindful to override mappings to other extended classes.
"""
__tablename__ = 'campaigns'
audience_id = Column(Integer, ForeignKey("audiences.id"))
active = Column(Boolean)
name = Column(Text)
type = Column(String(16))
__mapper_args__ = {
'polymorphic_on': type
}
class EmailCampaignDB(CampaignBaseDB):
__mapper_args__ = {
'polymorphic_identity': 'email'
}
class PushCampaignDB(CampaignBaseDB):
__mapper_args__ = {
'polymorphic_identity': 'push'
}
def send(self):
print("send push notifications to the audience")
class Campaign(CampaignDB):
pass
class EmailCampaign(EmailCampaignDB):
def send(self):
print("send emails to the audience")
class PushCampaign(PushCampaignDB):
def send(self):
print("send push notifications to the audience")
Is this possible? Is there a better way to achieve this in this "packaged ORM" context?
I managed to find a way to make this work by changing the way I think about the problem. I gave up on trying to create and query a Campaign subclass of CampaignDB. I also used the Declarative API which seems to facilitate subclassing EmailCampaignDB and PushCampaignDB.
The model.email module in the core project:
from sqlalchemy.ext.declarative import declared_attr
from sqlalchemy import Column, Index, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy import Boolean, Integer, Text
from . import Base
class CampaignBaseDB(Base):
"""
Note: this basic database mapper class is expected to be extended.
When sub-classing, be mindful to override mappings to other extended classes.
"""
__tablename__ = 'campaign_bases'
#declared_attr
def __mapper_args__(cls):
return {
'polymorphic_on': cls.type,
}
audience_id = Column(Integer, ForeignKey("audiences.id"))
active = Column(Boolean)
name = Column(Text)
type = Column(String(16))
class EmailCampaignDB(CampaignBaseDB):
#declared_attr
def __mapper_args__(cls):
return {
'polymorphic_identity': 'email'
}
class PushCampaignDB(CampaignBaseDB):
#declared_attr
def __mapper_args__(cls):
return {
'polymorphic_identity': 'push'
}
Subclassing the campaign classes in the importing project:
from model.email import EmailCampaignDB, PushCampaignDB
class EmailCampaign(EmailCampaignDB):
def send(self):
print("send emails to the audience")
class PushCampaign(PushCampaignDB):
def send(self):
print("send push notifications to the audience")
Polymorphic query:
for campaign in db.query(CampaignBaseDB).all():
campaign.send()
#> send emails to the audience
#> send push notifications to the audience
This approach does result in "SAWarning: Reassigning polymorphic association for identity", so I still feel like there would be a better way.
Related
The following code is a very simple implementation of a SqlAlchemy ORM with one simple table. The Mytable class tries to inherit from BaseAbstract.
The code throws the following exception:
Message: metaclass conflict: the metaclass of a derived class must be
a (non-strict) subclass of the metaclasses of all its bases
from abc import ABC
from sqlalchemy import Column, Integer, create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
class BaseAbstract(ABC):
"""description of class"""
SQLALCHEMY_DATABASE_URI =\
'mssql+pyodbc://(local)/TestDB?driver=SQL+Server+Native+Client+11.0'
SQLALCHEMY_TRACK_MODIFICATIONS = False
engine = create_engine(SQLALCHEMY_DATABASE_URI, echo=True)
Session = sessionmaker(bind=engine)
session = Session()
Base = declarative_base()
metadata = Base.metadata
class Mytable(Base, BaseAbstract):
__tablename__ = 'myTable'
id = Column(Integer, primary_key=True)
firstNum = Column(Integer, nullable=False)
secondNum = Column(Integer, nullable=False)
If you change the class declaration line to
class Mytable(Base):
the code will work fine. Also if you change class BaseAbstract(ABC): to class BaseAbstract(object): the code will again work fine.
How do I inherit from an abstract class in SQLAlchemy?
Mixing metaclasses is not easy and you should avoid it. SQLAlchemy offers a way to handle abstract base classes or augmenting the base, and on the other hand what you're trying to do looks a lot like a mixin.
You can instruct SQLAlchemy to skip creating a table and a mapper for a class using __abstract__:
Base = declarative_base()
class BaseAbstract(Base):
"""description of class"""
__abstract__ = True
class Mytable(BaseAbstract):
...
You could also augment the Base class:
class BaseAbstract:
"""description of class"""
Base = declarative_base(cls=BaseAbstract)
class Mytable(Base):
...
But in my opinion the easiest solution is to forego using an "abstract base" altogether and think of it as a mixin, as you had done already in a way:
class CommonMixin:
"""description of class"""
Base = declarative_base()
class Mytable(CommonMixin, Base):
...
But if you insist on using an actual abc.ABC abstract base class, register your model classes as virtual subclasses:
class BaseAbstract(ABC):
"""description of class"""
Base = declarative_base()
#BaseAbstract.register
class Mytable(Base):
...
The downside is that #abc.abstractmethod decorated methods are not checked upon instantiating virtual subclasses.
If the above do not fulfill your needs and you want to use ABC for checking that required methods are implemented, you could try and do as the exception instructed and create a new metaclass that is the combination of DeclarativeMeta and ABCMeta:
In [6]: class DeclarativeABCMeta(DeclarativeMeta, abc.ABCMeta):
...: pass
...:
In [7]: Base = declarative_base(metaclass=DeclarativeABCMeta)
In [8]: class BaseAbstract(abc.ABC):
...: #abc.abstractmethod
...: def foo(self):
...: pass
...:
In [13]: class MyTable(Base, BaseAbstract):
...: __tablename__ = 'mytable'
...: id = Column(Integer, primary_key=True)
...:
In [14]: MyTable()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-14-1686a36a17c6> in <module>()
----> 1 MyTable()
TypeError: "Can't instantiate abstract class MyTable with abstract methods foo"
In [18]: class MyOtherTable(Base, BaseAbstract):
...: __tablename__ = 'myothertable'
...: id = Column(Integer, primary_key=True)
...: def foo(self):
...: return 'bar'
...:
In [19]: MyOtherTable()
Out[19]: <__main__.MyOtherTable at 0x7f01b4b592b0>
I cannot vouch for this, though. It might contain more than a few surprises.
I came across the same problem. In addition to the very good answer of Ilja and just because in a comment on their answer you are asking
The reason I am trying to inherit from an abstract class inheriting from ABC is to use isinstance in a function that takes as an argument an object and if that returns true then do additional processing by calling methods defined in the abstract class. If I do not inherit from ABC, Python will not enforce my need to make sure the inherited class implements all the attributes of the abstract class, and if I have missed some in an implementation, then I will get a run-time exception. I would rather get a build exception. That is what ABC gives me. Is this not a good reason to use ABC?
I would like to share the workaround I created before I came here. Besides ABC, Python provides another way to check for the presence of a certain method during class instantiation, i.e. in the __init_subclass__ magic method. I did the following:
class BaseAbstract:
ABSTRACT_METHODS = [
"some_method", "another_method"
]
def __init_subclass__(cls):
for method_name in BaseAbstract.ABSTRACT_METHODS:
method = getattr(cls, method_name, None)
if not method or not callable(method):
raise TypeError(
f"Can't instantiate abstract class {cls.__name__} with "
f"abstract methods {method_name}."
)
Using inspect, you could also check for constraints on the signatures of the "abstract" methods.
It is a matter of argument whether this is the better way to go than using one of Ilja's ideas. I just want to share my approach, not claiming that it was superior.
I see two neat characteristics of my workaround:
I have only one SQLalchemy Base. So I don't have to worry about mutiple Bases, for instance when using alembic's feature to autogenerate migrations or when calling Base.metadata.create_all()
I do not actually have to understand what the problem was. If you have never dealt with metadata conflicts, with my approach is easier to understand why it fixes the problem.
But still, it could be considered some sort of hacky. It also has the characteristic of checking for method presence during class initialization, not during instantiation. This is a limitation, in case you have cascading inheritances, where all the methods are present on the last child, but not on all intermediate classes in the inheritance chain.
You should be able to solve the issue by introducing an intermediate base class for your Mytable. For example:
Base = declarative_base()
metadata = Base.metadata
class IntermediateBase(type(Base), BaseAbstract):
pass
class Mytable(IntermediateBase):
__tablename__ = 'myTable'
id = Column(Integer, primary_key=True)
firstNum = Column(Integer, nullable=False)
secondNum = Column(Integer, nullable=False)
Following one of Ilja Everilä's examples above was really useful for me. Mixing metaclasses can be tricky but I haven't noticed any issue yet. I actually subclass the declarative_base into my parent class where common functions, attributes, abstractmethods can be defined
from abc import abstractmethod, ABCMeta
from typing import Dict, Any
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base, DeclarativeMeta
class DeclarativeABCMeta(DeclarativeMeta, ABCMeta):
"""
AbstractBaseClass Metaclass
This allows us to enforce the #abstractmethods defined on MLRegistryBase
on the underlying sub-classes (Database Tables)
"""
pass
Base = declarative_base(metaclass=DeclarativeABCMeta)
class TableAbstractParentClass(Base):
"""
Helper Class for Database Model Inheritance
"""
# Inform SQLAlchemy This in an AbstractBaseClass Model
__abstract__ = True
#classmethod
#abstractmethod
def __tablename__(cls) -> str:
"""
Every Database Table Needs a Name
"""
pass
#abstractmethod
def to_dict(self) -> Dict[str, Any]:
"""
Example Required Method
"""
pass
class ExampleTable(TableAbstractParentClass):
"""
Example Table
"""
__tablename__ = "example_table"
id = Column(Integer, unique=True, autoincrement=True,
primary_key=True, index=True)
name = Column(String(256), nullable=False)
def to_dict(self) -> Dict[str, Any]:
return {
"id": self.id,
"name": self.name
}
I am trying to sync data between our accounting software and our internal software.
The accounting software runs on MSSQL and our software runs on PGSQL
Is it possible to create a single class with two sources of data and possibly two different structure and a mapping method inside? or is it easier to create two distinct class with a mapping method?
One class:
class test(Base):
__tablename__ = 'test'
def __init__(self, type='MSSQL'):
self.type = type
self.__tablename__ = __tablename__
#classmethod
def DBMSSQL(cls):
return cls('MSSQL')
#classmethod
def DBPGSQL(cls):
return cls('PGSQL')
How to map fields between the two type and how to change the engine used?
Or with two classes:
class testMSSQL(Base):
__tablename__ = 'test'
__table_args__ = {"schema": "dbo"}
to_testPGSQL():
pass
class testPGSQL(Base):
__tablename__ = 'testPG'
to_testMSSQL():
pass
Thanks
I have one model which is the a polymorphic identity of an abstract class
class AbstractModel(Base):
type = Column(String())
__mapper_args__ = {"polymorphic_on": type}
class ModelA(AbstractModel):
__mapper_args__ = {"polymorphic_identity": "model_a"}
class FlaskModel(ModelA):
__mapper_args__ = {"polymorphic_identity": "model_a"}
I need FlaskModel to to have the same polymorphic relationship as ModelA, because FlaskModel has flask specific restraint that can't exist within ModelA (request context, user permissions, etc)
However, when creating the second class, SQLAlchemy throws a warning about having duplicate, which is for good reason since the any queries always points to the FlaskModel, even if they were queried from ModelA.
Any suggestions on accomplishing this? Splitting the code into a package and then importing isn't an option.
I ended up using this route, which worked well and allowed me to create flask specific functions.
class AbstractModel(Base):
type = Column(String())
__mapper_args__ = {"polymorphic_on": type}
class ModelA(AbstractModel):
__mapper_args__ = {"polymorphic_identity": "model_a"}
class FlaskModel(ModelA):
pass
##Event listeners are requireed for the inherited polymorphic relationships
##If additional work needs to be done with different aspect, including after_delete and after_update
##They can als be added here
#event.listens_for(FlaskModel, 'init')
def receive_label_init(target, args, kwargs):
kwargs["type"] = "model_a"
#event.listens_for(FlaskModel, 'mapper_configured')
def receive_label_mapper_configured(mapper, class_):
mapper.polymorphic_identity = "model_a"
I am currently trying to create the following database schema with SQLAlchemy (using ext.declarative):
I have a base class MyBaseClass which provides some common functionality for all of my publicly accessible classes, a mixin class MetadataMixin that provides functionality to query metadata from imdb and store it.
Every class that subclasses MetadataMixin has a field persons which provides a M:N relationship to instances of the Person class, and a field persons_roles which provides a 1:N relationship to an object (one for each subclass) which stores the role a concrete Person plays in the instance of the subclass.
This is an abbreviated version of what my code looks like at the moment:
from sqlalchemy import Column, Integer, Enum, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.ext.associationproxy import association_proxy
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyBaseClass(object):
"""Base class for all publicly accessible classes"""
id = Column(Integer, primary_key=True)
class Person(MyBaseClass):
"""A Person"""
name = Column(Unicode)
movies = association_proxy('movie_roles', 'movie',
creator=lambda m: _PersonMovieRole(movie=m))
shows = association_proxy('show_roles', 'show',
creator=lambda s: _PersonShowRole(show=s=))
class _PersonMovieRole(Base):
"""Role for a Person in a Movie"""
__tablename__ = 'persons_movies'
id = Column(Integer, primary_key=True)
role = Column(Enum('none', 'actor', 'writer', 'director', 'producer'),
default='none')
person_id = Column(Integer, ForeignKey('persons.id'))
person = relationship('Person', backref='movie_roles')
movie_id = Column(Integer, ForeignKey('movies.id'))
movie = relationship('Movie', backref='persons_roles')
class _PersonShowRole(Base):
"""Role for a Person in a Show"""
__tablename__ = 'persons_shows'
id = Column(Integer, primary_key=True)
role = Column(Enum('none', 'actor', 'writer', 'director', 'producer'),
default='none')
person_id = Column(Integer, ForeignKey('persons.id'))
person = relationship('Person', backref='show_roles')
show_id = Column(Integer, ForeignKey('shows.id'))
show = relationship('Episode', backref='persons_roles')
class MetadataMixin(object):
"""Mixin class that provides metadata-fields and methods"""
# ...
persons = association_proxy('persons_roles', 'person',
creator= #...???...#)
class Movie(Base, MyBaseClass, MetadataMixin):
#....
pass
What I'm trying to do is to create a generic creator function for association_proxy that creates either a PersonMovieRole or a PersonShowRole object, depending on the class of the concrete instance that a Person is added to. What I'm stuck on at the moment is that I don't know how to pass the calling class to the creator function.
Is this possible, or is there maybe even an easier way for what I'm trying to accomplish?
By the time your persons field is defined, you cannot really know what class it will end up in. Python takes up ready dictionaries of class members and creates classes out of them (via type.__new__), but when it happens, those members are already fully defined.
So you need to provide the required information directly to the mixin, and tolerate the small duplication it will create in your code. I'd opt for interface similar to this one:
class Movie(Base, MyBaseClass, MetadataMixin('Movie')):
pass
(You cannot have MetadataMixin(Movie) either, for the exact same reasons: Movie requires its base classes to be completely defined by the time the class is created).
To implement such "parametrized class", simply use a function:
def MetadataMixin(cls_name):
"""Mixin class that provides metadata-fields and methods"""
person_role_cls_name = 'Person%sRole' % cls_name
person_role_cls = Base._decl_class_registry[person_role_cls_name]
class Mixin(object):
# ...
persons = association_proxy('persons_roles', 'person',
creator=person_role_cls)
return Mixin
This works because what we're looking up in Base._decl_class_registry - the registry of all classes descending from your declarative base - is not the final class (e.g. Movie), but the association object (e.g. PersonMovieRole).
It will be simplest to explain with code example, in Python I can do so to achieve model inheritance:
"""Image model"""
from sqlalchemy import Column, ForeignKey
from sqlalchemy.types import Integer, String, Text
from miasto_3d.model.meta import Base
class Image(Base):
__tablename__ = "image"
image_id = Column(Integer, primary_key=True)
path = Column(String(200))
def get_mime(self):
#function to get mime type from file
pass
"""WorkImage model"""
class WorkImage(Image, Base):
__tablename__ = "work_images"
image_id = Column(Integer, ForeignKey("image.image_id"), primary_key=True)
work_id = Column(Integer, ForeignKey("work.id"))
work = relation("Work", backref=backref('images',order_by='WorkImage.work_id'))
"""UserAvatar model"""
class UserAvatar(Image, Base):
__tablename__ = "user_avatars"
image_id = Column(Integer, ForeignKey("image.image_id"), primary_key=True)
user_id = Column(Integer, ForeignKey("user.id"))
user = relation("User", backref=backref('images',order_by='UserAvatar.user_id'))
How I do similar things in Rails? Or maybe there is another, better way to do it?
I know paperclip, but I don't like it's conception to use shared table to store photo and model data.
It looks like you're wanting either a polymorphic association or perhaps single table inheritance.
Since you don't define database fields in the model, you cannot inherit database schema in this way - all your fields will need to be specified per table in a migration. You probably should use paperclip, if only because reinventing the wheel is a pain. It works really well, and abstracts away from the actual database structure for you.
In Rails, rather than model inheritance, shared functionality tends to be implemented in modules, like so:
http://handyrailstips.com/tips/14-drying-up-your-ruby-code-with-modules