I'm using SQLAlchemy, and many classes in my object model have the same two attributes: id and (integer & primary key), and name (a string). I'm trying to avoid declaring them in every class like so:
class C1(declarative_base()):
id = Column(Integer, primary_key = True)
name = Column(String)
#...
class C2(declarative_base()):
id = Column(Integer, primary_key = True)
name = Column(String)
#...
What's a good way to do that? I tried using metaclasses but it didn't work yet.
You could factor out your common attributes into a mixin class, and multiply inherit it alongside declarative_base():
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
class IdNameMixin(object):
id = Column(Integer, primary_key=True)
name = Column(String)
class C1(declarative_base(), IdNameMixin):
__tablename__ = 'C1'
class C2(declarative_base(), IdNameMixin):
__tablename__ = 'C2'
print C1.__dict__['id'] is C2.__dict__['id']
print C1.__dict__['name'] is C2.__dict__['name']
EDIT: You might think this would result in C1 and C2 sharing the same Column objects, but as noted in the SQLAlchemy docs, Column objects are copied when originating from a mixin class. I've updated the code sample to demonstrate this behavior.
Could you also use the Column's copy method? This way, fields can be defined independently of tables, and those fields that are reused are just field.copy()-ed.
id = Column(Integer, primary_key = True)
name = Column(String)
class C1(declarative_base()):
id = id.copy()
name = name.copy()
#...
class C2(declarative_base()):
id = id.copy()
name = name.copy()
#...
I think I got it to work.
I created a metaclass that derives from DeclarativeMeta, and made that the metaclass of C1 and C2. In that new metaclass, I simply said
def __new__(mcs, name, base, attr):
attr['__tablename__'] = name.lower()
attr['id'] = Column(Integer, primary_key = True)
attr['name'] = Column(String)
return super().__new__(mcs, name, base, attr)
And it seems to work fine.
Related
Python 3.6 and SQLAlchemy 1.2.
I have a package called events which defines Match as a type of Event and uses joined table inheritance to distinguish it from other types of Event. The other type of event is Competition, all events are one or the other:
class Event(Base):
__tablename__ = 'tbl_events'
event_id = Column(Integer, primary_key=True)
event_type_id = Column(String(50)) # 0 is Match, 1 is Competition
__mapper_args__ = {'polymorphic_on': event_type_id}
class Match(Event):
__tablename__ = 'tbl_match_details'
event_id = Column(Integer,
ForeignKey('tbl_events.event_id'),
primary_key=True)
team_1 = Column(String(50))
team_2 = Column(String(50))
__mapper_args__ = {'polymorphic_identity': 0}
I'm using Match in another package which distinguishes multiple types of Match and relies on the Match object's attribs and methods to pull event info from the database but operates away from the database otherwise to that:
from events import Match
class BaseMatch(Match):
# define common methods and attrs
class TennisMatch(BaseMatch):
# do Tennis based stuff
class FootballMatch(BaseMatch):
# do football based things
Any difference between events.Match and the classes that inherit from it only matter in this package and this package doesn't otherwise insert or update the database, only reads from it.
The issue I'm having is that attempting to instantiate an instance of any of the classes that inherits from Match results in a NULL value being passed into the query for the event_type_id field. This is the WHERE part of the query:
WHERE tbl_match_details.event_id = %s AND tbl_events.match_comp_id IN (NULL)
I can't simply give each class their own polymorphic identifier as those identifiers won't exist in the database.
I tried this:
class BaseMatch(Match):
#declared_attr
def __mapper_args__(cls):
return {'polymorphic_identity': 0}
class TennisMatch(BaseMatch):
# do tennis stuff
class FootballMatch(BaseMatch):
# do footy stuff
but importing the module, I get warnings like:
SAWarning: Reassigning polymorphic association for identity 0 from <Mapper at 0x7f80197f0550; Match> to <Mapper at 0x7f80197a9fd0; BaseModel>: Check for duplicate use of 0 as value for polymorphic_identity.
SAWarning: Reassigning polymorphic association for identity 0 from <Mapper at 0x7f80197a9fd0; BaseModel> to <Mapper at 0x7f800dfdf940; TennisMatch>: Check for duplicate use of 0 as value for polymorphic_identity.
I get one of those for each class that inherits from Match and when I attempt to instantiate any of the match types, I get an instance of the type last to have been associated with that polymorphic id.
I'd really appreciate a nudge in the right direction!
Thanks.
Here's what I've done to work around this - I'm not sure if it is 'right' but it has allowed me to move forward with what I'm doing and helped me to understand a bit more of the goings on under the hood.
I've created a factory methods on my Event, Competition and Match classes and a class attribute on Competition and Match that gives me access to each event type's event_type_id value:
from sqlalchemy import inspect
class Event(Base):
__tablename__ = 'tbl_events'
event_id = Column(Integer, primary_key=True)
event_type_id = Column(String(50)) # 0 is Match, 1 is Competition
__mapper_args__ = {'polymorphic_on': event_type_id}
#classmethod
def from_id(cls, id, session):
mapper = inspect(cls).mapper
mapper.polymorphic_map[cls.EVENT_TYPE_ID] = mapper
mapper.polymorphic_identity = cls.EVENT_TYPE_ID
return session.query(cls).filter_by(event_id=id).one()
class Match(Event):
EVENT_TYPE_ID = 0
__tablename__ = 'tbl_match_details'
event_id = Column(Integer,
ForeignKey('tbl_events.event_id'),
primary_key=True)
team_1 = Column(String(50))
team_2 = Column(String(50))
__mapper_args__ = {'polymorphic_identity': EVENT_TYPE_ID}
This way, whenever classes that inherit from Match or Competition are instantiated using the factory methods, the polymorphic identity is forced to the identity defined on the parent class and the polymorphic map points that identity to the class that the factory is being called upon.
A disadvantage obvious to me is that this will only work when objects are instantiated through the factory methods. Fine in this case but maybe not for all.
Would appreciate any feedback on how I've gone about this and any pointers toward a cleaner solution.
Thanks
I have two models and one-to-one relationship between them. I would like to access columns of ItemData through Item (e.g. item = Item(); item.value). I tried to overwrote __getattr__, but this method is intensively used by SQLAlachemy base model. Any help appreciated.
class Item(Model):
__tablename__ = "item_data"
id = Column(Integer, primary_key=True)
data = relationship("ItemData", back_populates="_item", uselist=False,
foreign_keys="ItemData._item_id")
class ItemData(Model):
__tablename__ = "items_data"
id = Column(Integer, primary_key=True)
_item_id = Column(Integer, ForeignKey("items.id"))
_item = relationship("Item", back_populates="data",
foreign_keys=[_item_id])
value = Column(Integer)
I solved the problem by writing class decorator which should be apply to Item.
from sqlalchemy.ext.hybrid import hybrid_property
def direct_data_getter(cls):
'''Provide direct access to attributes of data.'''
fields = set(cls.data_cls.__mapper__.columns.keys()) - set(dir(cls))
for field in fields:
method = hybrid_property(
lambda self, field=field: getattr(self.data, field)
)
setattr(cls, field, method)
return cls
I'm working on an application that uses sqlalchemy to pull a lot of data. The application does a lot of computations on the resulting objects, but never makes any changes to those objects. The application is too slow, and profiling suggests a lot of the time is spent accessing attributes that are managed by sqlalchemy. Therefore, I'm trying to figure out how to prevent sqlalchemy from instrumenting these objects.
So far, I've made a little progress based on this answer and this example. I've found two different ways to do it, but neither one seems to work with relationships. I'm mostly proceeding by poking around without any comprehensive understanding of the internals of sqlalchemy. In both cases, I'm using the following test session and data:
engine = create_engine('sqlite://')
engine.execute('CREATE TABLE foo (id INTEGER, name TEXT)')
engine.execute('INSERT INTO foo VALUES (0, \'Jeremy\')')
engine.execute('CREATE TABLE bar (id INTEGER, foo_id Integer)')
engine.execute('INSERT INTO bar VALUES (0, 0)')
engine.execute('INSERT INTO bar VALUES (1, 0)')
Session = sessionmaker(bind=engine)
session = Session()
Then I either (1) use a custom subclass of InstrumentationManager:
class ReadOnlyInstrumentationManager(InstrumentationManager):
def install_descriptor(self, class_, key, inst):
pass
Base = declarative_base()
Base.__sa_instrumentation_manager__ = ReadOnlyInstrumentationManager
class Bar(Base):
__tablename__ = 'bar'
id = Column(Integer, primary_key=True)
foo_id = Column(Integer, ForeignKey('foo.id'))
class Foo(Base):
__tablename__ = 'foo'
id = Column(Integer, primary_key=True)
name = Column(String(32))
bars = relationship(Bar)
f = session.query(Foo).first()
print type(f.bars)
which gives:
<class 'sqlalchemy.orm.relationships.RelationshipProperty'>
instead of the expected list of bars.
Or, (2) use a custom subclass of ClassManager:
class MyClassManager(ClassManager):
def new_instance(self, state=None):
if hasattr(self.class_, '__readonly_type__'):
instance = self.class_.__readonly_type__.__new__(self.class_.__readonly_type__)
else:
instance = self.class_.__new__(self.class_)
self.setup_instance(instance, state)
return instance
Base = declarative_base()
Base.__sa_instrumentation_manager__ = MyClassManager
class ReadonlyFoo(object):
pass
class ReadonlyBar(object):
pass
class Bar(Base, ReadonlyBar):
__tablename__ = 'bar'
__readonly_type__ = ReadonlyBar
id = Column(Integer, primary_key=True)
foo_id = Column(Integer, ForeignKey('foo.id'))
class Foo(Base, ReadonlyFoo):
__tablename__ = 'foo'
__readonly_type__ = ReadonlyFoo
id = Column(Integer, primary_key=True)
name = Column(String(32))
bars = relationship(Bar)
f = session.query(Foo).first()
print f.bars
which gives:
AttributeError: 'ReadonlyFoo' object has no attribute 'bars'
Is there a way to modify one of these approaches so that relationships still work? Or, is there another approach to this problem that's better?
I'm working on a SQLAlchemy defining a bunch of mixin classes that applications should be able to import and extend their model.
When looking at the documentation, mixin classes are create knowing the final table name however, in the case of a generic library, the final table name that will be used by the application is not known.
Take the following mixin classes:
import sqlalchemy as sa
class UserMixin(object):
id = sa.Column(sa.Integer(), primary_key=True)
first_name = sa.Column(sa.Unicode(255))
last_name = sa.Column(sa.Unicode(255))
class ItemMixin(object):
id = sa.Column(sa.Integer(), primary_key=True)
name = sa.Column(sa.Unicode(255))
short_description = sa.Column(sa.Unicode(255))
class OrdersMixin(object):
id = sa.Column(sa.Integer(), primary_key=True)
user_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
item_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
Then an application defining its models:
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyUser(UserMixin, Base):
__tablename__ = 'myuser'
class MyItem(ItemMixin, Base):
__tablename__ = 'myitem'
total = sa.Column(sa.Integer())
class MyOrders(OrdersMixin, Base):
__tablename__ = 'myorders'
I have two issues with this model:
Except from redefining the relationship columns in the extending models, how can the mixin class build the relationship on its own.
Type of the foreign key is assumed by the mixin class, but the id of the table may come from the application itself or from another mixin class.
Is the model I'm trying to implement correct? What would be the right way to tackle this problem?
A Mixin is a class that copies its data into a table, but for SQL it matters if you're the owner of that data (the table) vs being a reference (the foreign key).
It looks like you're attempting to create Mixins that are both sources of truth and references. Which isn't possible in SQL.
Taking your example one step further and defining OrdersMixin like this will make the issues more obvious I think.
class OrdersMixin(UserMixin, ItemMixin):
id = sa.Column(sa.Integer(), primary_key=True)
For example MyOrders would end up like this once things are resolved.
class MyOrders(Base):
__tablename__ = 'myorders'
# This is from UserMixin
id = sa.Column(sa.Integer(), primary_key=True)
first_name = sa.Column(sa.Unicode(255))
last_name = sa.Column(sa.Unicode(255))
# This is from ItemMixin
id = sa.Column(sa.Integer(), primary_key=True)
name = sa.Column(sa.Unicode(255))
short_description = sa.Column(sa.Unicode(255))
# From OrdersMixin
id = sa.Column(sa.Integer(), primary_key=True) # This is defined last so it overrides all others with the same name.
user_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
item_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
With how you have the Mixin defined any table that used that Mixin would have primary_keys for the id column, which would conflict. Additionally you are duplicating every column in the Mixin, which in general you want to avoid in SQL (see Database normal form).
The final result would be something like this. Which is a whole bunch of columns meaning you wouldn't need to refer to any other tables and all of the references id you had were overwritten, meaning you wouldn't be able to join them anyway.
class MyOrders(Base):
__tablename__ = 'myorders'
first_name = sa.Column(sa.Unicode(255))
last_name = sa.Column(sa.Unicode(255))
name = sa.Column(sa.Unicode(255))
short_description = sa.Column(sa.Unicode(255))
id = sa.Column(sa.Integer(), primary_key=True) # This is defined last so it overrides all others with the same name.
user_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
item_id = sa.Column(sa.Integer(), sa.ForeignKey('???'))
To avoid that I keep my Mixins separate from initial table definition. I.e. I use a Mixin for when I want another table to refer to that table.
The following is close to what I think you were hoping to achieve.
import sqlalchemy as sa
from sqlalchemy import orm
class UserMixin(object):
user_id = sa.Column(sa.Integer(), ForeignKey("myuser.id"), index=True)
user = orm.relationship("MyUser")
class ItemMixin(object):
item_id = sa.Column(sa.Integer(), ForeignKey("myitem.id"), index=True)
item = orm.relationship("MyItem")
class OrdersMixin(UserMixin, ItemMixin):
order_id = sa.Column(sa.Integer(), sa.ForeignKey('myorders.id'))
user_id = sa.Column(sa.Integer(), sa.ForeignKey('myorders.user_id'))
item_id = sa.Column(sa.Integer(), sa.ForeignKey('myorders.item_id'))
Note in the Mixins I gave every column a unique name so that there aren't conflicts and in OrdersMixin even though I'm using UserMixin and ItemMixin I'm overriding the user_id and item_id columns because otherwise anything using the OrdersMixin would have foreign keys pointing to three different tables which would confuse the automatic query builder. But it will still add the user and item relations (and since they are defined as foreign keys to the original tables in MyOrders table I think the relationship will just work).
Then I would change your tables to look like this.
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyUser(Base):
__tablename__ = "myuser"
id = sa.Column(sa.Integer(),primary_key=True)
first_name = sa.Column(sa.Unicode(255))
last_name = sa.Column(sa.Unicode(255))
class MyItem(Base):
__tablename__ = "myitem"
id = sa.Column(sa.Integer(),primary_key=True)
name = sa.Column(sa.Unicode(255))
short_description = sa.Column(sa.Unicode(255))
class MyOrders(Base, UserMixin, OrdersMixin):
__tablename__ = "myorders"
id = sa.Column(sa.Integer(),primary_key=True)
The original table definition owns the columns (source of truth) defining them individually and Mixins (of this kind) are good to define references so subsequent references don't need define each of them individually. A Mixin can't be defined to be both a reference and a source of truth. In light of that instead of overriding the column each time like OrdersMixin it's better to just define it once canonically (the table) and once as a reference (the Mixin).
I have a SyncEntities class (shown below).
I have several other classes (such as CommodityTypes also shown below) related to the SyncEntities class.
All of my Base subclasses have this column uuidKey = Column(String, primary_key=True)
Assume se is an instance of SyncEntities.
se.entityKind is the name of a Base subclass.
How do I query for an object that is in the se.entityKind class filtering for se.uuidKey?
class SyncEntities(Base):
__tablename__ = 'SyncEntities'
uuidKey = Column(String, primary_key=True)
dateCreated = Column(DateTime, index=True)
dateModified = Column(DateTime, index=True)
dateSynced = Column(DateTime, index=True)
username = Column(String)
entityKind = Column(String)
deleted = Column(Boolean)
def __init__(self, entity, security):
self.uuidKey = newUUID()
self.dateCreated = security.now
self.dateModified = security.now
self.dateSynced = security.then
self.username = security.username
self.entityKind = entity.__tablename__
self.deleted = False
def modified(self, security):
self.dateModified = security.now
self.username = security.username
class CommodityTypes(Base):
__tablename__ = 'CommodityTypes'
uuidKey = Column(String, ForeignKey('SyncEntities.uuidKey'), primary_key=True)
myName = Column(String, unique = True)
sortKey = Column(Integer, unique = True)
mySyncEntity = relationship("SyncEntities")
def __init__(self, security, myName, sortKey):
self.syncEntity = SyncEntities(self, security)
self.uuidKey = self.syncEntity.uuidKey
self.myName = myName
self.sortKey = sortKey
The structure here is similar, though not quite the same, as a "polymorphic association", and you can read about this pattern over at this blog post: http://techspot.zzzeek.org/2007/05/29/polymorphic-associations-with-sqlalchemy/ . It's an old post but the example at http://techspot.zzzeek.org/files/2007/discriminator_on_association.py was added later as an updated example.
This case is a little different in that an object like CommodityTypes only refers to a single SyncEntities, not multiple as in the usual polymorphic association. The SyncEntities also can only refer to a single type of related object since you have entityKind on it locally.
I would note that a potential problem with this design is that you could have rows in other tables that have a uuidKey pointing to a particular SyncEntities instance, but are not of a type that matches "entityKind". If the relationship between CommodityTypes and SyncEntities is actually one-to-one, that changes everything - this pattern is really simple joined table inheritance and you'd use the patterns described at http://docs.sqlalchemy.org/en/rel_0_7/orm/inheritance.html.
You also don't have backrefs between the target and SyncEntities, which is often a way to automate these styles of lookup. But you can still approximate things using a lookup of entityKind types to classes:
def lookup_related(se):
types = {
'commodity':CommodityTypes,
'foobar':FooBarTypes
}
cls = types[se.entityKind]
session = object_session(se)
return session.query(cls).filter(cls.mySyncEntity==se).all()
here's a mixin that could do it also, using a backref:
class HasSyncEntity(object):
entity_kind = None
"subclasses need to populate this"
#declared_attr
def uuidKey(cls):
return Column(String, ForeignKey("SyncEntities.uuidKey"), primary_key=True)
#declared_attr
def mySyncEntity(cls):
return relationship("SyncEntities", backref="_%s_collection" % cls.entity_kind)
CommodityTypes becomes:
class CommodityTypes(HasSyncEntity, Base):
entity_kind = "commodity"
# ...
You then add a method like this to SyncEntities, which looks up the appropriate backref, and you're done:
def get_related(self):
return getattr(self, "_%s_collection" % self.entityKind)