I'm trying to get my head around the "Discriminator on association" example shipped with SQLAlchemy, which defines HasAddresses mixin so each model subclassing HasAddresses magically gets an addresses attribute, which is a collection to which Address objects can be added. The linking is performed through an intermediate table so at the first glance the relationship looks like many-to-many, I hoped to be able to have multiple Addresses linked to a Customer, AND also multiple Customers and Suppliers linked to an Address.
The Address model, however, is set up in such a way that it has a single parent attribute which can only reference a single object. So, in the example, an Address can only be linked to a single Customer or Supplier.
How do I modify that example so Address is able to back-reference multiple parent objects?
we can modify sqlalchemy/examples/generic_associations/table_per_association.py to add a named backref to Address, then a #property which rolls up all backrefs created.
"""table_per_association.py
The HasAddresses mixin will provide a new "address_association" table for
each parent class. The "address" table will be shared
for all parents.
This configuration has the advantage that all Address
rows are in one table, so that the definition of "Address"
can be maintained in one place. The association table
contains the foreign key to Address so that Address
has no dependency on the system.
"""
from sqlalchemy.ext.declarative import declarative_base, declared_attr
from sqlalchemy import create_engine, Integer, Column, \
String, ForeignKey, Table
from sqlalchemy.orm import Session, relationship
import itertools
class Base(object):
"""Base class which provides automated table name
and surrogate primary key column.
"""
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
id = Column(Integer, primary_key=True)
Base = declarative_base(cls=Base)
class Address(Base):
"""The Address class.
This represents all address records in a
single table.
"""
street = Column(String)
city = Column(String)
zip = Column(String)
#property
def all_owners(self):
return list(
itertools.chain(
*[
getattr(self, attr)
for attr in [a for a in dir(self) if a.endswith("_parents")]
]
))
def __repr__(self):
return "%s(street=%r, city=%r, zip=%r)" % \
(self.__class__.__name__, self.street,
self.city, self.zip)
class HasAddresses(object):
"""HasAddresses mixin, creates a new address_association
table for each parent.
"""
#declared_attr
def addresses(cls):
address_association = Table(
"%s_addresses" % cls.__tablename__,
cls.metadata,
Column("address_id", ForeignKey("address.id"),
primary_key=True),
Column("%s_id" % cls.__tablename__,
ForeignKey("%s.id" % cls.__tablename__),
primary_key=True),
)
return relationship(Address, secondary=address_association,
backref="%s_parents" % cls.__name__.lower())
class Customer(HasAddresses, Base):
name = Column(String)
class Supplier(HasAddresses, Base):
company_name = Column(String)
engine = create_engine('sqlite://', echo=True)
Base.metadata.create_all(engine)
session = Session(engine)
a1 = Address(
street='123 anywhere street',
city="New York",
zip="10110")
a2 = Address(
street='40 main street',
city="San Francisco",
zip="95732")
session.add_all([
Customer(
name='customer 1',
addresses=[a1, a2]
),
Supplier(
company_name="Ace Hammers",
addresses=[a1]
),
])
session.commit()
for customer in session.query(Customer):
for address in customer.addresses:
print address.all_owners
Related
Problem: Simply put, I am trying to redefine a SQLAlchemy ORM table's primary key after it has already been defined.
Example:
class Base:
#declared_attr
def __tablename__(cls):
return f"{cls.__name__}"
#declared_attr
def id(cls):
return Column(Integer, cls.seq, unique=True,
autoincrement=True, primary_key=True)
Base = declarative_base(cls=Base)
class A_Table(Base):
newPrimaryKeyColumnsDerivedFromAnotherFunction = []
# Please Note: as the variable name tries to say,
# these columns are auto-generated and not known until after all
# ORM classes (models) are defined
# OTHER CLASSES
def changePriKeyFunc(model):
pass # DO STUFF
# Then do
Base.metadata.create_all(bind=arbitraryEngine)
# After everything has been altered and tied into a little bow
*Please note, this is a simplification of the true problem I am trying to solve.
Possible Solution: Your first thought might have been to do something like this:
def possibleSolution(model):
for pricol in model.__table__.primary_key:
pricol.primary_key = False
model.__table__.primary_key = PrimaryKeyConstraint(
*model.newPrimaryKeyColumnsDerivedFromAnotherFunction,
# TODO: ADD all the columns that are in the model that are also a primary key
# *[col for col in model.__table__.c if col.primary_key]
)
But, this doesn't work, because when trying to add, flush, and commit, an error gets thrown:
InvalidRequestError: Instance <B_Table at 0x104aa1d68> cannot be refreshed -
it's not persistent and does not contain a full primary key.
Even though this:
In [2]: B_Table.__table__.primary_key
Out[2]: PrimaryKeyConstraint(Column('a_TableId', Integer(),
ForeignKey('A_Table.id'), table=<B_Table>,
primary_key=True, nullable=False))
as well as this:
In [3]: B_Table.__table__
Out[3]: Table('B_Table', MetaData(bind=None),
Column('id', Integer(), table=<B_Table>, nullable=False,
default=Sequence('test_1', start=1, increment=1,
metadata=MetaData(bind=None))),
Column('a_TableId', Integer(),
ForeignKey('A_Table.id'), table=<B_Table>,
primary_key=True, nullable=False),
schema=None)
and finally:
In [5]: b.a_TableId
Out[5]: 1
Also note that the database actually reflects the changed (and true) primary key, so I know that there's something going on with the ORM/SQLAlchemy.
Question: In summary, how can I change the model's primary key after the model has already been defined?
edit: See below for full code (same type of error, just in SQLite)
from sqlalchemy import Column, Integer, ForeignKey
from sqlalchemy.orm import relationship, sessionmaker
from sqlalchemy.ext.declarative import declared_attr, declarative_base
from sqlalchemy.schema import PrimaryKeyConstraint
from sqlalchemy import Sequence, create_engine
class Base:
#declared_attr
def __tablename__(cls):
return f"{cls.__name__}"
#declared_attr
def seq(cls):
return Sequence("test_1", start=1, increment=1)
#declared_attr
def id(cls):
return Column(Integer, cls.seq, unique=True, autoincrement=True, primary_key=True)
Base = declarative_base(cls=Base)
def relate(model, x):
"""Model is the original class, x is what class needs to be as
an attribute for model"""
attributeName = x.__tablename__
idAttributeName = "{}Id".format(attributeName)
setattr(model, idAttributeName,
Column(ForeignKey(x.id)))
setattr(model, attributeName,
relationship(x,
foreign_keys=getattr(model, idAttributeName),
primaryjoin=getattr(
model, idAttributeName) == x.id,
remote_side=x.id
)
)
return model.__table__.c[idAttributeName]
def possibleSolution(model):
if len(model.defined):
newPriCols = []
for x in model.defined:
newPriCols.append(relate(model, x))
for priCol in model.__table__.primary_key:
priCol.primary_key = False
priCol.nullable = True
model.__table__.primary_key = PrimaryKeyConstraint(
*newPriCols
# TODO: ADD all the columns that are in the model that are also a primary key
# *[col for col in model.__table__.c if col.primary_key]
)
class A_Table(Base):
pass
class B_Table(Base):
defined = [A_Table]
possibleSolution(B_Table)
engine = create_engine('sqlite://')
Base.metadata.create_all(bind=engine)
Session = sessionmaker(bind=engine)
session = Session()
a = A_Table()
b = B_Table(A_TableId=a.id)
print(B_Table.__table__.primary_key)
session.add(a)
session.commit()
session.add(b)
session.commit()
Originally, the error you say the PK reassignment is causing is:
InvalidRequestError: Instance <B_Table at 0x104aa1d68> cannot be refreshed -
it's not persistent and does not contain a full primary key.
I don't get that running you MCVE, instead I get a pretty helpful warning first:
SAWarning: Column 'B_Table.A_TableId' is marked as a member of the
primary key for table 'B_Table', but has no Python-side or server-side
default generator indicated, nor does it indicate 'autoincrement=True'
or 'nullable=True', and no explicit value is passed. Primary key
columns typically may not store NULL.
And a very detailed exception message when the script fails:
sqlalchemy.orm.exc.FlushError: Instance has
a NULL identity key. If this is an auto-generated value, check that
the database table allows generation of new primary key values, and
that the mapped Column object is configured to expect these generated
values. Ensure also that this flush() is not occurring at an
inappropriate time, such as within a load() event.
So assuming that the example accurately describes your problem, the answer is straightforward. A primary key cannot be null.
A_Table inherits off Base:
class A_Table(Base):
pass
Base gives A_Table an autoincrement PK through declared_attr id():
#declared_attr
def id(cls):
return Column(Integer, cls.seq, unique=True, autoincrement=True, primary_key=True)
Similarly, B_Table is defined off Base but the PK is overwritten in possibleSolution() such that it becomes a ForeignKey to A_Table:
PrimaryKeyConstraint(Column('A_TableId', Integer(), ForeignKey('A_Table.id'), table=<B_Table>, primary_key=True, nullable=False))
Then, we instantiate an instance of A_Table without any kwargs and immediately allocate the id attribute of instance a to field A_TableId when constructing b:
a = A_Table()
b = B_Table(A_TableId=a.id)
At this point we can stop and inspect the attribute values of each:
print(a.id, b.A_TableId)
# None None
a.id is None because it's an autoincrement which needs to be populated by the database, not the ORM. So SQLAlchemy doesn't know it's value until after the instance is flushed to the database.
So what happens if we include a flush() operation after adding instance a to the session:
a = A_Table()
session.add(a)
session.flush()
b = B_Table(A_TableId=a.id)
print(a.id, b.A_TableId)
# 1 1
So by issuing the flush first, we've got a value for a.id, meaning that we also have a value for b.A_TableId.
session.add(b)
session.commit()
# no error
I have an Order - FoodItem Many-To-Many relationship that is as follows:
association_table = db.Table('association', db.Model.metadata,
db.Column('left_id', db.Integer, db.ForeignKey('orders.order_id')),
db.Column('right_id', db.Integer, db.ForeignKey('fooditems.fooditem_id'))
)
class OrderModel(ReviewableModel):
__tablename__ = 'orders'
order_id = db.Column(db.Integer, db.ForeignKey('reviewables.id'), primary_key=True)
food_items = db.relationship("FoodItemModel", secondary = association_table)
__mapper_args__ = {'polymorphic_identity':'orders'}
class FoodItemModel(ReviewableModel):
__tablename__ = 'fooditems'
fooditem_id = db.Column(db.Integer, db.ForeignKey('reviewables.id'), primary_key=True)
__mapper_args__ = {'polymorphic_identity':'fooditems'}
The user can request an order with duplicate foodItems. This is created properly, but when I save the changes to the database, the duplicates are removed. For e.g., I order 3 Pizzas:
def save_to_db(self):
print('before: '+str(self.food_items))
db.session.add(self)
db.session.commit()
print('after: '+str(self.food_items))
The output is like this:
before: [<FoodItemModel u'Pizza'>, <FoodItemModel u'Pizza'>, <FoodItemModel u'Pizza'>]
after: [<FoodItemModel u'Pizza'>]
The association table is updated properly:
"left_id" "right_id"
"6" "3"
"6" "3"
"6" "3"
However, the food_items in the OrderModel only contains 1 item
What Juan Mellado was getting at in his answer is that the relational data (RD) and object relational mapping (ORM) clash: the ORM cannot distinguish separate objects that have the same data. To solve this, simply add an id column as primary key to the association_table - that way the ORM has something to distinguish different records with the same left_id and right_id.
But that would be a workaround and not a solution.
The solution is in thinking about what it means when "The user can request an order with duplicate foodItems". The relation from the order to the food is not direct, it is indirect via an order-item. Each order-item belongs to an order (which in turn belongs to a customer or a dining-table) and each order-item can have a relation with a food item. By making each order-item unique, the problem of "duplicate food-items" disappears. At the same time, we can now have an infinite amount of variations of the food-item by adding an optional "customer request" to each order item. E.g. "food: fries, request: easy on the salt".
Below a demonstration in code where customer "I scream" places 1 order with 3 portions of "ice cream" of which 1 portion is "with sprinkles on top".
from sqlalchemy import Column, Integer, String, ForeignKey, create_engine
from sqlalchemy.orm import relationship, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.pool import StaticPool
Base = declarative_base()
class Order(Base):
__tablename__ = 'order'
id = Column(Integer, primary_key=True)
customer = Column(String(127))
items = relationship("OrderItem")
def __repr__(self):
return "<Order(id='{}', customer='{}', items='{}')>".format(self.id, self.customer, self.items)
class Food(Base):
__tablename__ = 'food'
id = Column(Integer, primary_key=True)
name = Column(String(127))
def __repr__(self):
return "<Food(id='{}', name='{}')>".format(self.id, self.name)
class OrderItem(Base):
__tablename__ = 'order_item'
id = Column(Integer, primary_key=True)
order_id = Column(Integer, ForeignKey(Order.id))
order = relationship(Order)
food_id = Column(Integer, ForeignKey(Food.id))
food = relationship(Food)
comment = Column(String(127))
def __repr__(self):
return "<OrderItem(id='{}', order_id='{}', food_id='{}, comment={}')>" \
.format(self.id, self.order_id, self.food_id, self.comment)
def orderFood():
engine = create_engine('sqlite:///:memory:', echo=True, connect_args={'check_same_thread':False}, poolclass=StaticPool)
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
session = Session()
food = Food(name='ice cream')
session.add(food)
order = Order(customer='I scream')
session.add(order)
session.commit()
print("Food: {}".format(food))
print("Order: {}".format(order))
order.items = [OrderItem(order=order, food=food), OrderItem(order=order, food=food), \
OrderItem(order=order, food=food, comment='with sprinkles on top')]
session.merge(order)
session.commit()
print("Order: {}".format(order))
print("Order.items")
for item in order.items:
print(item)
print("OrderItems for order")
orderFilter = OrderItem.order_id == order.id
for order_item in session.query(OrderItem).filter(orderFilter).all():
print(order_item)
print("Food in order")
for row in session.query(Food).join(OrderItem).filter(orderFilter).all():
print(row)
session.close();
if __name__ == "__main__":
orderFood()
You must declare a primary key for the association table.
Flask-SQLALchemy is an ORM, and it needs a series of columns that uniquely identify a row.
Take a look to this part of the documentation, a bit outdated, but still valid:
http://docs.sqlalchemy.org/en/rel_1_1/faq/ormconfiguration.html#faq-mapper-primary-key
Flask-SQLALchemy is using all the fields (left_id, right_id) to identify the rows, and all the rows have the same values (6, 3). So, all the rows are stored in the database (as there are not any declared constraint on it), but only one is retained in the context (memory).
I'm trying to work out how to get an association proxy on a many to many relationship to return an empty list by default.
So based on the example at:
https://raw.githubusercontent.com/zzzeek/sqlalchemy/master/examples/generic_associations/discriminator_on_association.py
We have:
from sqlalchemy.ext.declarative import as_declarative, declared_attr
from sqlalchemy import create_engine, Integer, Column, \
String, ForeignKey
from sqlalchemy.orm import Session, relationship, backref
from sqlalchemy.ext.associationproxy import association_proxy
#as_declarative()
class Base(object):
"""Base class which provides automated table name
and surrogate primary key column.
"""
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
id = Column(Integer, primary_key=True)
class AddressAssociation(Base):
"""Associates a collection of Address objects
with a particular parent.
"""
__tablename__ = "address_association"
discriminator = Column(String)
"""Refers to the type of parent."""
__mapper_args__ = {"polymorphic_on": discriminator}
class Address(Base):
"""The Address class.
This represents all address records in a
single table.
"""
association_id = Column(Integer, ForeignKey("address_association.id"))
street = Column(String)
city = Column(String)
zip = Column(String)
association = relationship("AddressAssociation", backref="addresses")
parent = association_proxy("association", "parent")
def __repr__(self):
return "%s(street=%r, city=%r, zip=%r)" % \
(self.__class__.__name__, self.street,
self.city, self.zip)
class HasAddresses(object):
"""HasAddresses mixin, creates a relationship to
the address_association table for each parent.
"""
#declared_attr
def address_association_id(cls):
return Column(Integer, ForeignKey("address_association.id"))
#declared_attr
def address_association(cls):
name = cls.__name__
discriminator = name.lower()
assoc_cls = type(
"%sAddressAssociation" % name,
(AddressAssociation, ),
dict(
__tablename__=None,
__mapper_args__={
"polymorphic_identity": discriminator
}
)
)
cls.addresses = association_proxy(
"address_association", "addresses",
creator=lambda addresses: assoc_cls(addresses=addresses)
)
return relationship(assoc_cls,
backref=backref("parent", uselist=False))
class Customer(HasAddresses, Base):
name = Column(String)
class Supplier(HasAddresses, Base):
company_name = Column(String)
engine = create_engine('sqlite://', echo=True)
Base.metadata.create_all(engine)
session = Session(engine)
But if I try to look at the address of a Customer, I get None rather than [] that I'd expect:
(Pdb) foo = Customer()
(Pdb) foo.addresses
(Pdb)
(Pdb) type(foo.addresses)
<type 'NoneType'>
Which means I can't simply iterate over foo.addresses without first checking it is not None.
I tried setting uselist = True on the association proxy:
return relationship(assoc_cls,
uselist=True,
backref=backref("parent", uselist=False))
Which sort of works, but then the constructor fails:
(Pdb) foo.addresses
[]
(Pdb) foo.addresses.append(Address())
*** TypeError: Incompatible collection type: Address is not list-like
(Pdb)
I tried messing with the creator to make it construct a list, but then cause all manner of side effects, included nested lists:
cls.addresses = association_proxy(
"address_association", "addresses",
creator=lambda addresses: assoc_cls(addresses=[addresses,])
Any idea the best way to sort this out properly?
-Matt
I have a question concerning the mapping of entities in SQLAlchemy.
I have a transient object, which already contains foreign keys to some persistent objects. I want that SQLAlchemy fetches the referenced objects and assigns them to their relationship-attributes. From the SQLAlchemy documentation, I thought that I have to use the merge-operation on the session to achieve this. But in my configuration, it doesn't work.
This is a minimum example demonstrating my problem:
# -*- coding: utf-8 -*-
from sqlalchemy import create_engine
from sqlalchemy.orm import mapper
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
from sqlalchemy.orm import relationship, sessionmaker
class User(object):
def __init__(self, id, name, fullname, password, best_friend_id=None):
self.id = id
self.name = name
self.fullname = fullname
self.password = password
self.best_friend_id = best_friend_id
def __repr__(self):
return "<User('%s','%s', '%s')>" % (self.name, self.fullname, self.password)
class Dog(object):
def __init__(self, id, name):
self.id = id
self.name = name
def __repr__(self):
return "<User('%s','%s', '%s')>" % (self.name, self.fullname, self.password)
engine = create_engine('sqlite:///:memory:', echo=True)
Session = sessionmaker(bind=engine)
session = Session()
metadata = MetaData()
dogs_table = Table('dogs', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
)
users_table = Table('users', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
Column('fullname', String),
Column('password', String),
Column('best_friend_id', Integer, ForeignKey('dogs.id'))
)
metadata.create_all(engine)
mapper(User, users_table, properties={'best_friend': relationship(Dog, uselist=False)})
mapper(Dog, dogs_table)
dog = Dog(id=1, name='Hasso')
lordling = User(id=2, name='John', fullname='Miller', password='very_secret', best_friend_id=1)
session.add(dog)
session.commit()
merged_lordling = session.merge(lordling)
print str(merged_lordling.best_friend.name)
I expect that merged_lordling.best_friend contains the dog 'Hasso'. But it is still None.
I was bit by this same problem recently. Once you established a relationship, you should simply assign your Dog instance to User.best_friend directly, not explicitly using the foreign key. I don't know why exactly that happens, but while investigating a similar problem I realized that if you do that, SQLAlchemy doesn't populate the relationship property until you flushed all the related instances.
So, instead of:
dog = Dog(id=1, name='Hasso')
lordling = User(id=2, name='John', fullname='Miller', password='very_secret',
best_friend_id=1)
session.add(dog)
Simply do:
dog = Dog(id=1, name='Hasso')
lordling = User(id=2, name='John', fullname='Miller', password='very_secret',
best_friend=dog)
session.add(lordling)
Or even:
lordling = User(id=2, name='John', fullname='Miller', password='very_secret',
best_friend=Dog(id=1, name='Hasso'))
session.add(lordling)
As a general rule, avoid using the foreign key columns directly when you have a relationship established. Embrace the ORM, and only assign or query directly from foreign keys when you really have no other choice. I learned that the hard way.
I would like to have a 'relationship' in an inherited (mixin) class.
However, when I create the inherited object, the relationship object is None. I cannot append to it.
How do I resolve this?
Here is code based upon the documentation
from sqlalchemy import Column, Integer, String, DateTime, Boolean, BigInteger, Float
from sqlalchemy import ForeignKey
from sqlalchemy.orm import relationship, backref
from sqlalchemy.ext.declarative import declared_attr
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Target(Base):
__tablename__ = "target"
id = Column(Integer, primary_key=True)
class RefTargetMixin(object):
#declared_attr
def target_id(cls):
return Column('target_id', ForeignKey('target.id'))
#declared_attr
def target(cls):
return relationship("Target",
primaryjoin="Target.id==%s.target_id" % cls.__name__
)
class Foo(RefTargetMixin, Base):
__tablename__ = 'foo'
id = Column(Integer, primary_key=True)
print repr(RefTargetMixin.target)
print repr(Foo.target)
print repr(Foo().target)
The output is:
<sqlalchemy.orm.properties.RelationshipProperty object at 0x24e7890>
<sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x24e7690>
None
In general, I should be able to append to the relationship object (target), but here I cannot because it is None. Why?
the reason the value is None is because you've defined this as a many-to-one relationship. Many-to-one, from parent-to-child, means there is a foreign key on the parent, which can only refer to one and only one child. If you'd like something of class RefTargetMixin to refer to a collection of items, then foreign keys must be on the remote side.
So then the goal here is to make any object that is a subclass of RefTargetMixin be a potential parent for a Target. This pattern is called the polymorphic association pattern. While it is common in many ORM toolkits to provide this by declaring a "polymorphic foreign key" on Target, this is not a good practice relationally, so the answer is to use multiple tables in some way. There are three scenarios for this provided in SQLAlchemy core in the examples/generic_association folder, including "single association table with discriminator", "table per association", and "table per related". Each pattern provides the identical declarative pattern for RefTargetMixin here but the structure of the tables changes.
For example, here is your model using "table per association", which in my view tends to scale the best provided you don't need to query multiple types of RefTargetMixin objects at once (note I literally used the example as is, just changed the names):
from sqlalchemy.ext.declarative import declarative_base, declared_attr
from sqlalchemy import create_engine, Integer, Column, \
String, ForeignKey, Table
from sqlalchemy.orm import Session, relationship
class Base(object):
"""Base class which provides automated table name
and surrogate primary key column.
"""
#declared_attr
def __tablename__(cls):
return cls.__name__.lower()
id = Column(Integer, primary_key=True)
Base = declarative_base(cls=Base)
class Target(Base):
pass
class RefTargetMixin(object):
#declared_attr
def targets(cls):
target_association = Table(
"%s_targets" % cls.__tablename__,
cls.metadata,
Column("target_id", ForeignKey("target.id"),
primary_key=True),
Column("%s_id" % cls.__tablename__,
ForeignKey("%s.id" % cls.__tablename__),
primary_key=True),
)
return relationship(Target, secondary=target_association)
class Customer(RefTargetMixin, Base):
name = Column(String)
class Supplier(RefTargetMixin, Base):
company_name = Column(String)
engine = create_engine('sqlite://', echo=True)
Base.metadata.create_all(engine)
session = Session(engine)
session.add_all([
Customer(
name='customer 1',
targets=[
Target(),
Target()
]
),
Supplier(
company_name="Ace Hammers",
targets=[
Target(),
]
),
])
session.commit()
for customer in session.query(Customer):
for target in customer.targets:
print target
This is the normal behaviour : Foo has one Target. When you create the Foo object, it has no Target yet, so the value of Foo().target is None.
If you want Foo to have multiple Targets, you should put a foo_id in Target, and not a target_id in Foo, and use a backref.
Also, in that case, it is not needed to specify the primary join.