We Keep Coding

Enforcing uniqueness using SQLAlchemy association proxies

I'm trying to use association proxies to make dealing with tag-style records a little simpler, but I'm running into a problem enforcing uniqueness and getting objects to reuse existing tags rather than always create new ones.
Here is a setup similar to what I have. The examples in the documentation have a few recipes for enforcing uniqueness, but they all rely on having access to a session and usually require a single global session, which I cannot do in my case.
from sqlalchemy import Column, Integer, String, create_engine, ForeignKey
from sqlalchemy.orm import sessionmaker, relationship
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.associationproxy import association_proxy
Base = declarative_base()
engine = create_engine('sqlite://', echo=True)
Session = sessionmaker(bind=engine)
def _tag_find_or_create(name):
# can't use global objects here, may be multiple sessions and engines
# ?? No access to session here, how to do a query
tag = session.query(Tag).filter_by(name=name).first()
tag = Tag.query.filter_by(name=name).first()
if not tag:
tag = Tag(name=name)
return tag
class Item(Base)
__tablename__ = 'item'
id = Column(Integer, primary_key=True)
tags = relationship('Tag', secondary='itemtag')
tagnames = association_proxy('tags', 'name', creator=_tag_find_or_create)
class ItemTag(Base)
__tablename__ = 'itemtag'
id = Column(Integer, primary_key=True)
item_id = Column(Integer, ForeignKey('item.id'))
tag_id = Column(Integer, ForeignKey('tag.id'))
class Tag(Base)
__tablename__ = 'tag'
id = Column(Integer, primary_key=True)
name = Column(String(50), nullable=False)
# Scenario 1
session = Session()
item = Item()
session.add(item)
item.tagnames.append('red')
# Scenario 2
item2 = Item()
item2.tagnames.append('blue')
item2.tagnames.append('red')
session.add(item2)
Without the creator function, I just get tons of duplicate Tag items. The creator function seems like the most obvious place to put this type of check, but I'm unsure how to do a query from inside the creator function.
Consider the two scenarios provided at the bottom of the example. In the first example, it seems like there should be a way to get access to the session in the creator function, since the object the tags are being added to is already associated with a session.
In the second example, the Item object isn't yet associated with a session, so the validation check can't happen in the creator function. It would have to happen later when the object is actually added to a session.
For the first scenario, how would I go about getting access to the session object in the creator function?
For the second scenario, is there a way to "listen" for when the parent object is added to a session and validate the association proxies at that point?

For the first scenario, you can use object_session.
As for the question overall: true, you need access to the current session; if using scoped_session in your application is appropriate, then the second part of the Recipe you link to should work fine to use. See Contextual/Thread-local Sessions for more info.
Working with events and change objects when they change from transient to persistent state will not make your code pretty or very robust. So I would immediately add new Tag objects to the session, and if the transaction is rolled back, they would not be in the database.
Note that in a multi-user environment you are likely to have race condition: the same tag is new and created in simultaneously by two users. The user who commits last will fail (if you have a unique constraint on the database).
In this case you might consider be without the unique constraint, and have a (daily) procedure to clean those duplicates up (and reassign relations). With time there would be less and less new items, and less possibilities for such clashes.

Is it possible to have a collection on an object that does not have a foreign key relationship to each other?

I'm using an declarative SQLAlchemy class to perform computations. Part of the computations require me to perform the computations for all configurations provided by a different table which doesn't have any foreign key relationships between the two tables.
This analogy is nothing like my real application, but hopefully will help to comprehend what I want to happen.
I have a set of cars and a list of paint colors.
The car object has a factory which provides a car in all possible colors
from sqlalchemy import *
from sqlachemy.orm import *
def PaintACar(car, color):
pass
Base = declarative_base()
class Colors(Base):
__table__ = u'colors'
id = Column('id', Integer)
color= Column('color', Unicode)
class Car(Base):
__table__ = u'car'
id = Column('id', Integer)
model = Column('model', Unicode)
# is this somehow possible?
all_color_objects = collection(...)
# I know this is possible, but would like to know if there's another way
#property
def all_colors(self):
s = Session.object_session(self)
return s.query(A).all()
def CarColorFactory(self):
for color in self.all_color_objects:
yield PaintACar(self, color)
My question: Is it possible to produce all_color_objects somehow? Without having to resort to finding the session and manually issuing a query as in the all_colors property?

It's been a while, so I'm providing the best answer I saw (as a comment by zzzeek). Basically, I was looking for one-off syntactic sugar. My original 'ugly' implementation works just fine.
what better way would there be here besides getting a Session and producing the query you
want? Are you looking for being able to add to the collection and that automatically
flushes things? (just add the objects to the Session?) Do you not like using
object_session(self) >(you can build some mixin class or something that hides that for
you?) It's not really clear >what the problem is. The objects here have no relationship to
the parent class so there's no particular intelligence SQLAlchemy would be able to add.
– zzzeek Jun 17 at 5:03

SQLAlchemy: a better way for update with declarative?

Let's say I have a user table in declarative mode:
class User(Base):
__tablename__ = 'user'
id = Column(u'id', Integer(), primary_key=True)
name = Column(u'name', String(50))
When I know user's id without object loaded into session, I update such user like this:
ex = update(User.__table__).where(User.id==123).values(name=u"Bob Marley")
Session.execute(ex)
I dislike using User.__table__, should I stop worrying with that?
Is there a better way to do this?

There's also some update capability at the ORM level. It doesn't handle any tricky cases yet but for the trivial case of single row update (or bulk update) it works fine. It even goes over any already loaded objects and applies the update on them also. You can use it like this:
session.query(User).filter_by(id=123).update({"name": u"Bob Marley"})

You're working on clause level here, not on model/entity/object level. Clause level is lower than mapped objects. And yes, something have to be done to convert one terms into others.
You could also stay on object level and do:
session = Session()
u = session.query(User).get(123)
u.name = u"Bob Marley"
session.commit()
but it will be significantly slower since it leads to the mapped object construction. And I'm not sure that it is more readable.
In the example your provided I see the most natural and “right” solution. I would not worry about little __table__ magic.

Similar functionality is available via the update() method on Table object.
class User(Base):
__tablename__ = 'user'
id = Column('id', Integer(), primary_key=True)
name = Column('name', String(50))
stmt = User.__table__.update().where(User.id==5).values(name='user #5')
To use User.__table__ is how its done in SQLAlchemy.

Sqlalchemy+elixir: How query with a ManyToMany relationship?

I'm using sqlalchemy with Elixir and have some troubles trying to make a query..
I have 2 entities, Customer and CustomerList, with a many to many relationship.
customer_lists_customers_table = Table('customer_lists_customers',
metadata,
Column('id', Integer, primary_key=True),
Column('customer_list_id', Integer, ForeignKey("customer_lists.id")),
Column('customer_id', Integer, ForeignKey("customers.id")))
class Customer(Entity):
[...]
customer_lists = ManyToMany('CustomerList', table=customer_lists_customers_table)
class CustomerList(Entity):
[...]
customers = ManyToMany('Customer', table=customer_lists_customers_table)
I'm tryng to find CustomerList with some customer:
customer = [...]
CustomerList.query.filter_by(customers.contains(customer)).all()
But I get the error:
NameError:
global name 'customers' is not defined
customers seems to be unrelated to the entity fields, there's an special query form to work with relationships (or ManyToMany relationships)?
Thanks

You can use regular filter: query.filter(CustomerList.customers.contains(customer)). See SQLAlchemy documentation for more examples. It's actually filter_by that's a special case. The query.filter_by(**kwargs) shorthand works only for simple equality comparisons. Under the cover query.filter_by(foo="bar", baz=42) is delegated to the equivalent of query.filter(and_(MyClass.foo == "bar", MyClass.baz == 42)). (There's actually slightly more magic to figure out which property you meant you have many entities, but it still uses simple delegation)

Read the error message with attention, it points to the source of problem. Did you mean
CustomerList.query.filter_by(CustomerList.customers.contains(customer)).all()?
Update: When using declarative definition you can use just defined relation in class scope, but these properties are not visible outside class:
class MyClass(object):
prop1 = 'somevalue'
prop2 = prop1.upper() # prop1 is visible here
val2 = MyClass.prop1 # This is OK
val1 = prop1.lower() # And this will raise NameError, since there is no
# variable `prop1` is global scope

CustomerList.query.filter_by(CustomerList.customers.contains(customer)).all() should work fine.

Should I create mapper objects or use the declarative syntax in SQLAlchemy?

There are two (three, but I'm not counting Elixir, as its not "official") ways to define a persisting object with SQLAlchemy:
Explicit syntax for mapper objects
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
from sqlalchemy.orm import mapper
metadata = MetaData()
users_table = Table('users', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
)
class User(object):
def __init__(self, name):
self.name = name
def __repr__(self):
return "<User('%s')>" % (self.name)
mapper(User, users_table) # <Mapper at 0x...; User>
Declarative syntax
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String)
def __init__(self, name):
self.name = name
def __repr__(self):
return "<User('%s')>" % (self.name)
I can see that while using the mapper objects, I separate completely the ORM definition from the business logic, while using the declarative syntax, whenever I modify the business logic class, I can edit right there the database class (which ideally should be edited little).
What I'm not completely sure, is which approach is more maintainable for a business application?
I haven't been able to find a comparative between the two mapping methods, to be able to decide which one is a better fit for my project.
I'm leaning towards using the "normal" way (i.e. not the declarative extension) as it allows me to "hide", and keep out of the business view all the ORM logic, but I'd like to hear compelling arguments for both approaches.

"What I'm not completely sure, is which approach is more maintainable for a business application?"
Can't be answered in general.
However, consider this.
The Django ORM is strictly declarative -- and people like that.
SQLAlchemy does several things, not all of which are relevant to all problems.
SQLAlchemy creates DB-specific SQL from general purpose Python. If you want to mess with the SQL, or map Python classes to existing tables, then you have to use explicit mappings, because your focus is on the SQL, not the business objects and the ORM.
SQLAlchemy can use declarative style (like Django) to create everything for you. If you want this, then you are giving up explicitly writing table definitions and explicitly messing with the SQL.
Elixir is an alternative to save you having to look at SQL.
The fundamental question is "Do you want to see and touch the SQL?"
If you think that touching the SQL makes things more "maintainable", then you have to use explicit mappings.
If you think that concealing the SQL makes things more "maintainable", then you have to use declarative style.
If you think Elixir might diverge from SQLAlchemy, or fail to live up to it's promise in some way, then don't use it.
If you think Elixir will help you, then use it.

In our team we settled on declarative syntax.
Rationale:
metadata is trivial to get to, if needed: User.metadata.
Your User class, by virtue of subclassing Base, has a nice ctor that takes kwargs for all fields. Useful for testing and otherwise. E.g.: user=User(name='doe', password='42'). So no need to write a ctor!
If you add an attribute/column, you only need to do it once. "Don't Repeat Yourself" is a nice principle.
Regarding "keeping out ORM from business view": in reality your User class, defined in a "normal" way, gets seriously monkey-patched by SA when mapper function has its way with it. IMHO, declarative way is more honest because it screams: "this class is used in ORM scenarios, and may not be treated just as you would treat your simple non-ORM objects".

I've found that using mapper objects are much simpler then declarative syntax if you use sqlalchemy-migrate to version your database schema (and this is a must-have for a business application from my point of view). If you are using mapper objects you can simply copy/paste your table declarations to migration versions, and use simple api to modify tables in the database. Declarative syntax makes this harder because you have to filter away all helper functions from your class definitions after copying them to the migration version.
Also, it seems to me that complex relations between tables are expressed more clearly with mapper objects syntax, but this may be subjective.

as of current (2019), many years later, sqlalchemy v1.3 allows a hybrid approach with the best of both worlds
https://docs.sqlalchemy.org/en/13/orm/extensions/declarative/table_config.html#using-a-hybrid-approach-with-table
metadata = MetaData()
users_table = Table('users', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
)
# possibly in a different file/place the orm-declaration
Base = declarative_base(metadata)
class User(Base):
__table__ = Base.metadata.tables['users']
def __str__():
return "<User('%s')>" % (self.name)