I'm writing a web based jukebox for my own amusement and as a learning experiencing.
I'm finally getting around to finishing my models except for one big component: member genre tags. I know I'll have to relate three models and it'll involve using association_proxy and collection classes.
Here's the relevant models (I had an abstract model to handle declaring the id and name fields, but that caused issues that I'll look at later):
class Member(db.Model):
__tablename__ = 'members'
id = db.Column('id', db.Integer, primary_key=True)
name = db.Column('name', db.String(128))
##tagged_artists is a backref from MemberTaggedArtist
tags = association_proxy('tagged_artists', 'artist',
creator=lambda k,v: MemberTaggedArtist(tag=k, artist=v)
)
class Artist(db.Model):
__tablename__ = 'artists'
id = db.Column('id', db.Integer, primary_key=True)
name = db.Column('name', db.String(128))
class Tag(db.Model):
__tablename__ = 'tags'
id = db.Column('id', db.Integer, primary_key=True)
name = db.Column('name', db.String(128))
class MemberTaggedArtist(db.Model):
__tablename__ = 'membertaggedartists'
id = db.Column('id', db.Integer, primary_key=True)
member_id = db.Column('member_id', db.Integer, db.ForeignKey('members.id'))
artist_id = db.Column('artist_id', db.Integer, db.ForeignKey('artists.id'))
tag_id = db.Column('tag_id', db.Integer, db.ForeignKey('tags.id'))
member = db.relationship(Member, backref=db.backref(
'tagged_artists',
collection_class=attribute_mapped_collection('tag')
)
artist = db.relationship(Artist, backref='applied_tags')
tag = db.relationship(Tag, backref='applied_artists')
What I'd like to happen is this:
>>> member = Member(name='justanr')
>>> artist = Artist(name='Gorguts')
>>> tag = Tag('Death Metal')
>>> member.tags['Death Metal'].append('Gorguts')
>>> member.tags
... {'Death Metal':['Gorguts']}
What currently happens is this (note, I built a mixin to handle repr calls):
>>> member.tags
... {Tag (ID:1 Name:Death Metal): MemberTaggedArtist (ID: 1 Member:justanr Artist:Gorguts Tag:Death Metal)}
I haven't been working with association_proxy long enough to understand what I'm doing wrong and even the brief tutorial in the documentation is giving me issues (I'm not sure why and I don't think it's because I'm using Flask-SQLAlchemy).
In short, I'm attempting to build an association proxy to create a dict of lists and I'm completely lost. I'm unsure what values I should proxy along, if using one middle table is over complicating this, and how to construct a secondary (and possibly tertiary) middle table
I finally came up with a solution. I forgot what I knew about SQL...lol I guess? But I recently watched several talks on SQLA that made me realize that I had an 'O' problem -- I was focusing specifically on the Object portion of ORM instead of thinking "SQLA just makes writing Queries easier, it doesn't do it for me." It also didn't help that FSQLA's query object lulled me in complacency either. Don't get me wrong, it's fantastic for basic queries and joins, but for something this complex...not really.
I also didn't realize that my question was actually several questions. One part was "How do I write this query with SQLA's tools?" The other was, "Help me puzzle the correct association_proxy to get this desired outcome."
I'm still not sure about the association proxy portion of it yet, or if I even want to do that anymore. But I finally puzzled the solution to my first question out:
# Query database for tag names and count for a particular artist
artist = models.Artist.find_by_some_means(magic, vars)
q = db.session.query(
models.Tag.id.label('id'),
models.Tag.name.label('tag'),
db.func.count(models.MemberTaggedArtist.member_id).label('count')
)
q = q.join(models.MemberTaggedArtist, models.Tag.id == models.MemberTaggedArtist.tag_id)
q = q.filter(models.MemberTaggedArtist.artist_id == artist.id)
q = q.group_by(models.MemberTaggedArtist.tag_id)
# order by tag count first, then tag name if two tags happen to have the same count
# there's probably a much better way to get there without recalculating the result
q = q.order_by(db.desc(db.func.count(models.MemberTaggedArtist.member_id)))
q = q.order_by(models.Tag.name)
My original planned query was something along the lines of:
SELECT tags.name AS tag, tags.id AS id, count(mta.member_id) AS count
FROM tags, membertaggedartists AS mta
WHERE tags.id = mta.tag_id AND mta.artist_id = :artist_id
GROUP BY id
ORDER BY count DESC, name ASC
And SQLA interprets my request as:
SELECT tags.id AS id, tags.name AS tag, count(membertaggedartists.member_id) AS count
FROM tags
JOIN membertaggedartists ON tags.id = membertaggedartists.tag_id
WHERE membertaggedartists.artist_id = :artist_id_1
GROUP BY membertaggedartists.tag_id
ORDER BY count(membertaggedartists.member_id) DESC, tags.name
Which is remarkably similar (and uses an explicit join unlike mine, which negates the need for one of my original WHERE clauses).
Related
I know I can simply update many to many relationship like this:
tags = db.Table('tags',
db.Column('tag_id', db.Integer, db.ForeignKey('tag.id'), primary_key=True),
db.Column('page_id', db.Integer, db.ForeignKey('page.id'), primary_key=True)
)
class Page(db.Model):
id = db.Column(db.Integer, primary_key=True)
tags = db.relationship('Tag', secondary=tags, lazy='subquery',
backref=db.backref('pages', lazy=True))
class Tag(db.Model):
id = db.Column(db.Integer, primary_key=True)
tag1 = Tag()
tag2 = Tag()
page = Page( tags=[tag1])
and later for updating:
page.append(tag2)
but I want to update them only by the tag id, Assume I have to create a general function that only accepts person and ids for addresses and update it.
What I want is something like this:
page = Page(tags=[1,2]) # 1 and 2 are primary keys of (tag)s
or in a function
def update_with_foreignkey(page, tags=[1,2]):
# dosomething to update page without using Tag object
return updated page
It was a little tricky and by using the evil eval but finally, I find a general way to update many to many relations using foreign keys. My goal was to update my object by getting data from a PUT request and in that request, I only get foreign keys for the relationship with other data.
Step by step solution:
1- Find relationships in the object, I find them using __mapper__.relationships
2- Find the key that represents the relationship.
for rel in Object.__mapper__.relationships:
key = str(rel).rsplit('.',1)[-1]
in question case it return 'tags' as the result.
3- Find the model for another side of the relation ( in this example Tag).
3-1 Find name of the table.
3-2 Convert table name to camleCase because sqlalchemy use underscore for the table name and camelCase for the model.
3-3 Use eval to get the model.
if key in data:
table = eval(convert_to_CamelCase(rel.table.name))
temp = table.query.filter(table.id.in_(data[key])).all() # this line convert ids to sqlacemy objects
All together
def convert_to_CamelCase(word):
return ''.join(x.capitalize() or '_' for x in word.split('_'))
def update_relationship_withForeingkey(Object, data):
for rel in Object.__mapper__.relationships:
key = str(rel).rsplit('.',1)[-1]
if key in data:
table = eval(convert_to_CamelCase(rel.table.name))
temp = table.query.filter(table.id.in_(data[key])).all() # this line convert ids to sqlacemy objects
data[key] = temp
return data
data is what I get from the request, and Object is the sqlalchemy Model that I want to update.
running this few lines update give me the result:
item = Object.query.filter_by(id=data['id'])
data = update_relationship_withForeingkey(Object,data)
for i,j in data.items():
setattr(item,i,j)
db.session.commit()
I'm not sure about caveats of this approach but it works for me. Any improvement and sugesstion are welcome.
my filter on children doesn't work. I am not sure, what it is done in wrong way.
country.py
product_country = Table('product_country', Base.metadata,
Column('product_id', Integer, ForeignKey('product.id'), primary_key=True),
Column('country_id', Integer, ForeignKey('country.id'), primary_key=True)
)
class Country(Base):
__tablename__="country"
id = Column(Integer, primary_key=True)
name = Column(String(200))
products = relationship(Product, secondary=product_country, backref='countries')
product.py
class Product(Base):
__tablename__="product"
id = Column(Integer, primary_key=True)
color = Column(Integer)
....
then sqlalchemy search:
country = s.query(Country).join(Country.products).filter(Country.id==1).filter(Product.color==1).first()
Well, I get country with id=1, what is what I want, but in list country.products, I would expect only products with color = 1, but there are all products assigned to country. Please could you help me. Thank you
That's a misunderstanding. The relationship loading is separate from your query, for good reasons. In other words the join you've used is not used to eagerly load the relationship. You could instruct the query that it contains an eager load join with contains_eager(), and read "the zen of joined eager loading" in order to understand and "Using contains_eager() to load a custom-filtered collection result" for an example how to do what you're trying. In your case just add
options(contains_eager(Country.products))
to your query.
With this models:
class Ball(Base):
__tablename__ = 'balls'
id = Column(Integer, primary_key=True)
field_id = Column(Integer, ForeignKey('fields.id'))
field = relationship("Field", back_populates="fields")
class Field(Base):
__tablename__ = 'fields'
id = Column(Integer, primary_key=True)
nickname = Column(String)
fields= relationship("Ball", order_by=Ball.id, back_populates="field")
I'm trying to write query to access Field.nickname and Ball.field_id. With use of
result = session.query(Field, Ball).all()
print(result)
I'm able to retrieve
(<Field(id=1, nickname=stadium>, <Ball(id=1, field_id=newest)>), (<Field(id=1, nickname=stadium>, <Ball(id=2, field_id=oldest)>
but when using
result = session.query(Field).join(Ball).all()
print(result)
I'm getting empty list [], doesn't matter if I'm applying any filters or not. According to official docs, join would come in handy if dealing with two tables, so this way I would be able to filter for the same id in both and I guess it would came in handy when displaying query results in jinja template.
You have a typo in your Field class - back_populates='fields' doesn't match up with the attribute name in Ball.
Try changing it to back_populates='field' so that it matches the attribute defined in Ball. With the relationship defined correctly sqlalchemy should be able to do the join automatically.
I have two models, joined by a many-to-many relationship
image_tag = Table('image_tag', Base.metadata,
Column('image_id', Integer, ForeignKey('images.id')),
Column('tag_id', Integer, ForeignKey('tags.id'))
)
class Image(Base):
__tablename__='images'
id = db.Column(db.Integer, primary_key=True)
tags = relationship('Tag', secondary=image_tag, backref=backref('images', order_by=id.desc()), lazy="joined")
class Tag(Base):
__tablename__ = 'tags'
id = Column(Integer, primary_key=True)
tag = Column(String(64), unique=True)
Now lets say I want to filter Images by tag – easy:
_tag = "foo"
Image.query.filter(Image.tags.any(tag=_tag)).all()
But what if I want to filter by many tags and only want to match those Images which match all of the tags?
tags = ["foo", "bar"]
???
Any help is incredibly appreciated. Thanks!
I see two possibilites: Either combine two EXIST clauses that check for each element separately or specify a where clause like this:
WHERE (SELECT COUNT(*) FROM ... WHERE tags.tag in ("foo", "bar")) = 2
Both solutions are kinda ugly but I see a problem here more with the way SQL is than with SQLAlchemy.
In both cases I recommend you build a normal SQL query first, test it and them build that in SQLAlchemy equivalently.
By the way, I could imagine both solutions being very inefficient on large sets of data, but I don't see a solution here.
Initially, I would have tried something like WHERE (SELECT tags.tag FROM ...) = ("foo", "bar") but that does not seem to be valid SQL (at least MySQL threw an error at me) because the subquery in WHERE must return a scalar result.
This is my Flask-SQLAlchemy Declarative code:
from sqlalchemy.ext.associationproxy import association_proxy
from my_flask_project import db
tagging = db.Table('tagging',
db.Column('tag_id', db.Integer, db.ForeignKey('tag.id', ondelete='cascade'),
primary_key=True),
db.Column('role_id', db.Integer, db.ForeignKey('role.id', ondelete='cascade'),
primary_key=True)
)
class Tag(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(100), unique=True, nullable=False)
def __init__(self, name=None):
self.name = name
class Role(db.Model):
id = db.Column(db.Integer, primary_key=True)
user_id = db.Column(db.Integer, db.ForeignKey('user.id', ondelete='cascade'))
user = db.relationship('User', backref=db.backref('roles', cascade='all',
lazy='dynamic'))
...
tags = db.relationship('Tag', secondary=tagging, cascade='all',
backref=db.backref('roles', cascade='all'))
tag_names = association_proxy('tags', 'name')
__table_args__ = (
db.UniqueConstraint('user_id', 'check_id'),
)
I think it's pretty standard many-to-many tagging solution. Now, I'd like to get all tags for a role and set new set of tags to a role.
The first one is pretty easy:
print role.tags
print role.tag_names
However, the second one made me stumbling upon my Python code all day long :-( I thought I could do this:
role.tag_names[:] = ['red', 'blue', 'white']
...or at least something similar using role.tags[:] = ..., but everything I invented raised many integrity errors, as SQLAlchemy didn't check if there are any existing tags and tried to insert all of them as completely new entities.
My final solution is:
# cleanup input
tag_names = set(filter(None, tag_names))
# existings tags to be updated
to_update = [t for t in role.tags if t.name in tag_names]
# existing tags to be added
to_add = list(
Tag.query.filter(Tag.name.in_(tag_names - set(role.tag_names)))
)
# tags to be created
existing_tags = to_update + to_add
to_create = [Tag(name) for name in tag_names - set([t.name for t in existing_tags])]
# assign new tags
role.tags[:] = existing_tags + to_create
# omitted bonus: find a way how to get rid of orphan tags
The question is: Is this really the right solution? Is there any more elegant way how to solve this trivial problem? I thik the whole matter is related to this question. Maybe I'm just silly, maybe I'm making things overcomplicated... anyway, thank you for any suggestions!
Actually SQLAlchemy does check if the object exists by calling Session.merge(). But it does it by identity — its primary key. The simplest solution is to make name primary key and everything will work. Sure, the three tables chain will become redundant in this case unless you are going to add some additional fields into Tag (e.g. counter).