Essentially I have a SQLAlchemy query that looks like this:
foos = Foo.query.filter(Foo.expiration < cutoff)
valid_foos = []
for foo in foos:
last_bar = foo.bars.order_by('created_at desc').first()
if last_bar.state != 'fatal':
valid_foos.append(foo)
The goal being to select all the foos for which the first related bar's state is not "fatal". It seems like subqueries might help here. However, I'm struggling to grasp how I can express last_bar = foo.bars.order_by('created_at desc').first() in this way.
I think the easiest and most versatile way of doing it is using Hybrid Attributes extension. When you extend you model as per below:
class Foo(Base):
__tablename__ = "foo"
# ...
#hybrid_property
def last_bar_state(self):
_last_bar = self.bars.order_by(Bar.created_at.desc()).first()
return _last_bar.state
#last_bar_state.expression
def _last_bar_state_expression(cls):
q = (select([Bar.state.label("state")])
.where(Bar.foo_id == cls.foo_id)
.order_by(Bar.created_at.desc())
.limit(1)
).label("last_bar_state_sub")
return q
you will be able to use last_bar_state both in-memory, as well as in the query:
foos = session.query(Foo).filter(Foo.expiration < cutoff)
foos = foos.filter(Foo.last_bar_state != 'fatal').all()
Use a subquery to find the latest Bar for each Foo. Then use that to join and filter Bar when querying Foo.
sub = db.session.query(
Foo.id,
db.func.max(Bar.created_at).label('latest')
).join(Foo.bars).group_by(Foo.id).subquery()
foos = db.session.query(Foo).join(Bar, db.and_(
Bar.foo_id == sub.c.id,
Bar.created_at == sub.c.latest
)).filter(Bar.status != 'fatal').all()
This will not select Foo if it has no Bar, you can use an outer join instead for that.
Related
with SQLalchemy I query my database. I'm able to select all multiple rows / objects with something like this:
def selectAllObjects():
objects = session.query(Object).all()
However, I want to build a function that checks if there any constraints given and if so incorporates these. So I was thinking along the following lines:
def selectAllObjects(attribute1="default",attribute2="default"):
if attribute1 != "default" and attribute2 != default:
objects = session.query(Object).filter_by(attribute1=attribute1,
attribute2=attribute2).all()
elif attribute1 != "default":
objects = session.query(Object).filter_by(attribute1=attribute1).all()
...etc, etc...
As you can see this gets really ugly when the amount of attributes increase. What is the pythonic way to do this?
Query objects are chainable, from docs:
Query is produced in terms of a given Session, using the query()
method:
q = session.query(SomeMappedClass)
So you can write something like this:
def selectAllObjects(attribute1="default", attribute2="default"):
query = session.query(Object)
if attribute1 != "default" and attribute2 != "default":
query = query.filter_by(
attribute1=attribute1,
attribute2=attribute2,
)
elif attribute1 != "default":
query = query.filter_by(attribute1=attribute1)
return query.all()
Assume the following setup:
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyClass(Base):
id = Column(Integer, primary_key=True)
name = Column(String)
The normal paradigm to query the DB with SQLAlchemy is to do the following:
Session = sessionmaker()
engine = 'some_db_location_string'
session = Session(bind=engine)
session.query(MyClass).filter(MyClass.id == 1).first()
Suppose, I want to simplify the query to the following:
MyClass(s).filter(MyClass.id == 1).first()
OR
MyClass(s).filter(id == 1).first()
How would I do that? My first attempt at that to use a model Mixin class failed. This is what I tried:
class ModelMixins(object)
def __init__(self, session):
self.session = session
def filter(self, *args):
self.session.query(self).filter(*args)
# Redefine MyClass to use the above class
class MyClass(ModelMixins, Base):
id = Column(Integer, primary_key=True)
name = Column(String)
The main failure seems to be that I can't quite transfer the expression 'MyClass.id == 1' to the actual filter function that is part of the session object.
Folks may ask why would I want to do:
MyClass(s).filter(id == 1).first()
I have seen something similar like this used before and thought that the syntax becomes so much cleaner I can achieve this. I wanted to replicate this but have not been able to. Being able to do something like this:
def get_stuff(some_id):
with session_scope() as s:
rec = MyClass(s).filter(MyClass.id== some_id').first()
if rec:
return rec.name
else:
return None
...seems to be the cleanest way of doing things. For one, session management is kept separate. Secondly, the query itself is simplified. Having a Mixin class like this would allow me to add the filter functionality to any number of classes...So can someone help in this regard?
session.query takes a class; you're giving it self, which is an instance. Replace your filter method with:
def filter(self, *args):
return session.query(self.__class__).filter(*args)
and at least this much works:
In [45]: MyClass(session).filter(MyClass.id==1)
Out[45]: <sqlalchemy.orm.query.Query at 0x10e0bbe80>
The generated SQL looks right, too (newlines added for clarity):
In [57]: str(MyClass(session).filter(MyClass.id==1))
Out[57]: 'SELECT "MyClass".id AS "MyClass_id", "MyClass".name AS "MyClass_name"
FROM "MyClass"
WHERE "MyClass".id = ?'
No guarantees there won't be oddities; I've never tried anything like this before.
Ive been using this mixin to good success. Most likely not the most efficient thing in the world and I am no expert. I define a date_created column for every table
class QueryBuilder:
"""
This class describes a query builer.
"""
q_debug = False
def query_from_dict(self, db_session: Session, **q_params: dict):
"""
Creates a query.
:param db_session: The database session
:type db_session: Session
:param q_params: The quarter parameters
:type q_params: dictionary
"""
q_base = db_session.query(type(self))
for param, value in q_params.items():
if param == 'start_date':
q_base = q_base.filter(
type(self).__dict__.get('date_created') >= value
)
elif param == 'end_date':
q_base = q_base.filter(
type(self).__dict__.get('date_created') <= value
)
elif 'like' in param:
param = param.replace('_like', '')
member = type(self).__dict__.get(param)
if member:
q_base = q_base.filter(member.ilike(f'%{value}%'))
else:
q_base = q_base.filter(
type(self).__dict__.get(param) == value
)
if self.q_debug:
print(q_base)
return q_base
Is there any way to flip the relationship in a Django querset? Such as turning a Foo queryset into a Bar queryset if all Foo objects have a foreign key to Bar?
Take the following example:
#condition_n are previously defined statements that results in either True or False
if condition_1:
foo_qs = Foo.objects.filter(bar__something=...)
if condition_2:
foo_qs = foo_qs.filter(bar__another_thing=...)
if condition_3:
#you get the point
now this will result in some queryset foo_qs depending on the evaluation of the condition_n type statements. If I wanted to also get the bar queryset that would result from that, I could do the following:
#condition_n are previously defined statements that results in either True or False
if condition_1:
foo_qs = Foo.objects.filter(bar__something=...)
bar_qs = Bar.objects.filter(something=...)
if condition_2:
foo_qs = foo_qs.filter(bar__another_thing=...)
bar_qs = Bar.objects.filter(another_thing=...)
if condition_3:
#you get the point
but ideally I would like to take any queryset, and give it the name of the foreign key ("bar" in this case), and have it flip to the equivalent queryset. Is there any way about doing this?
Such as turning a Foo queryset into a Bar queryset if all Foo objects have a foreign key to Bar?
Why not? Here's one approach, if you always write the base query as the one local to the object in question.
bar_query = {
'something': 'something',
}
def prefix_fk(query, prefix):
prefixed_query = {}
for key, value in query.items():
prefixed_query[prefix+key] = value
return prefixed_query
Foo.objects.filter(**prefix_fk(bar_query, 'bar__'))
Bar.objects.filter(**bar_query)
if condition_2:
bar_query['another_query'] = 'foobar'
Foo.objects.filter(**prefix_fk(bar_query, 'bar__'))
Bar.objects.filter(**bar_query)
I need to implement a "related items" feature, i.e. to allow items from the same table to be arbitrarily linked to each other in a many-to-many fashion. Something similar to how news websites show related articles.
Also, I need the relationship to be bi-directional, something like this:
a = Item()
b = Item()
a.related.append(b)
assert a in b.related # True
Now, on SQL level I imagine this could be solved by modifying the "standard" many-to-many relationship so 2 records are inserted into the association table each time an association is made, so (a -> b) and (b -> a) are two separate records.
Alternatively, the join condition for the many-to-many table could somehow check both sides of the association, so roughly instead of ... JOIN assoc ON a.id = assoc.left_id ... SQLAlchemy would produce something like ... JOIN assoc ON a.id = assoc.left_id OR a.id = assoc.right_id ...
Is there a way to configure this with SQLAlchemy so the relation works similar to a "normal" many-to-many relationship?
It's likely that I'm just don't know the correct terminology - everything I came up with - "self-referential", "bidirectional", "association" - is used to describe something else in SQLAlchemy.
Using Attribute Events should do the job. See the sample code below, where little ugly piece of code is solely for the purpose of avoid endless recursion:
class Item(Base):
__tablename__ = "item"
id = Column(Integer, primary_key=True)
name = Column(String(255), nullable=False)
# relationships
related = relationship('Item',
secondary = t_links,
primaryjoin = (id == t_links.c.from_id),
secondaryjoin = (id == t_links.c.to_id),
)
_OTHER_SIDE = set()
from sqlalchemy import event
def Item_related_append_listener(target, value, initiator):
global _OTHER_SIDE
if not((target, value) in _OTHER_SIDE):
_OTHER_SIDE.add((value, target))
if not target in value.related:
value.related.append(target)
else:
_OTHER_SIDE.remove((target, value))
event.listen(Item.related, 'append', Item_related_append_listener)
# ...
a = Item()
b = Item()
a.related.append(b)
assert a in b.related # True
For completeness sake, here's the code I ended up with; the listener method is slightly different to avoid using a global variable, an also there's a listener for remove event.
import sqlalchemy as sa
related_items = sa.Table(
"related_items",
Base.metadata,
sa.Column("id", sa.Integer, primary_key=True),
sa.Column("from_id", sa.ForeignKey("items.id")),
sa.Column("to_id", sa.ForeignKey("items.id")),
)
class Item(Base):
__tablename__ = 'items'
...
related = sa.orm.relationship('Item',
secondary = related_items,
primaryjoin = (id == related_items.c.from_id),
secondaryjoin = (id == related_items.c.to_id),
)
def item_related_append_listener(target, value, initiator):
if not hasattr(target, "__related_to__"):
target.__related_to__ = set()
target.__related_to__.add(value)
if target not in getattr(value, "__related_to__", set()):
value.related.append(target)
sa.event.listen(Item.related, 'append', item_related_append_listener)
def item_related_remove_listener(target, value, initiator):
if target in value.related:
value.related.remove(target)
sa.event.listen(Item.related, 'remove', item_related_remove_listener)
Looking at the bottom of the post you can see i have three classes. The code here is pseudo code written on the fly and untested however it adequately shows my problem. If we need the actual classes I can update this question tomorrow when at work. So ignore syntax issues and code that only represents a thought rather than the actual "code" that would do what i describe there.
Question 1
If you look at the Item search class method you can see that when the user does a search i call search on the base class then based on that result return the correct class/object. This works but seems kludgy. Is there a better way to do this?
Question 2
If you look at the KitItem class you can see that I am overriding the list price. If the flag calc_list is set to true then I sum the list price of the components and return that as the list price for the kit. If its not marked as true I want to return the "base" list price. However as far as I know there is no way to access a parent attribute since in a normal setup it would be meaningless but with sqlalchemy and shared table inheritance it could be useful.
TIA
class Item(DeclarativeBase):
__tablename__ = 'items'
item_id = Column(Integer,primary_key=True,autoincrement=True)
sku = Column(Unicode(50),nullable=False,unique=True)
list_price = Column(Float)
cost_price = Column(Float)
item_type = Column(Unicode(1))
__mapper_args__ = {'polymorphic_on': item_type}
__
def __init__(self,sku,list_price,cost_price):
self.sku = sku
self.list_price = list_price
self.cost_price = cost_price
#classmethod
def search(cls):
"""
" search based on sku, description, long description
" return item as proper class
"""
item = DBSession.query(cls).filter(...) #do search stuff here
if item.item_type == 'K': #Better way to do this???
return DBSession.query(KitItem).get(item.item_id)
class KitItem(Item):
__mapper_args__ = {'polymorphic_identity': 'K'}
calc_list = Column(Boolean,nullable=False,default=False)
#property
def list_price(self):
if self.calc_list:
list_price = 0.0
for comp in self.components:
list_price += comp.component.list_price * comp.qty
return list_price
else:
#need help here
item = DBSession.query(Item).get(self.item_id)
return item.list_price
class KitComponent(DeclarativeBase):
__tablename__ = "kit_components"
kit_id = Column(Integer,ForeignKey('items.item_id'),primarykey=True)
component_id = Column(Integer,ForeignKey('items.item_id'),primarykey=True)
qty = Column(Integer,nullable=False, default=1)
kit = relation(KitItem,backref=backref("components"))
component = relation(Item)
Answer-1: in fact you do not need to do anything special here: given that you configured your inheritance hierarchy properly, your query will already return proper class for every row (Item or KitItem). This is the advantage of the ORM part. What you could do though is to configure the query to immediatelly load also the additional columns which do belong to children of Item (from your code this is only calc_list column), which you can do by specifying with_polymorphic('*'):
#classmethod
def search(cls):
item = DBSession.query(cls).with_polymorphic('*').filter(...) #do search stuff here
return item
Read more on this in Basic Control of Which Tables are Queried.
To see the difference, enabled SQL logging, and compare your tests scripts with and without with_polymorphic(...) - you will most probably require less SQL statements being executed.
Answer-2: I would not override one entry attributed with one which is purely computed. Instead I would just create another computed attribute (lets call it final_price), which would look like following for each of two classes:
class Item(Base):
...
#property
def total_price(self):
return self.list_price
class KitItem(Item):
...
#property
def total_price(self):
if self.calc_list:
_price = 0.0
for comp in self.components:
_price += comp.component.list_price * comp.qty
return _price
else:
# #note: again, you do not need to perform any query here at all, as *self* is that you need
return self.list_price
Also in this case, you might think of configuring the relationship KitItem.components to be eagerly loaded, so that the calculation of the total_price will not trigger additional SQL. But you have to decide yourself if this is beneficial for your use cases (again, analyse the SQLs generated in your scenario).