I'm trying to implement dynamic filtering using SQLAlchemy ORM.
I was looking through StackOverflow and found very similar question:SQLALchemy dynamic filter_by
It's useful for me, but not enough.
So, here is some example of code, I'm trying to write:
# engine - MySQL engine
session_maker = sessionmaker(bind=engine)
session = session_maker()
# my custom model
model = User
def get_query(session, filters):
if type(filters) == tuple:
query = session.query(model).filter(*filters)
elif type(filters) == dict:
query = session.query(model).filter(**filters)
return query
then I'm trying to reuse it with something very similar:
filters = (User.name == 'Johny')
get_query(s, filters) # it works just fine
filters = {'name': 'Johny'}
get_query(s, filters)
After the second run, there are some issues:
TypeError: filter() got an unexpected keyword argument 'name'
When I'm trying to change my filters to:
filters = {User.name: 'Johny'}
it returns:
TypeError: filter() keywords must be strings
But it works fine for manual querying:
s.query(User).filter(User.name == 'Johny')
What is wrong with my filters?
BTW, it looks like it works fine for case:
filters = {'name':'Johny'}
s.query(User).filter_by(**filters)
But following the recommendations from mentioned post I'm trying to use just filter.
If it's just one possible to use filter_by instead of filter, is there any differences between these two methods?
Your problem is that filter_by takes keyword arguments, but filter takes expressions. So expanding a dict for filter_by **mydict will work. With filter, you normally pass it one argument, which happens to be an expression. So when you expand your **filters dict to filter, you pass filter a bunch of keyword arguments that it doesn't understand.
If you want to build up a set of filters from a dict of stored filter args, you can use the generative nature of the query to keep applying filters. For example:
# assuming a model class, User, with attributes, name_last, name_first
my_filters = {'name_last':'Duncan', 'name_first':'Iain'}
query = session.query(User)
for attr,value in my_filters.iteritems():
query = query.filter( getattr(User,attr)==value )
# now we can run the query
results = query.all()
The great thing about the above pattern is you can use it across multiple joined columns, you can construct 'ands' and 'ors' with and_ and or_, you can do <= or date comparisons, whatever. It's much more flexible than using filter_by with keywords. The only caveat is that for joins you have to be a bit careful you don't accidentally try to join a table twice, and you might have to specify the join condition for complex filtering. I use this in some very complex filtering over a pretty involved domain model and it works like a charm, I just keep a dict going of entities_joined to keep track of the joins.
I have a similar issue, tried to filter from a dictionary:
filters = {"field": "value"}
Wrong:
...query(MyModel).filter(**filters).all()
Good:
...query(MyModel).filter_by(**filters).all()
FWIW, There's a Python library designed to solve this exact problem: sqlalchemy-filters
It allows to dynamically filter using all operators, not only ==.
from sqlalchemy_filters import apply_filters
# `query` should be a SQLAlchemy query object
filter_spec = [{'field': 'name', 'op': '==', 'value': 'name_1'}]
filtered_query = apply_filters(query, filter_spec)
more_filters = [{'field': 'foo_id', 'op': 'is_not_null'}]
filtered_query = apply_filters(filtered_query, more_filters)
result = filtered_query.all()
For the people using FastAPI and SQLAlchemy, here is a example of dynamic filtering:
api/app/app/crud/order.py
from typing import Optional
from pydantic import UUID4
from sqlalchemy.orm import Session
from app.crud.base import CRUDBase
from app.models.order import Order
from app.schemas.order import OrderCreate, OrderUpdate
class CRUDOrder(CRUDBase[Order, OrderCreate, OrderUpdate]):
def get_orders(
self,
db: Session,
owner_id: UUID4,
status: str,
trading_type: str,
pair: str,
skip: int = 0,
limit: int = 100,
) -> Optional[Order]:
filters = {
arg: value
for arg, value in locals().items()
if arg != "self" and arg != "db" and arg != "skip" and arg != "limit" and value is not None
}
query = db.query(self.model)
for attr, value in filters.items():
query = query.filter(getattr(self.model, attr) == value)
return (
query
.offset(skip)
.limit(limit)
.all()
)
order = CRUDOrder(Order)
class Place(db.Model):
id = db.Column(db.Integer, primary_key=True)
search_id = db.Column(db.Integer, db.ForeignKey('search.id'), nullable=False)
#classmethod
def dynamic_filter(model_class, filter_condition):
'''
Return filtered queryset based on condition.
:param query: takes query
:param filter_condition: Its a list, ie: [(key,operator,value)]
operator list:
eq for ==
lt for <
ge for >=
in for in_
like for like
value could be list or a string
:return: queryset
'''
__query = db.session.query(model_class)
for raw in filter_condition:
try:
key, op, value = raw
except ValueError:
raise Exception('Invalid filter: %s' % raw)
column = getattr(model_class, key, None)
if not column:
raise Exception('Invalid filter column: %s' % key)
if op == 'in':
if isinstance(value, list):
filt = column.in_(value)
else:
filt = column.in_(value.split(','))
else:
try:
attr = list(filter(lambda e: hasattr(column, e % op), ['%s', '%s_', '__%s__']))[0] % op
except IndexError:
raise Exception('Invalid filter operator: %s' % op)
if value == 'null':
value = None
filt = getattr(column, attr)(value)
__query = __query.filter(filt)
return __query
Execute like:
places = Place.dynamic_filter([('search_id', 'eq', 1)]).all()
I want to use outerjoin operation on a subquery and also include values from the subquery also.
My code
q_responses = session.query(Candidate, CandidateProfile)
.join(CandidateProfile, CandidateProfile.candidate_id == Candidate.id)
subq = (session.query(AppAction.candidate_id, Activity.archived)\
.join(Activity, and_(AppAction.candidate_id == Activity.candidate_id,
Activity.archived == 1)))\
.subquery("subq")
responses = q_responses.outerjoin(subq, Candidate.id == subq.c.candidate_id).all()
So I get the result in this format
(Candidate, CandidateProfile)
But I also want to include the archived value from subquery in the result.
By reading many relevant posts from the internet, I have tried
add_entity(subq.c.archived)
with_entities
add_column
select_from
But all those have resulted in some error.
Please help me out.
Please share your error for when you try add_column. The code below should work just fine (assuming that it does work without like which contains add_column):
responses = (
q_responses
.add_column(subq.c.archived) # #new
.outerjoin(subq, Candidate.id == subq.c.candidate_id)
).all()
Also you could have created a query straight away with this column included:
subq = (
session.query(AppAction.candidate_id, Activity.archived)
.join(Activity, and_(AppAction.candidate_id == Activity.candidate_id,
Activity.archived == 1))
).subquery("subq")
q_responses = (
session.query(Candidate, CandidateProfile, subq.c.archived)
.join(CandidateProfile, CandidateProfile.candidate_id == Candidate.id)
.outerjoin(subq, Candidate.id == subq.c.candidate_id)
).all()
I have the following query call:
SearchList = (DBSession.query(
func.count(ExtendedCDR.uniqueid).label("CallCount"),
func.sum(ExtendedCDR.duration).label("TotalSeconds"),
ExtendedCDR,ExtensionMap)
.filter(or_(ExtensionMap.exten == ExtendedCDR.extension,ExtensionMap.prev_exten == ExtendedCDR.extension))
.filter(between(ExtendedCDR.start,datebegin,dateend))
.filter(ExtendedCDR.extension.in_(SelectedExtension))
.group_by(ExtendedCDR.extension)
.order_by(func.count(ExtendedCDR.uniqueid).desc()))
.all()
)
I would like to be able to define the order_by clause prior to calling the .query(), is this possible?
I tried doing as this stackoverflow answer suggests for a filter spec, but I had no idea how to create the filter_group syntax.
From that post:
filter_group = list(Column.in_('a','b'),Column.like('%a'))
query = query.filter(and_(*filter_group))
You build a SQL query with the DBSession.query() call, and this query is not executed until you call .all() on it.
You can store the intermediary results and add more filters or other clauses as needed:
search =DBSession.query(
func.count(ExtendedCDR.uniqueid).label("CallCount"),
func.sum(ExtendedCDR.duration).label("TotalSeconds"),
ExtendedCDR,ExtensionMap)
search = search.filter(or_(
ExtensionMap.exten == ExtendedCDR.extension,
ExtensionMap.prev_exten == ExtendedCDR.extension))
search = search.filter(between(ExtendedCDR.start, datebegin, dateend))
search = search.filter(ExtendedCDR.extension.in_(SelectedExtension))
search = search.group_by(ExtendedCDR.extension)
search = search.order_by(func.count(ExtendedCDR.uniqueid).desc())
The value you pass to order_by can be created ahead of time:
search_order = func.count(ExtendedCDR.uniqueid).desc()
then used like:
search = search.order_by(search_order)
Once your query is complete, get the results by calling .all():
SearchList = search.all()
Say I have a query object
query = session.query(SomeModel) \
.filter(SomeModel.foo == 'bar') \
.filter(SomeModel.active == True)
Can you modify the object list prior to executing the query, maintaining all the same filters
i.e.
.change_query(SomeModel.id, SomeModel.name)
so that the resulting query object is:
session.query(SomeModel.id, SomeModel.name) \
.filter(SomeModel.foo == 'bar') \
.filter(SomeModel.active == True)
Wow already found the answer. Will leave this open in case people are searching for this:
query = query.with_entities(SomeModel.id, SomeModel.name)
I've just introspected a pretty nasty schema from a CRM app with sqlalchemy. All of the tables have a deleted column on them and I wanted to auto filter all those entities and relations flagged as deleted. Here's what I came up with:
class CustomizableQuery(Query):
"""An overridden sqlalchemy.orm.query.Query to filter entities
Filters itself by BinaryExpressions
found in :attr:`CONDITIONS`
"""
CONDITIONS = []
def __init__(self, mapper, session=None):
super(CustomizableQuery, self).__init__(mapper, session)
for cond in self.CONDITIONS:
self._add_criterion(cond)
def _add_criterion(self, criterion):
criterion = self._adapt_clause(criterion, False, True)
if self._criterion is not None:
self._criterion = self._criterion & criterion
else:
self._criterion = criterion
And it's used like this:
class UndeletedContactQuery(CustomizableQuery):
CONDITIONS = [contacts.c.deleted != True]
def by_email(self, email_address):
return EmailInfo.query.by_module_and_address('Contacts', email_address).contact
def by_username(self, uname):
return self.filter_by(twod_username_c=uname).one()
class Contact(object):
query = session.query_property(UndeletedContactQuery)
Contact.query.by_email('someone#some.com')
EmailInfo is the class that's mapped to the join table between emails and the other Modules that they're related to.
Here's an example of a mapper:
contacts_map = mapper(Contact, join(contacts, contacts_cstm), {
'_emails': dynamic_loader(EmailInfo,
foreign_keys=[email_join.c.bean_id],
primaryjoin=contacts.c.id==email_join.c.bean_id,
query_class=EmailInfoQuery),
})
class EmailInfoQuery(CustomizableQuery):
CONDITIONS = [email_join.c.deleted != True]
# More methods here
This gives me what I want in that I've filtered out all deleted Contacts. I can also use this as the query_class argument to dynamic_loader in my mappers - However...
Is there a better way to do this, I'm not really happy with poking around with the internals of a compicated class like Query as I am.
Has anyone solved this in a different way that they can share?
You can map to a select. Like this:
mapper(EmailInfo, select([email_join], email_join.c.deleted == False))
I'd consider seeing if it was possible to create views for these tables that filter out the deleted elements, and then you might be able to map directly to that view instead of the underlying table, at least for querying operations. However I've never tried this myself!