with SQLalchemy I query my database. I'm able to select all multiple rows / objects with something like this:
def selectAllObjects():
objects = session.query(Object).all()
However, I want to build a function that checks if there any constraints given and if so incorporates these. So I was thinking along the following lines:
def selectAllObjects(attribute1="default",attribute2="default"):
if attribute1 != "default" and attribute2 != default:
objects = session.query(Object).filter_by(attribute1=attribute1,
attribute2=attribute2).all()
elif attribute1 != "default":
objects = session.query(Object).filter_by(attribute1=attribute1).all()
...etc, etc...
As you can see this gets really ugly when the amount of attributes increase. What is the pythonic way to do this?
Query objects are chainable, from docs:
Query is produced in terms of a given Session, using the query()
method:
q = session.query(SomeMappedClass)
So you can write something like this:
def selectAllObjects(attribute1="default", attribute2="default"):
query = session.query(Object)
if attribute1 != "default" and attribute2 != "default":
query = query.filter_by(
attribute1=attribute1,
attribute2=attribute2,
)
elif attribute1 != "default":
query = query.filter_by(attribute1=attribute1)
return query.all()
Related
I have a SQLAlchemy model:
class Ticket(db.Model):
__tablename__ = 'ticket'
id = db.Column(INTEGER(unsigned=True), primary_key=True, nullable=False,
autoincrement=True)
cluster = db.Column(db.VARCHAR(128))
#classmethod
def get(cls, cluster=None):
query = db.session.query(Ticket)
if cluster is not None:
query = query.filter(Ticket.cluster==cluster)
return query.one()
If I add a new column and would like to extend the get method, I have to add one if xxx is not None like this below:
#classmethod
def get(cls, cluster=None, user=None):
query = db.session.query(Ticket)
if cluster is not None:
query = query.filter(Ticket.cluster==cluster)
if user is not None:
query = query.filter(Ticket.user==user)
return query.one()
Is there any way I could make this more efficient? If I have too many columns, the get method would become so ugly.
As always, if you don't want to write something repetitive, use a loop:
#classmethod
def get(cls, **kwargs):
query = db.session.query(Ticket)
for k, v in kwargs.items():
query = query.filter(getattr(table, k) == v)
return query.one()
Because we're no longer setting the cluster=None/user=None as defaults (but instead depending on things that weren't specified by the caller simply never being added to kwargs), we no longer need to prevent filters for null values from being added: The only way a null value will end up in the argument list is if the user actually asked to search for a value of None; so this new code is able to honor that request should it ever take place.
If you prefer to retain the calling convention where cluster and user can be passed positionally (but the user can't search for a value of None), see the initial version of this answer.
Assume the following setup:
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyClass(Base):
id = Column(Integer, primary_key=True)
name = Column(String)
The normal paradigm to query the DB with SQLAlchemy is to do the following:
Session = sessionmaker()
engine = 'some_db_location_string'
session = Session(bind=engine)
session.query(MyClass).filter(MyClass.id == 1).first()
Suppose, I want to simplify the query to the following:
MyClass(s).filter(MyClass.id == 1).first()
OR
MyClass(s).filter(id == 1).first()
How would I do that? My first attempt at that to use a model Mixin class failed. This is what I tried:
class ModelMixins(object)
def __init__(self, session):
self.session = session
def filter(self, *args):
self.session.query(self).filter(*args)
# Redefine MyClass to use the above class
class MyClass(ModelMixins, Base):
id = Column(Integer, primary_key=True)
name = Column(String)
The main failure seems to be that I can't quite transfer the expression 'MyClass.id == 1' to the actual filter function that is part of the session object.
Folks may ask why would I want to do:
MyClass(s).filter(id == 1).first()
I have seen something similar like this used before and thought that the syntax becomes so much cleaner I can achieve this. I wanted to replicate this but have not been able to. Being able to do something like this:
def get_stuff(some_id):
with session_scope() as s:
rec = MyClass(s).filter(MyClass.id== some_id').first()
if rec:
return rec.name
else:
return None
...seems to be the cleanest way of doing things. For one, session management is kept separate. Secondly, the query itself is simplified. Having a Mixin class like this would allow me to add the filter functionality to any number of classes...So can someone help in this regard?
session.query takes a class; you're giving it self, which is an instance. Replace your filter method with:
def filter(self, *args):
return session.query(self.__class__).filter(*args)
and at least this much works:
In [45]: MyClass(session).filter(MyClass.id==1)
Out[45]: <sqlalchemy.orm.query.Query at 0x10e0bbe80>
The generated SQL looks right, too (newlines added for clarity):
In [57]: str(MyClass(session).filter(MyClass.id==1))
Out[57]: 'SELECT "MyClass".id AS "MyClass_id", "MyClass".name AS "MyClass_name"
FROM "MyClass"
WHERE "MyClass".id = ?'
No guarantees there won't be oddities; I've never tried anything like this before.
Ive been using this mixin to good success. Most likely not the most efficient thing in the world and I am no expert. I define a date_created column for every table
class QueryBuilder:
"""
This class describes a query builer.
"""
q_debug = False
def query_from_dict(self, db_session: Session, **q_params: dict):
"""
Creates a query.
:param db_session: The database session
:type db_session: Session
:param q_params: The quarter parameters
:type q_params: dictionary
"""
q_base = db_session.query(type(self))
for param, value in q_params.items():
if param == 'start_date':
q_base = q_base.filter(
type(self).__dict__.get('date_created') >= value
)
elif param == 'end_date':
q_base = q_base.filter(
type(self).__dict__.get('date_created') <= value
)
elif 'like' in param:
param = param.replace('_like', '')
member = type(self).__dict__.get(param)
if member:
q_base = q_base.filter(member.ilike(f'%{value}%'))
else:
q_base = q_base.filter(
type(self).__dict__.get(param) == value
)
if self.q_debug:
print(q_base)
return q_base
I am building an Open Source Project, Python MongoDB ORM (for Flask especially) using flask_pymongo and I am kind of stuck at building dynamic conditions.
Below code is what I have written in corresponding files
Model.py
from app.database import Database
class Model:
conditions = {"and":[], "or":[], "in":[]}
operators = {
"!=": "$ne",
"<": "$lt",
">": "$gt",
"<=": "$lte",
">=": "$gte",
"in": "$in",
"not in":"$nin",
"and": "$and",
"or": "$or"
}
def __init__(self):
# collection property from User class
# Database class takes collection to fire MongoDB queries
self.db = Database(self.collection)
def where(self, field, operator, value=None):
if value is None:
# to enable Model.where("first_name", "John")
value = operator
operator = "="
self._handle_condition("and", field, operator, value)
# to enable Model.where().where_or() and etc
return self
def where_or(self, field, operator, value=None):
if value is None:
# to enable Model.where("first_name", "John")
value = operator
operator = "="
self._handle_condition("or", field, operator, value)
# to enable Model.where().where_or() and etc
return self
def _handle_condition(self, type, field, operator, value):
self.conditions[type].append({"field":field, "operator":operator, value:value})
def get(self):
filetrs = {}
for type in self.conditions:
filetrs[self.operators[type]] = []
for condition in self.conditions[type]:
if condition["operator"] == "=":
filter = {condition["field"]:condition["value"]}
else:
filter = {condition["field"]:{self.operators[condition["operator"]]:condition["value"]}}
filetrs[self.operators[type]].append(filter)
return self.db.find(filters)
User.py
from app.Model import Model
class UserModel(Model):
# MongoDB collection name
collection = "users"
def __init__(self):
Model.__init__(self)
User = UserModel()
What I want to achieve is, from UserController.py where User.py is imported and used like the mentioned code.
Where multiple conditions are being added using where and where_or Model methods, get methods is parsing all the conditions and passing as filter to find method
UserController.py
from app.User import User
class UserController:
def index(self):
# Should return all the users where _id is not blank or their first_name is equal to John
return User.where("_id", "!=", "").where_or("first_name", "John").get()
The problem is this is not working at it should be, it seems working fine for any one condition, where or where_or but when I try to add multiple where and where_or conditions it is not working.
Your help is really appreciated.
PS: This question seems to have lots of code but to make you understand the complete scenario I had to, please feel free to comment if you still need any clarifications.
Eagerly looking forward.
Essentially I have a SQLAlchemy query that looks like this:
foos = Foo.query.filter(Foo.expiration < cutoff)
valid_foos = []
for foo in foos:
last_bar = foo.bars.order_by('created_at desc').first()
if last_bar.state != 'fatal':
valid_foos.append(foo)
The goal being to select all the foos for which the first related bar's state is not "fatal". It seems like subqueries might help here. However, I'm struggling to grasp how I can express last_bar = foo.bars.order_by('created_at desc').first() in this way.
I think the easiest and most versatile way of doing it is using Hybrid Attributes extension. When you extend you model as per below:
class Foo(Base):
__tablename__ = "foo"
# ...
#hybrid_property
def last_bar_state(self):
_last_bar = self.bars.order_by(Bar.created_at.desc()).first()
return _last_bar.state
#last_bar_state.expression
def _last_bar_state_expression(cls):
q = (select([Bar.state.label("state")])
.where(Bar.foo_id == cls.foo_id)
.order_by(Bar.created_at.desc())
.limit(1)
).label("last_bar_state_sub")
return q
you will be able to use last_bar_state both in-memory, as well as in the query:
foos = session.query(Foo).filter(Foo.expiration < cutoff)
foos = foos.filter(Foo.last_bar_state != 'fatal').all()
Use a subquery to find the latest Bar for each Foo. Then use that to join and filter Bar when querying Foo.
sub = db.session.query(
Foo.id,
db.func.max(Bar.created_at).label('latest')
).join(Foo.bars).group_by(Foo.id).subquery()
foos = db.session.query(Foo).join(Bar, db.and_(
Bar.foo_id == sub.c.id,
Bar.created_at == sub.c.latest
)).filter(Bar.status != 'fatal').all()
This will not select Foo if it has no Bar, you can use an outer join instead for that.
Looking at the bottom of the post you can see i have three classes. The code here is pseudo code written on the fly and untested however it adequately shows my problem. If we need the actual classes I can update this question tomorrow when at work. So ignore syntax issues and code that only represents a thought rather than the actual "code" that would do what i describe there.
Question 1
If you look at the Item search class method you can see that when the user does a search i call search on the base class then based on that result return the correct class/object. This works but seems kludgy. Is there a better way to do this?
Question 2
If you look at the KitItem class you can see that I am overriding the list price. If the flag calc_list is set to true then I sum the list price of the components and return that as the list price for the kit. If its not marked as true I want to return the "base" list price. However as far as I know there is no way to access a parent attribute since in a normal setup it would be meaningless but with sqlalchemy and shared table inheritance it could be useful.
TIA
class Item(DeclarativeBase):
__tablename__ = 'items'
item_id = Column(Integer,primary_key=True,autoincrement=True)
sku = Column(Unicode(50),nullable=False,unique=True)
list_price = Column(Float)
cost_price = Column(Float)
item_type = Column(Unicode(1))
__mapper_args__ = {'polymorphic_on': item_type}
__
def __init__(self,sku,list_price,cost_price):
self.sku = sku
self.list_price = list_price
self.cost_price = cost_price
#classmethod
def search(cls):
"""
" search based on sku, description, long description
" return item as proper class
"""
item = DBSession.query(cls).filter(...) #do search stuff here
if item.item_type == 'K': #Better way to do this???
return DBSession.query(KitItem).get(item.item_id)
class KitItem(Item):
__mapper_args__ = {'polymorphic_identity': 'K'}
calc_list = Column(Boolean,nullable=False,default=False)
#property
def list_price(self):
if self.calc_list:
list_price = 0.0
for comp in self.components:
list_price += comp.component.list_price * comp.qty
return list_price
else:
#need help here
item = DBSession.query(Item).get(self.item_id)
return item.list_price
class KitComponent(DeclarativeBase):
__tablename__ = "kit_components"
kit_id = Column(Integer,ForeignKey('items.item_id'),primarykey=True)
component_id = Column(Integer,ForeignKey('items.item_id'),primarykey=True)
qty = Column(Integer,nullable=False, default=1)
kit = relation(KitItem,backref=backref("components"))
component = relation(Item)
Answer-1: in fact you do not need to do anything special here: given that you configured your inheritance hierarchy properly, your query will already return proper class for every row (Item or KitItem). This is the advantage of the ORM part. What you could do though is to configure the query to immediatelly load also the additional columns which do belong to children of Item (from your code this is only calc_list column), which you can do by specifying with_polymorphic('*'):
#classmethod
def search(cls):
item = DBSession.query(cls).with_polymorphic('*').filter(...) #do search stuff here
return item
Read more on this in Basic Control of Which Tables are Queried.
To see the difference, enabled SQL logging, and compare your tests scripts with and without with_polymorphic(...) - you will most probably require less SQL statements being executed.
Answer-2: I would not override one entry attributed with one which is purely computed. Instead I would just create another computed attribute (lets call it final_price), which would look like following for each of two classes:
class Item(Base):
...
#property
def total_price(self):
return self.list_price
class KitItem(Item):
...
#property
def total_price(self):
if self.calc_list:
_price = 0.0
for comp in self.components:
_price += comp.component.list_price * comp.qty
return _price
else:
# #note: again, you do not need to perform any query here at all, as *self* is that you need
return self.list_price
Also in this case, you might think of configuring the relationship KitItem.components to be eagerly loaded, so that the calculation of the total_price will not trigger additional SQL. But you have to decide yourself if this is beneficial for your use cases (again, analyse the SQLs generated in your scenario).