Or how do I make this thing work?
I have an Interval object:
class Interval(Base):
__tablename__ = 'intervals'
id = Column(Integer, primary_key=True)
start = Column(DateTime)
end = Column(DateTime, nullable=True)
task_id = Column(Integer, ForeignKey('tasks.id'))
#hybrid_property #used to just be #property
def hours_spent(self):
end = self.end or datetime.datetime.now()
return (end-start).total_seconds()/60/60
And a Task:
class Task(Base):
__tablename__ = 'tasks'
id = Column(Integer, primary_key=True)
title = Column(String)
intervals = relationship("Interval", backref="task")
#hybrid_property # Also used to be just #property
def hours_spent(self):
return sum(i.hours_spent for i in self.intervals)
Add all the typical setup code, of course.
Now when I try to do session.query(Task).filter(Task.hours_spent > 3).all()
I get NotImplementedError: <built-in function getitem> from the sum(i.hours_spent... line.
So I was looking at this part of the documentation and theorized that there might be some way that I can write something that will do what I want. This part also looks like it may be of use, and I'll be looking at it while waiting for an answer here ;)
For a simple example of SQLAlchemy's coalesce function, this may help: Handling null values in a SQLAlchemy query - equivalent of isnull, nullif or coalesce.
Here are a couple of key lines of code from that post:
from sqlalchemy.sql.functions import coalesce
my_config = session.query(Config).order_by(coalesce(Config.last_processed_at, datetime.date.min)).first()
SQLAlchemy is not smart enough to build SQL expression tree from these operands, you have to use explicit propname.expression decorator to provide it. But then comes another problem: there is no portable way to convert interval to hours in-database. You'd use TIMEDIFF in MySQL, EXTRACT(EPOCH FROM ... ) / 3600 in PostgreSQL etc. I suggest changing properties to return timedelta instead, and comparing apples to apples.
from sqlalchemy import select, func
class Interval(Base):
...
#hybrid_property
def time_spent(self):
return (self.end or datetime.now()) - self.start
#time_spent.expression
def time_spent(cls):
return func.coalesce(cls.end, func.current_timestamp()) - cls.start
class Task(Base):
...
#hybrid_property
def time_spent(self):
return sum((i.time_spent for i in self.intervals), timedelta(0))
#time_spent.expression
def hours_spent(cls):
return (select([func.sum(Interval.time_spent)])
.where(cls.id==Interval.task_id)
.label('time_spent'))
The final query is:
session.query(Task).filter(Task.time_spent > timedelta(hours=3)).all()
which translates to (on PostgreSQL backend):
SELECT task.id AS task_id, task.title AS task_title
FROM task
WHERE (SELECT sum(coalesce(interval."end", CURRENT_TIMESTAMP) - interval.start) AS sum_1
FROM interval
WHERE task.id = interval.task_id) > %(param_1)s
I needed to use the text function and could not use 0 as an integer.
import sqlalchemy as sa
session.query(sa.func.coalesce(table1.col1, sa.text("0"))).all()
There is a complete example of making a func action similar to coalesc or nvl.
Note how it takes in arguements, and renders an expression... in this case NVL(a, b) when used with Oracle.
http://docs.sqlalchemy.org/en/latest/core/compiler.html#subclassing-guidelines
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.sql.expression import FunctionElement
class coalesce(FunctionElement):
name = 'coalesce'
#compiles(coalesce)
def compile(element, compiler, **kw):
return "coalesce(%s)" % compiler.process(element.clauses)
#compiles(coalesce, 'oracle')
def compile(element, compiler, **kw):
if len(element.clauses) > 2:
raise TypeError("coalesce only supports two arguments on Oracle")
return "nvl(%s)" % compiler.process(element.clauses)
Then when you want to use it...
from my_oracle_functions_sqla import coalesce
select([coalesce(A.value, '---')]) # etc
Hope that helps.
Related
what I try to do is to create a query that finds all records where a date is greater than today - days_before_activation.
For this I use a #hybrid_property that shows the correct start_day (today - days_before_activation).
The issue is, that timedelta does not work in filter queries, at least with sqllite.
Error:
E TypeError: unsupported type for timedelta days component:
InstrumentedAttribute
For this I created an additional column expression (#start.expression). To add the days using plain sql. Unfortunately it does not work when I use the cls.days_before_activation value as part of the expression. The result is always None. However, when I hardcode the integer value it does work. So it looks like I'm doing something wrong with using the value of the property as part of the expression.
Not Working:
#start.expression
def start(cls):
return func.date(datetime.today(), f'+{cls.days_before_activation} days')
Working:
#start.expression
def start(cls):
return func.date(datetime.today(), '+ 10 days')
Any help are much appreciated.
class DefPayoutOffer(db.Model):
__tablename__ = 'def_payout_option_offer'
id = db.Column(db.Integer, primary_key=True)
visual_text = db.Column(db.String(1000), unique=False, nullable=False)
days_before_activation = db.Column(db.Integer, nullable=True)
#hybrid_property
def start(self):
return datetime.today() - timedelta(days=self.days_before_activation)
#start.expression
def start(cls):
return func.date(datetime.today(), f'+{cls.days_before_activation} days')
Query:
DefPayoutOffer.query.filter(DefPayoutOffer.start > created_date).all()
UPDATE:
After finding the solution with the help of python_user.
I had the challenge to make it work for different dialects.
For this I had to move away from the column expression approach to creating a expression.FunctionElement. Advantage is that it can be made dialect specific.
class StartCheck(expression.FunctionElement):
name = 'start_check'
inherit_cache = True
#compiles(StartCheck, 'otherDialect')
def compile(element, compiler, **kw):
return "foo"
#compiles(StartCheck, 'sqlite')
def compile(element, compiler, **kw):
return compiler.process(func.date('now', '+' + expression.cast(list(element.clauses)[0], Unicode) + ' days'))
DefPayoutOffer.query.filter(StartCheck(DefPayoutOffer.days_before_activation) >= date.today()).first()
You have to use expression.cast with types.Unicode to get what you want.
In addition to what you have, you need these imports and then change start.expression like so
from sqlalchemy.types import Unicode
from sqlalchemy.sql import expression
#start.expression
def start(cls):
return func.date('now', '+' + expression.cast(cls.days_before_activation, Unicode) + ' days')
Also as a side note, you can use date('now') to get the current date, you do not have to use datetime.today() like you have shown.
PS: 999 rep, enjoy 1k+ with that upvote
Edit: To select based on dialect, sqlite / postgres
If you have access to engine that is used by the session you can use this
#start.expression
def start(cls):
if engine.dialect.name == 'sqlite':
return func.date('now', '+' + cast(cls.days_before_activation, Unicode) + ' days')
elif engine.dialect.name == 'postgresql':
return func.current_date() + cls.days_before_activation
Assume the following setup:
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyClass(Base):
id = Column(Integer, primary_key=True)
name = Column(String)
The normal paradigm to query the DB with SQLAlchemy is to do the following:
Session = sessionmaker()
engine = 'some_db_location_string'
session = Session(bind=engine)
session.query(MyClass).filter(MyClass.id == 1).first()
Suppose, I want to simplify the query to the following:
MyClass(s).filter(MyClass.id == 1).first()
OR
MyClass(s).filter(id == 1).first()
How would I do that? My first attempt at that to use a model Mixin class failed. This is what I tried:
class ModelMixins(object)
def __init__(self, session):
self.session = session
def filter(self, *args):
self.session.query(self).filter(*args)
# Redefine MyClass to use the above class
class MyClass(ModelMixins, Base):
id = Column(Integer, primary_key=True)
name = Column(String)
The main failure seems to be that I can't quite transfer the expression 'MyClass.id == 1' to the actual filter function that is part of the session object.
Folks may ask why would I want to do:
MyClass(s).filter(id == 1).first()
I have seen something similar like this used before and thought that the syntax becomes so much cleaner I can achieve this. I wanted to replicate this but have not been able to. Being able to do something like this:
def get_stuff(some_id):
with session_scope() as s:
rec = MyClass(s).filter(MyClass.id== some_id').first()
if rec:
return rec.name
else:
return None
...seems to be the cleanest way of doing things. For one, session management is kept separate. Secondly, the query itself is simplified. Having a Mixin class like this would allow me to add the filter functionality to any number of classes...So can someone help in this regard?
session.query takes a class; you're giving it self, which is an instance. Replace your filter method with:
def filter(self, *args):
return session.query(self.__class__).filter(*args)
and at least this much works:
In [45]: MyClass(session).filter(MyClass.id==1)
Out[45]: <sqlalchemy.orm.query.Query at 0x10e0bbe80>
The generated SQL looks right, too (newlines added for clarity):
In [57]: str(MyClass(session).filter(MyClass.id==1))
Out[57]: 'SELECT "MyClass".id AS "MyClass_id", "MyClass".name AS "MyClass_name"
FROM "MyClass"
WHERE "MyClass".id = ?'
No guarantees there won't be oddities; I've never tried anything like this before.
Ive been using this mixin to good success. Most likely not the most efficient thing in the world and I am no expert. I define a date_created column for every table
class QueryBuilder:
"""
This class describes a query builer.
"""
q_debug = False
def query_from_dict(self, db_session: Session, **q_params: dict):
"""
Creates a query.
:param db_session: The database session
:type db_session: Session
:param q_params: The quarter parameters
:type q_params: dictionary
"""
q_base = db_session.query(type(self))
for param, value in q_params.items():
if param == 'start_date':
q_base = q_base.filter(
type(self).__dict__.get('date_created') >= value
)
elif param == 'end_date':
q_base = q_base.filter(
type(self).__dict__.get('date_created') <= value
)
elif 'like' in param:
param = param.replace('_like', '')
member = type(self).__dict__.get(param)
if member:
q_base = q_base.filter(member.ilike(f'%{value}%'))
else:
q_base = q_base.filter(
type(self).__dict__.get(param) == value
)
if self.q_debug:
print(q_base)
return q_base
I have the following SQLAlchemy class defined:
Base = sqlalchemy.ext.declarative.declarative_base()
class NSASecrets(Base):
__tablename__ = 'nsasecrets';
id = sqlalchemy.Column(sqlalchemy.Integer, primary_key=True);
text = sqlalchemy.Column(sqlalchemy.String);
author = sqlalchemy.Column(sqlalchemy.String);
Now what I want to do is to be able to mask "author" field depending on some logic, something like:
if (allowed):
nsasecrets = session.query(NSASecrets,**mask=False**);
else:
nsasecrets = session.query(NSASecrets,**mask=True**);
for nsasecret in nsasecrets:
print '{0} {1}'.format(author, text);
So depending on this "mask" parameter I would like output to be "John Smith" in False case - output not masked, or "J*** **h" when output is masked. Now obviously I could do it in this very print, but problem is that prints are scattered around the code and the only way I see to do this in controlled centralized manner is to create SQLAlchemy objects with already masked values. So is there any well known solution to this? Or should I just create my own session manager that would overload "query" interface or am I missing some other possible solutions to this?
Thanks
this is typically the kind of thing in Python we do with something called descriptors. A simple way to combine descriptors with SQLAlchemy mapped columns is to use the synonym, though synonym is a bit dated at this point, in favor of a less "magic" system called hybrids. Either can be used here, below is an example of a hybrid:
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base, synonym_for
from sqlalchemy.ext.hybrid import hybrid_property
Base = declarative_base()
class NSASecrets(Base):
__tablename__ = 'nsasecrets'
id = Column(Integer, primary_key=True)
_text = Column("text", String)
_author = Column("author", String)
def _obfuscate(self, value):
return "%s%s" % (value[0], ("*" * (len(value) - 2)))
#hybrid_property
def text(self):
return self._obfuscate(self._text)
#text.setter
def text(self, value):
self._text = value
#text.expression
def text(cls):
return cls._text
#hybrid_property
def author(self):
return self._obfuscate(self._author)
#author.setter
def author(self, value):
self._author = value
#author.expression
def author(cls):
return cls._author
n1 = NSASecrets(text='some text', author="some author")
print n1.text
print n1.author
note that this doesn't have much to do with querying. The idea of formatting the data as it arrives in a rowset is a different way to go, and there's some ways to make that happen too, though if you're only concerned about print statements that refer to "text" and "author", it's likely more convenient to keep that as a python access pattern.
I made this statement using flask-sqlalchemy and I've chosen to keep it in its original form. Post.query is equivalent to session.query(Post)
I attempted to make a subquery that would filter out all posts in a database which are in the draft state and not made or modified by the current user. I made this query,
Post.query\
.filter(sqlalchemy.and_(
Post.post_status != Consts.PostStatuses["Draft"],
sqlalchemy.or_(
Post.modified_by_id == current_user.get_id(),
Post.created_by_id == current_user.get_id()))
which created:
Where true AND ("Post".modified_by_id = :modified_by_id_1 OR
"Post".created_by_id = :created_by_id_1)
Expected outcome:
Where "Post".post_status != "Draft" AND (
"Post".modified_by_id = :modified_by_id_1 OR
"Post".created_by_id = :created_by_id_1)
I'm wondering, why this is happening? How can I increase the error level in SQLAlchemy? I think my project is silently failing and I would like to confirm my guess.
Update:
I used the wrong constants dictionary. One dictionary contains ints, the other contains strings (one for data base queries, one for printing).
_post_status = db.Column(
db.SmallInteger,
default=Consts.post_status["Draft"])
post_status contains integers, Consts.PostStatuses contains strings. In hind sight, really bad idea. I'm going to make a single dictionary that returns a tuple instead of two dictionaries.
#property
def post_status(self):
return Consts.post_status.get(getattr(self, "_post_status", None))
the problem is that your post_status property isn't acceptable for usage in an ORM level query, as this is a python descriptor which at the class level by default returns itself:
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class A(Base):
__tablename__ = 'a'
id = Column(Integer, primary_key=True)
_post_status = Column(String)
#property
def post_status(self):
return self._post_status
print (A.post_status)
print (A.post_status != 5678)
output:
$ python test.py
<property object at 0x10165bd08>
True
the type of usage you're looking for seems like that of a hybrid attribute, which is a SQLAlchemy-included extension to a "regular" python descriptor which produces class-level behavior that's compatible with core SQL expressions:
from sqlalchemy.ext.hybrid import hybrid_property
class A(Base):
__tablename__ = 'a'
id = Column(Integer, primary_key=True)
_post_status = Column(String)
#hybrid_property
def post_status(self):
return self._post_status
print (A.post_status)
print (A.post_status != 5678)
output:
$ python test.py
A._post_status
a._post_status != :_post_status_1
be sure to read the hybrid doc carefully including how to establish the correct SQL expression behavior, descriptors that work both at the instance and class level is a somewhat advanced Python technique.
I have this class:
class Monitor(db.Model):
'''
Base Monitor class.
'''
__tablename__ = 'monitor'
id = db.Column(db.Integer(), primary_key=True)
last_checked = db.Column(db.DateTime(timezone=False))
poll_interval = db.Column(db.Interval(),
default=datetime.timedelta(seconds=300))
And I have this query where I attempt to return only objects that haven't been checked since (now - interval):
monitors = db.session.query(Monitor).\
filter(or_(Monitor.last_checked < (datetime.utcnow() - Monitor.poll_interval)),
Monitor.last_checked == None).\
all()
But the query returns nothing. I'm having a hard time figuring out the proper way to do this. Am I on the right track or am I missing something? I'm using MySQL as the database.
Your parenthesis are wrong. I believe what you want is:
monitors = db.session.query(Monitor).\
filter(or_(Monitor.last_checked < (datetime.utcnow() - Monitor.poll_interval),
Monitor.last_checked == None)).\
all()