what I try to do is to create a query that finds all records where a date is greater than today - days_before_activation.
For this I use a #hybrid_property that shows the correct start_day (today - days_before_activation).
The issue is, that timedelta does not work in filter queries, at least with sqllite.
Error:
E TypeError: unsupported type for timedelta days component:
InstrumentedAttribute
For this I created an additional column expression (#start.expression). To add the days using plain sql. Unfortunately it does not work when I use the cls.days_before_activation value as part of the expression. The result is always None. However, when I hardcode the integer value it does work. So it looks like I'm doing something wrong with using the value of the property as part of the expression.
Not Working:
#start.expression
def start(cls):
return func.date(datetime.today(), f'+{cls.days_before_activation} days')
Working:
#start.expression
def start(cls):
return func.date(datetime.today(), '+ 10 days')
Any help are much appreciated.
class DefPayoutOffer(db.Model):
__tablename__ = 'def_payout_option_offer'
id = db.Column(db.Integer, primary_key=True)
visual_text = db.Column(db.String(1000), unique=False, nullable=False)
days_before_activation = db.Column(db.Integer, nullable=True)
#hybrid_property
def start(self):
return datetime.today() - timedelta(days=self.days_before_activation)
#start.expression
def start(cls):
return func.date(datetime.today(), f'+{cls.days_before_activation} days')
Query:
DefPayoutOffer.query.filter(DefPayoutOffer.start > created_date).all()
UPDATE:
After finding the solution with the help of python_user.
I had the challenge to make it work for different dialects.
For this I had to move away from the column expression approach to creating a expression.FunctionElement. Advantage is that it can be made dialect specific.
class StartCheck(expression.FunctionElement):
name = 'start_check'
inherit_cache = True
#compiles(StartCheck, 'otherDialect')
def compile(element, compiler, **kw):
return "foo"
#compiles(StartCheck, 'sqlite')
def compile(element, compiler, **kw):
return compiler.process(func.date('now', '+' + expression.cast(list(element.clauses)[0], Unicode) + ' days'))
DefPayoutOffer.query.filter(StartCheck(DefPayoutOffer.days_before_activation) >= date.today()).first()
You have to use expression.cast with types.Unicode to get what you want.
In addition to what you have, you need these imports and then change start.expression like so
from sqlalchemy.types import Unicode
from sqlalchemy.sql import expression
#start.expression
def start(cls):
return func.date('now', '+' + expression.cast(cls.days_before_activation, Unicode) + ' days')
Also as a side note, you can use date('now') to get the current date, you do not have to use datetime.today() like you have shown.
PS: 999 rep, enjoy 1k+ with that upvote
Edit: To select based on dialect, sqlite / postgres
If you have access to engine that is used by the session you can use this
#start.expression
def start(cls):
if engine.dialect.name == 'sqlite':
return func.date('now', '+' + cast(cls.days_before_activation, Unicode) + ' days')
elif engine.dialect.name == 'postgresql':
return func.current_date() + cls.days_before_activation
Related
Assume the following setup:
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class MyClass(Base):
id = Column(Integer, primary_key=True)
name = Column(String)
The normal paradigm to query the DB with SQLAlchemy is to do the following:
Session = sessionmaker()
engine = 'some_db_location_string'
session = Session(bind=engine)
session.query(MyClass).filter(MyClass.id == 1).first()
Suppose, I want to simplify the query to the following:
MyClass(s).filter(MyClass.id == 1).first()
OR
MyClass(s).filter(id == 1).first()
How would I do that? My first attempt at that to use a model Mixin class failed. This is what I tried:
class ModelMixins(object)
def __init__(self, session):
self.session = session
def filter(self, *args):
self.session.query(self).filter(*args)
# Redefine MyClass to use the above class
class MyClass(ModelMixins, Base):
id = Column(Integer, primary_key=True)
name = Column(String)
The main failure seems to be that I can't quite transfer the expression 'MyClass.id == 1' to the actual filter function that is part of the session object.
Folks may ask why would I want to do:
MyClass(s).filter(id == 1).first()
I have seen something similar like this used before and thought that the syntax becomes so much cleaner I can achieve this. I wanted to replicate this but have not been able to. Being able to do something like this:
def get_stuff(some_id):
with session_scope() as s:
rec = MyClass(s).filter(MyClass.id== some_id').first()
if rec:
return rec.name
else:
return None
...seems to be the cleanest way of doing things. For one, session management is kept separate. Secondly, the query itself is simplified. Having a Mixin class like this would allow me to add the filter functionality to any number of classes...So can someone help in this regard?
session.query takes a class; you're giving it self, which is an instance. Replace your filter method with:
def filter(self, *args):
return session.query(self.__class__).filter(*args)
and at least this much works:
In [45]: MyClass(session).filter(MyClass.id==1)
Out[45]: <sqlalchemy.orm.query.Query at 0x10e0bbe80>
The generated SQL looks right, too (newlines added for clarity):
In [57]: str(MyClass(session).filter(MyClass.id==1))
Out[57]: 'SELECT "MyClass".id AS "MyClass_id", "MyClass".name AS "MyClass_name"
FROM "MyClass"
WHERE "MyClass".id = ?'
No guarantees there won't be oddities; I've never tried anything like this before.
Ive been using this mixin to good success. Most likely not the most efficient thing in the world and I am no expert. I define a date_created column for every table
class QueryBuilder:
"""
This class describes a query builer.
"""
q_debug = False
def query_from_dict(self, db_session: Session, **q_params: dict):
"""
Creates a query.
:param db_session: The database session
:type db_session: Session
:param q_params: The quarter parameters
:type q_params: dictionary
"""
q_base = db_session.query(type(self))
for param, value in q_params.items():
if param == 'start_date':
q_base = q_base.filter(
type(self).__dict__.get('date_created') >= value
)
elif param == 'end_date':
q_base = q_base.filter(
type(self).__dict__.get('date_created') <= value
)
elif 'like' in param:
param = param.replace('_like', '')
member = type(self).__dict__.get(param)
if member:
q_base = q_base.filter(member.ilike(f'%{value}%'))
else:
q_base = q_base.filter(
type(self).__dict__.get(param) == value
)
if self.q_debug:
print(q_base)
return q_base
Based on this question i wrote this code:
class CourseManager(models.Manager):
def current(self):
return self.filter(date_start < datetime.datetime.now()).filter(date_end > datetime.datetime.now())
class PlannedCourse(models.Model):
course = models.ForeignKey(ECourse)
date_start = models.DateTimeField()
date_end = models.DateTimeField(blank=True)
...
objects = CourseManager()
def __str__(self):
return self.course.name + ' ' + self.date_start.strftime("%d-%m-%Y %H:%M") + ' - ' + self.date_end.strftime("%d-%m-%Y %H:%M")
but when I try to run PlannedCourse.objects.current() I get the error:
NameError: name 'date_start' is not defined.
I'm a newbie and don't understand why this happens :(
You should use fields lookups: __gt instead of > and __lt instead of <:
class CourseManager(models.Manager):
def current(self):
return self.filter(date_start__lt=datetime.datetime.now(),
date_end__gt=datetime.datetime.now())
You're comparing an undefined python object (date_start) with a date in python. However, the goal of a queryset is to compare objects in the database. Even if date_start was defined, this would be the equivalent of calling .filter(True) or .filter(False).
To tell Django to check in the database, you have to pass keyword arguments to the filter that are converted to a query by the ORM:
return self.filter(date_start__lt=datetime.datetime.now(),
date_end__gt=datetime.datetime.now())
Note that you're not actually comparing date_start and datetime.datetime.now(), the .filter function uses **kwargs, and gets the following arguments as a dictionary:
{
'date_start__lt': datetime.datetime.now(),
'date_end__gt': datetime.datetime.now(),
}
The __lt and __gt are field lookups that tell the queryset how to perform the comparison in the database.
I have the following SQLAlchemy class defined:
Base = sqlalchemy.ext.declarative.declarative_base()
class NSASecrets(Base):
__tablename__ = 'nsasecrets';
id = sqlalchemy.Column(sqlalchemy.Integer, primary_key=True);
text = sqlalchemy.Column(sqlalchemy.String);
author = sqlalchemy.Column(sqlalchemy.String);
Now what I want to do is to be able to mask "author" field depending on some logic, something like:
if (allowed):
nsasecrets = session.query(NSASecrets,**mask=False**);
else:
nsasecrets = session.query(NSASecrets,**mask=True**);
for nsasecret in nsasecrets:
print '{0} {1}'.format(author, text);
So depending on this "mask" parameter I would like output to be "John Smith" in False case - output not masked, or "J*** **h" when output is masked. Now obviously I could do it in this very print, but problem is that prints are scattered around the code and the only way I see to do this in controlled centralized manner is to create SQLAlchemy objects with already masked values. So is there any well known solution to this? Or should I just create my own session manager that would overload "query" interface or am I missing some other possible solutions to this?
Thanks
this is typically the kind of thing in Python we do with something called descriptors. A simple way to combine descriptors with SQLAlchemy mapped columns is to use the synonym, though synonym is a bit dated at this point, in favor of a less "magic" system called hybrids. Either can be used here, below is an example of a hybrid:
from sqlalchemy import *
from sqlalchemy.orm import *
from sqlalchemy.ext.declarative import declarative_base, synonym_for
from sqlalchemy.ext.hybrid import hybrid_property
Base = declarative_base()
class NSASecrets(Base):
__tablename__ = 'nsasecrets'
id = Column(Integer, primary_key=True)
_text = Column("text", String)
_author = Column("author", String)
def _obfuscate(self, value):
return "%s%s" % (value[0], ("*" * (len(value) - 2)))
#hybrid_property
def text(self):
return self._obfuscate(self._text)
#text.setter
def text(self, value):
self._text = value
#text.expression
def text(cls):
return cls._text
#hybrid_property
def author(self):
return self._obfuscate(self._author)
#author.setter
def author(self, value):
self._author = value
#author.expression
def author(cls):
return cls._author
n1 = NSASecrets(text='some text', author="some author")
print n1.text
print n1.author
note that this doesn't have much to do with querying. The idea of formatting the data as it arrives in a rowset is a different way to go, and there's some ways to make that happen too, though if you're only concerned about print statements that refer to "text" and "author", it's likely more convenient to keep that as a python access pattern.
Or how do I make this thing work?
I have an Interval object:
class Interval(Base):
__tablename__ = 'intervals'
id = Column(Integer, primary_key=True)
start = Column(DateTime)
end = Column(DateTime, nullable=True)
task_id = Column(Integer, ForeignKey('tasks.id'))
#hybrid_property #used to just be #property
def hours_spent(self):
end = self.end or datetime.datetime.now()
return (end-start).total_seconds()/60/60
And a Task:
class Task(Base):
__tablename__ = 'tasks'
id = Column(Integer, primary_key=True)
title = Column(String)
intervals = relationship("Interval", backref="task")
#hybrid_property # Also used to be just #property
def hours_spent(self):
return sum(i.hours_spent for i in self.intervals)
Add all the typical setup code, of course.
Now when I try to do session.query(Task).filter(Task.hours_spent > 3).all()
I get NotImplementedError: <built-in function getitem> from the sum(i.hours_spent... line.
So I was looking at this part of the documentation and theorized that there might be some way that I can write something that will do what I want. This part also looks like it may be of use, and I'll be looking at it while waiting for an answer here ;)
For a simple example of SQLAlchemy's coalesce function, this may help: Handling null values in a SQLAlchemy query - equivalent of isnull, nullif or coalesce.
Here are a couple of key lines of code from that post:
from sqlalchemy.sql.functions import coalesce
my_config = session.query(Config).order_by(coalesce(Config.last_processed_at, datetime.date.min)).first()
SQLAlchemy is not smart enough to build SQL expression tree from these operands, you have to use explicit propname.expression decorator to provide it. But then comes another problem: there is no portable way to convert interval to hours in-database. You'd use TIMEDIFF in MySQL, EXTRACT(EPOCH FROM ... ) / 3600 in PostgreSQL etc. I suggest changing properties to return timedelta instead, and comparing apples to apples.
from sqlalchemy import select, func
class Interval(Base):
...
#hybrid_property
def time_spent(self):
return (self.end or datetime.now()) - self.start
#time_spent.expression
def time_spent(cls):
return func.coalesce(cls.end, func.current_timestamp()) - cls.start
class Task(Base):
...
#hybrid_property
def time_spent(self):
return sum((i.time_spent for i in self.intervals), timedelta(0))
#time_spent.expression
def hours_spent(cls):
return (select([func.sum(Interval.time_spent)])
.where(cls.id==Interval.task_id)
.label('time_spent'))
The final query is:
session.query(Task).filter(Task.time_spent > timedelta(hours=3)).all()
which translates to (on PostgreSQL backend):
SELECT task.id AS task_id, task.title AS task_title
FROM task
WHERE (SELECT sum(coalesce(interval."end", CURRENT_TIMESTAMP) - interval.start) AS sum_1
FROM interval
WHERE task.id = interval.task_id) > %(param_1)s
I needed to use the text function and could not use 0 as an integer.
import sqlalchemy as sa
session.query(sa.func.coalesce(table1.col1, sa.text("0"))).all()
There is a complete example of making a func action similar to coalesc or nvl.
Note how it takes in arguements, and renders an expression... in this case NVL(a, b) when used with Oracle.
http://docs.sqlalchemy.org/en/latest/core/compiler.html#subclassing-guidelines
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.sql.expression import FunctionElement
class coalesce(FunctionElement):
name = 'coalesce'
#compiles(coalesce)
def compile(element, compiler, **kw):
return "coalesce(%s)" % compiler.process(element.clauses)
#compiles(coalesce, 'oracle')
def compile(element, compiler, **kw):
if len(element.clauses) > 2:
raise TypeError("coalesce only supports two arguments on Oracle")
return "nvl(%s)" % compiler.process(element.clauses)
Then when you want to use it...
from my_oracle_functions_sqla import coalesce
select([coalesce(A.value, '---')]) # etc
Hope that helps.
I have this class:
class Monitor(db.Model):
'''
Base Monitor class.
'''
__tablename__ = 'monitor'
id = db.Column(db.Integer(), primary_key=True)
last_checked = db.Column(db.DateTime(timezone=False))
poll_interval = db.Column(db.Interval(),
default=datetime.timedelta(seconds=300))
And I have this query where I attempt to return only objects that haven't been checked since (now - interval):
monitors = db.session.query(Monitor).\
filter(or_(Monitor.last_checked < (datetime.utcnow() - Monitor.poll_interval)),
Monitor.last_checked == None).\
all()
But the query returns nothing. I'm having a hard time figuring out the proper way to do this. Am I on the right track or am I missing something? I'm using MySQL as the database.
Your parenthesis are wrong. I believe what you want is:
monitors = db.session.query(Monitor).\
filter(or_(Monitor.last_checked < (datetime.utcnow() - Monitor.poll_interval),
Monitor.last_checked == None)).\
all()