In SQLAlchemy, a hybrid attribute is either a property or method applied to an ORM-mapped class,
class Interval(Base):
__tablename__ = 'interval'
id = Column(Integer, primary_key=True)
start = Column(Integer, nullable=False)
end = Column(Integer, nullable=False)
def __init__(self, start, end):
self.start = start
self.end = end
#hybrid_property
def length(self):
return self.end - self.start
#hybrid_method
def contains(self,point):
return (self.start <= point) & (point < self.end)
#hybrid_method
def intersects(self, other):
return self.contains(other.start) | self.contains(other.end)
This allows for distinct behaviors at the class and instance levels, thus making it simpler to evaluate SQL statements using the same code,
>>> i1 = Interval(5, 10)
>>> i1.length
5
>>> print Session().query(Interval).filter(Interval.length > 10)
SELECT interval.id AS interval_id, interval.start AS interval_start,
interval."end" AS interval_end
FROM interval
WHERE interval."end" - interval.start > :param_1
Now in Django, if I have a property on a model,
class Person(models.Model):
first_name = models.CharField(max_length=50)
last_name = models.CharField(max_length=50)
def _get_full_name(self):
"Returns the person's full name."
return '%s %s' % (self.first_name, self.last_name)
full_name = property(_get_full_name)
It is my understanding that I can not do the following,
Person.objects.filter(full_name="John Cadengo")
Is there an equivalent of SQLAlchemy's hybrid attribute in Django? If not, is there perhaps a common workaround?
You're right that you cannot apply django queryset filter based on python properties, because filter operates on a database level. It seems that there's no equivalent of SQLAlchemy's hybrid attributes in Django.
Please, take a look here and here, may be it'll help you to find a workaround. But, I think there is no generic solution.
One of the quick possible workarounds is to implement some descriptor, that would apply expressions via the annotation.
Something like this:
from django.db import models
from django.db.models import functions
class hybrid_property:
def __init__(self, func):
self.func = func
self.name = func.__name__
self.exp = None
def __get__(self, instance, owner):
if instance is None:
return self
return self.func(instance)
def __set__(self, instance, value):
pass
def expression(self, exp):
self.exp = exp
return self
class HybridManager(models.Manager):
def get_queryset(self):
qs = super().get_queryset()
for name, value in vars(qs.model).items():
if isinstance(value, hybrid_property) and value.exp is not None:
qs = qs.annotate(**{name: value.exp(qs.model)})
return qs
class TestAttribute(models.Model):
val1 = models.CharField(max_length=256)
val2 = models.CharField(max_length=256)
objects = HybridManager()
#hybrid_property
def vals(self):
return f"{self.val1} {self.val2}"
#vals.expression
def vals(cls):
return functions.Concat(models.F("val1"), models.Value(" "), models.F("val2"))
class HybridTests(TestCase):
def setUp(self) -> None:
self.test_attr = TestAttribute.objects.create(val1="val1", val2="val2")
def test_access(self):
self.assertTrue(TestAttribute.objects.exists())
self.assertEqual(self.test_attr.vals, f"{self.test_attr.val1} {self.test_attr.val2}")
self.assertTrue(TestAttribute.objects.filter(vals=f"{self.test_attr.val1} {self.test_attr.val2}").exists())
self.assertTrue(TestAttribute.objects.filter(vals__iexact=f"{self.test_attr.val1} {self.test_attr.val2}").exists())
Related
I have a lot of unstructured data in Column(JSONB).
I would like to "standardize" this data and access it like normal Columns.
This actually does work:
class Order(Base):
__tablename__ = "orders"
id_= Column(Integer, primary_key=True, nullable=False)
type = Column(String)
data = Column(JSONB)
#hybrid_property
def email(self):
return self.data['email']
#email.setter
def email(self, value):
self.data['email'] = value
#email.expression
def email(cls):
return cls.data['email'].astext
However data can have different keys, eg: email, e_mail, email_address, etc..
The solution I'm thinking about is to "extend" main class with another class depending
on value in "type":
class Parser1:
#hybrid_property
def email(self):
return self.data['email']
#email.setter
def email(self, value):
self.data['email'] = value
#email.expression
def email(cls):
return cls.data['email'].astext
class Parser2:
#hybrid_property
def email(self):
return self.data['email_address']
#email.setter
def email(self, value):
self.data['email_address'] = value
#email.expression
def email(cls):
return cls.data['email_address'].astext
class Order(Base):
__tablename__ = "orders"
id_= Column(Integer, primary_key=True, nullable=False)
type = Column(String)
data = Column(JSONB)
if type == 'one':
extend with class "Parser1"
elif type == 'two':
extend with class "Parser2"
Would this work or is it stupid idea?
Do you have any better solution?
Any hints very welcome.
I'm new to MongoEngine and it looks like we need to create sub classes of the class Document from the mongoengine to model our DB. I'm a little concerned here because this violates the Dependency Inversion from the SOLID principles. So if I need to use another database at a later point of time, I will have to change my domain model classes which I shouldn't really be doing.
SQLAlchemy overcomes this by providing a beautiful classical mapping. Using this, the database dependent code is separated from my domain model, so I don't really need to worry about the database provider and I can easily abstract the details away should I have a need to change my database.
Is there a equivalent of this for MongoDB, preferrably in MongoEngine?
Pymongo's official doc provides a list of the existing ORM/ODM and frameworks but to my knowledge they all implement the Active Record Pattern (just like django ORM), which as you said, violates the SOLID principles but is good enough for many simple use cases.
MongoAlchemy, which was inspired by SQLAlchemy uses a concept of session so it may be closer to what you are looking for but the project is no longer maintained.
If I understand correctly, you're trying to map an object to document schema using mongoengine.
Let's create a document class for a user:
from mongoengine import Document, StringField
class UserDocument(Document):
username = StringField(required=True)
password = StringField(required=True)
email = StringField(required=True)
Now add a class method that creates new users:
from mongoengine import disconnect, connect, Document, StringField
class UserDocument(Document):
username = StringField(required=True)
password = StringField(required=True)
email = StringField(required=True)
#classmethod
def new(cls):
data = UserDocument(username=cls.username, password=cls.password, email=cls.email)
connect('test_collection')
data.save()
disconnect('test_collection')
As I understand your question, your issue in this example is that UserDocument would be aware of mongoengine thus violating the dependency inversion principle. This can be solved with a child class.
First allow inheritance in UserDocument:
...
class UserDocument(Document):
meta = {'allow_inheritance': True}
username = StringField(required=True)
...
Next we build the child:
from user_document import UserDocument
# Maps object to schema
class User(UserDocument):
def __init__(self, *args, **values):
super().__init__(*args, **values)
Next add a create method:
from user_document import UserDocument
# Maps object to schema
class User(UserDocument):
def __init__(self, *args, **values):
super().__init__(*args, **values)
def create(self, username, password, email):
self.username, self.password, self.email = username, password, email
User.new()
Now our User object inherits the UserDocument fields. UserDocument.new can be accessed directly or through the child with User.new().
from model import User
username, password, email = 'cool username', 'super secret password', 'mrcool#example.com'
User.create(User, username, password, email)
The User object is aware of UserDocument which in turn depends on mongoengine.
I apologize if I misunderstood or used incorrect vocabulary to describe the example solution. I'm relatively new, self-taught, and have no friends who code which makes discussion difficult.
This topic is covered in the first 6 chapters of CosmicPython/Architecture Patterns With Python.
However, in those chapters it uses SQLAlchemy with mappers.
The book does have a section with an example for other ORMs that use an ActiveRecord style - like mongoengine - in
Appendix D: Repository and Unit of Work Patterns with Django.
First the models are defined.
Please note the following example may be hard to follow without any background and so I recommend reading the first 6 chapters of CosmicPython if the example below is unclear.
src/djangoproject/alloc/models.py
from django.db import models
from allocation.domain import model as domain_model
class Batch(models.Model):
reference = models.CharField(max_length=255)
sku = models.CharField(max_length=255)
qty = models.IntegerField()
eta = models.DateField(blank=True, null=True)
#staticmethod
def update_from_domain(batch: domain_model.Batch):
try:
b = Batch.objects.get(reference=batch.reference)
except Batch.DoesNotExist:
b = Batch(reference=batch.reference)
b.sku = batch.sku
b.qty = batch._purchased_quantity
b.eta = batch.eta
b.save()
b.allocation_set.set(
Allocation.from_domain(l, b)
for l in batch._allocations
)
def to_domain(self) -> domain_model.Batch:
b = domain_model.Batch(
ref=self.reference, sku=self.sku, qty=self.qty, eta=self.eta
)
b._allocations = set(
a.line.to_domain()
for a in self.allocation_set.all()
)
return b
class OrderLine(models.Model):
orderid = models.CharField(max_length=255)
sku = models.CharField(max_length=255)
qty = models.IntegerField()
def to_domain(self):
return domain_model.OrderLine(
orderid=self.orderid, sku=self.sku, qty=self.qty
)
#staticmethod
def from_domain(line):
l, _ = OrderLine.objects.get_or_create(
orderid=line.orderid, sku=line.sku, qty=line.qty
)
return l
class Allocation(models.Model):
batch = models.ForeignKey(Batch, on_delete=models.CASCADE)
line = models.ForeignKey(OrderLine, on_delete=models.CASCADE)
#staticmethod
def from_domain(domain_line, django_batch):
a, _ = Allocation.objects.get_or_create(
line=OrderLine.from_domain(domain_line),
batch=django_batch,
)
return a
Then a port and adapter are defined for the repository pattern in
src/allocation/adapters/repository.py
# pylint: disable=no-member, no-self-use
from typing import Set
import abc
from allocation.domain import model
from djangoproject.alloc import models as django_models
class AbstractRepository(abc.ABC):
def __init__(self):
self.seen = set() # type: Set[model.Batch]
def add(self, batch: model.Batch):
self.seen.add(batch)
def get(self, reference) -> model.Batch:
p = self._get(reference)
if p:
self.seen.add(p)
return p
#abc.abstractmethod
def _get(self, reference):
raise NotImplementedError
class DjangoRepository(AbstractRepository):
def add(self, batch):
super().add(batch)
self.update(batch)
def update(self, batch):
django_models.Batch.update_from_domain(batch)
def _get(self, reference):
return (
django_models.Batch.objects.filter(reference=reference)
.first()
.to_domain()
)
def list(self):
return [b.to_domain() for b in django_models.Batch.objects.all()]
Along with the domain models
src/allocation/domain/model.py
from __future__ import annotations
from dataclasses import dataclass
from datetime import date
from typing import Optional, List, Set
class OutOfStock(Exception):
pass
def allocate(line: OrderLine, batches: List[Batch]) -> str:
try:
batch = next(b for b in sorted(batches) if b.can_allocate(line))
batch.allocate(line)
return batch.reference
except StopIteration:
raise OutOfStock(f"Out of stock for sku {line.sku}")
#dataclass(unsafe_hash=True)
class OrderLine:
orderid: str
sku: str
qty: int
class Batch:
def __init__(self, ref: str, sku: str, qty: int, eta: Optional[date]):
self.reference = ref
self.sku = sku
self.eta = eta
self._purchased_quantity = qty
self._allocations = set() # type: Set[OrderLine]
def __repr__(self):
return f"<Batch {self.reference}>"
def __eq__(self, other):
if not isinstance(other, Batch):
return False
return other.reference == self.reference
def __hash__(self):
return hash(self.reference)
def __gt__(self, other):
if self.eta is None:
return False
if other.eta is None:
return True
return self.eta > other.eta
def allocate(self, line: OrderLine):
if self.can_allocate(line):
self._allocations.add(line)
def deallocate(self, line: OrderLine):
if line in self._allocations:
self._allocations.remove(line)
#property
def allocated_quantity(self) -> int:
return sum(line.qty for line in self._allocations)
#property
def available_quantity(self) -> int:
return self._purchased_quantity - self.allocated_quantity
def can_allocate(self, line: OrderLine) -> bool:
return self.sku == line.sku and self.available_quantity >= line.qty
I am attempting to create a "relationship" between two indexed Documents using elasticsearch-dsl. When using the Object(EsPerson) as a field of EsComment. When I update EsPerson, the field in EsComment does not update.
I have tried using InnerDoc, but that is not indexed and also does not update
class EsPersonAttr(InnerDoc):
id = Long(required=True)
name = Text(fields={'keyword': Keyword()}, required=True)
def __repr__(self):
return '<EsPersonAttr: {}>'.format(
self.name,
)
class EsPersonIndex(Document):
"""
Elastic Search Person model.
"""
class Index:
name = 'es-person'
class meta:
doc_type = 'es-person'
id = Long(required=True)
name = Text(fields={'keyword': Keyword()}, required=True)
def save(self, **kwargs):
return super(EsPersonIndex, self).save(**kwargs)
def __repr__(self):
return '<EsPersonIndex: {}>'.format(
self.name,
)
class EsPerson(object):
def __init__(self, id, name):
self._id = id
self._name = name
self.index_doc = EsPersonIndex(
id=id,
name=name
)
self.attr_doc = EsPersonAttr(
id=id,
name=name
)
def __repr__(self):
return '<EsPerson: {}>'.format(
self._name,
)
#property
def id(self):
return self._id
#id.setter
# Set both Document & InnerDoc at the same time
def id(self, value):
self._id = value
# self.index_doc.id = value
self.index_doc.update()
self.attr_doc.id = value
#property
def name(self):
return self._id
#name.setter
# Set both Document & InnerDoc at the same time
def name(self, value):
self._name = value
self.index_doc.name = value
self.index_doc.save()
self.attr_doc.name = value
class EsComment(Document):
"""
Elastic Search Comment model.
"""
id = Long(required=True)
title = Text(fields={'keyword': Keyword()}, required=True)
text = Text(fields={'keyword': Keyword()})
author = Object(EsPersonAttr, required=True)
class Index:
name = 'es-comment'
class meta:
doc_type = 'es-comment'
def save(self, **kwargs):
# if there is no date, use now
return super(EsComment, self).save(**kwargs)
def __repr__(self):
return '<EsComment: {}>'.format(
self.title,
)
I expected that when I updated the name field for a EsPerson, it updates author.name in EsComment
You might want to have a look at the example using parent/child and nested which ar the two methods of joining objects in Elasticsearch: https://github.com/elastic/elasticsearch-dsl-py/blob/master/examples/parent_child.py
Here is my model.py
class Candidate(models.Model):
person = models.OneToOneField(
Person, related_name='person_candidate', on_delete=models.PROTECT)
def __str__(self):
return str(self.person)
#property
def total_candidate_votes(self):
return self.candidate_votes.filter(candidate=self).count()
#property
def amount_paid(self):
return self.candidate_payments.aggregate(models.Sum('fees'))['fees__sum'] or 0
#property
def is_qualified_to_vie(self):
return self.amount_paid >= 10000
Help me create a filter that will show candidates who have only paid >=10000
Filter.py
class CandidateFilter(django_filters.FilterSet):
name = django_filters.CharFilter(lookup_expr='iexact', name='person__first_name')
is_qualified_to_vie = django_filters.BooleanFilter(method='filter_by_qualified_candidates')
def filter_by_qualified_candidates(self, queryset, field, value):
return queryset.filter
The problem is that python properties can't be translated into django filter expressions. I'd recommend using custom queryset methods in addition to the python properties here. Something like the below:
class CandidateQuerySet(models.QuerySet):
def annotate_amount_paid(self):
return self.annotate(amount_paid=models.Sum('candidate_payments__fees'))
def qualified_to_vie(self, yes=True):
# eg, Candidate.objects.qualified_to_vie()
qs = return self.annotate_amount_paid()
if yes:
return qs.filter(amount_paid__gte=10000)
return qs.filter(amount_paid__lt=10000)
class Candidate(models.Model):
...
objects = CandidateQuerySet.as_manager()
From here, it's fairly straightforward.
class CandidateFilter(filters.FilterSet):
is_qualified_to_vie = django_filters.BooleanFilter(method='filter_by_qualified_candidates')
def filter_by_qualified_candidates(self, queryset, name, value):
return queryset.qualified_to_vie(value)
Note that the above is just the gist of the idea and will probably require some changes in order to actually function.
I'm stuck with a SqlAlchemy problem.
I just want to delete an relation. This relation is made by an association object.
models
class User(db.Model, UserMixin):
id = db.Column(db.Integer, primary_key=True)
email = db.Column(db.String(255), unique=True)
username = db.Column(db.String(255), unique=True)
password = db.Column(db.String(255))
following = db.relationship('Follower', foreign_keys='Follower.user_id')
followed_by = db.relationship('Follower', foreign_keys='Follower.follow_user_id')
def __repr__(self):
return '<%s (%i)>' % (self.username, self.id)
class Follower(db.Model):
__tablename__ = 'followers'
user_id = db.Column(db.Integer, db.ForeignKey('user.id'), primary_key=True)
follow_user_id = db.Column(db.Integer, db.ForeignKey('user.id'), primary_key=True)
created_at = db.Column(db.DateTime, default=datetime.datetime.now)
user_followed = db.relationship("User", primaryjoin=(follow_user_id==User.id))
user = db.relationship("User", primaryjoin=(user_id==User.id))
def __repr__(self):
return '<%i %i>' % (self.user_id, self.follow_user_id)
How I add a relation (it works !):
u1 = # user 1
u2 = # user 2
...
f = Follower()
f.user_followed = u2
u1.following.append(f)
db.session.commit()
How I try do delete a relation (it doesn't work):
f = Follower()
f.user_followed = u2
u1.following.remove(f)
db.session.commit()
The error
ValueError: list.remove(x): x not in list
I understand why it doesn't work, it's because this Follower() instance is not in the list u1.following. So, how can I delete this relation?
You can override __eq__, __ne__, and __hash__ so that instances that are not the same instance, but have the same values, compare and hash equal.
I use the following mixin for this purpose. Just override compare_value in the subclass to return whatever should actually be compared.
from sqlalchemy import inspect
class EqMixin(object):
def compare_value(self):
"""Return a value or tuple of values to use for comparisons.
Return instance's primary key by default, which requires that it is persistent in the database.
Override this in subclasses to get other behavior.
"""
return inspect(self).identity
def __eq__(self, other):
if not isinstance(other, self.__class__):
return NotImplemented
return self.compare_value() == other.compare_value()
def __ne__(self, other):
eq = self.__eq__(other)
if eq is NotImplemented:
return eq
return not eq
def __hash__(self):
return hash(self.__class__) ^ hash(self.compare_value())
One could also try querying for the object first and then delete it from the list.
follower_to_be_deleted = db.session.query(Follower).filter_by(user_id=u2.id).first()
u1.following.remove(follower_to_be_deleted)
db.session.commit()