I'm running into an issue using mongoengine. A raw query that works on Compass isn't working using _ _ raw _ _ on mongoengine. I'd like to rewrite it using mongoengine's methods, but I'd like to understand why it's not working using _ _ raw_ _ either.
I'm running an embedded document list field that has inheritence. The query is "give me all sequences that are have a 'type A' Assignment "
My schema:
class Sequence(Document):
seq = StringField(required = True)
samples = EmbeddedDocumentListField(Sample)
assignments = EmbeddedDocumentListField(Assignment)
class Sample(EmbeddedDocument):
name = StringField()
class Assignment(EmbeddedDocument):
name = StringField()
meta = {'allow_inheritance': True}
class TypeA(Assignment):
pass
class TypeB(Assignment):
other_field = StringField()
pass
Writing {'assignments._cls': 'TypeA'} into Compass returns a list. But on mongoengine I get an empty field:
from mongo_objects import Sequence
def get_samples_assigned_as_class(cls : str):
query_raw = Sequence.objects(__raw__={'assignments._cls': cls}) # raw query, fails
#query2 = Sequence.objects(assignments___cls = cls) # Fist attempt, failed
#query3 = Sequence.objects.get().assignments.filter(cls = cls) # Second attempt, also failed. Didn't like that it queried everything first
print(query_raw) # empty list, iterating does nothing.
get_samples_assigned_as_class('TypeA')
"Assignments" is a list because one sequence may have multiples of the same class. An in depth awnser on how to query these lists for categorical information would be ideal, as I'm not sure how to properly go about it. I'm mostly filtering on the inheritence _cls, but eventually I'd like to do nested queries (cls : TypeA, sample : Sample_1)
Thanks
I'm on peewee 3.14.0 ... is it a valid use case to do something like this? This is ofc a toy example of my actual use-case:
class A(BaseModel):
id = BigAutoField(primary_key=True)
a1 = CharField()
class B(BaseModel):
a = ForeignKeyField(A, backref='bs')
b1 = CharField()
and then do something like:
a = A.select()...
for b in a.bs:
b.b1 = "doesn't work, this will insert a new record rather than update the existing"
b.save()
to cover this update use-case I need to do the following instead:
a = A.select()...
for b in a.bs:
q = (B.update({b1: "this however, works"}).where(B.a==a, ..))
q.execute()
Is there a way to fix the first approach or is it not covered by peewee?
It does not insert new records in this example. I think your example is either missing some important information or something else is missing.
class User(Base):
username = TextField()
class Tweet(Base):
user = ForeignKeyField(User, backref='tweets')
content = TextField()
db.create_tables([User, Tweet])
u = User.create(username='u1')
for j in range(2):
Tweet.create(user=u, content='t%s' % j)
u = User.get(User.username == 'u1')
for tweet in u.tweets:
tweet.content = tweet.content + '-x'
tweet.save()
for tweet in u.tweets:
print(tweet.content)
This correctly prints:
t0-x
t1-x
I am implementing search in my project what I want is to concat to column in where clause to get results from table.
Here is what I am doing:
from django.db.models import Q
if 'search[value]' in request.POST and len(request.POST['search[value]']) >= 3:
search_value = request.POST['search[value]'].strip()
q.extend([
Q(id__icontains=request.POST['search[value]']) |
(Q(created_by__first_name=request.POST['search[value]']) & Q(created_for=None)) |
Q(created_for__first_name=request.POST['search[value]']) |
(Q(created_by__last_name=request.POST['search[value]']) & Q(created_for=None)) |
Q(created_for__last_name=request.POST['search[value]']) |
(Q(created_by__email__icontains=search_value) & Q(created_for=None)) |
Q(created_for__email__icontains=search_value) |
Q(ticket_category=request.POST['search[value]']) |
Q(status__icontains=request.POST['search[value]']) |
Q(issue_type__icontains=request.POST['search[value]']) |
Q(title__icontains=request.POST['search[value]']) |
Q(assigned_to__first_name__icontains=request.POST['search[value]']) |
])
Now I want to add another OR condition like:
CONCAT(' ', created_by__first_name, created_by__last_name) like '%'search_value'%'
But when I add this condition to the queryset it becomes AND
where = ["CONCAT_WS(' ', profiles_userprofile.first_name, profiles_userprofile.last_name) like '"+request.POST['search[value]']+"' "]
tickets = Ticket.objects.get_active(u, page_type).filter(*q).extra(where=where).exclude(*exq).order_by(*order_dash)[cur:cur_length]
How do I convert this into an OR condition?
Advanced filters can be solved by
Q() object and
Query expressions like Func(), Value() and F().
The only used trick is a
Custom Lookup "rhs_only" that uses the right-hand-side of the lookup and ignores the left side, because it is easier to use all concatenated fields directly on the right side. A memorable function concat_like encapsulates that all to be easily used in queries.
from django.db.models import F, Func, Lookup, Q, Value
from django.db.models.fields import Field
def concat_like(columns, pattern):
"""Lookup filter: CONCAT_WS(' ', column_0, column_1...) LIKE pattern"""
lhs = '%s__rhs_only' % columns[0]
expr = Func(*(F(x) for x in columns), template="CONCAT_WS(' ', %(expressions)s)")
return Q(**{lhs: Like(expr, Value(pattern))})
class Like(Func):
def as_sql(self, compiler, connection):
arg_sql, arg_params = zip(*[compiler.compile(x) for x in self.source_expressions])
return ("%s LIKE '%s'" % tuple(arg_sql)), arg_params[0] + arg_params[1]
#Field.register_lookup
class RhsOnly(Lookup):
"""Skip the LHS and evaluate the boolean RHS only"""
lookup_name = 'rhs_only'
def as_sql(self, compiler, connection):
return self.process_rhs(compiler, connection)
All boolean expression and related objects are supported by this code. All arguments are correctly escaped.
Example usage:
>>> qs = MyModel.objects.filter(Q(id=1) | concat_like(('first_name', 'surname'), 'searched'))
>>> str(qs.query) # sql output simplified here
"SELECT .. WHERE id=1 OR (CONCAT_WS(' ', first_name, surname) LIKE 'searched')"
Relevant documentation:
https://docs.djangoproject.com/en/1.11/ref/models/expressions/#func-expressions
See also... Value() and F()
https://docs.djangoproject.com/en/1.11/topics/db/aggregation/#aggregations-and-other-queryset-clauses
You can reference annotated fields from inside the filter method. As such, you can filter against two concatenated fields and add it as another OR condition like this:
from django.db.models import F, Func, Value
# Because we added user_full_name as an annotation below,
# we can refer to it in the filters
q.extend([
Q(id__icontains=request.POST['search[value]']) |
(Q(created_by__first_name=request.POST['search[value]']) & Q(created_for=None)) |
Q(created_for__first_name=request.POST['search[value]']) |
(Q(created_by__last_name=request.POST['search[value]']) & Q(created_for=None)) |
Q(created_for__last_name=request.POST['search[value]']) |
(Q(created_by__email__icontains=search_value) & Q(created_for=None)) |
Q(created_for__email__icontains=search_value) |
Q(ticket_category=request.POST['search[value]']) |
Q(status__icontains=request.POST['search[value]']) |
Q(issue_type__icontains=request.POST['search[value]']) |
Q(title__icontains=request.POST['search[value]']) |
Q(assigned_to__first_name__icontains=request.POST['search[value]']) |
Q(user_full_name__icontains=request.POST['search[value]']) # <------
])
# Add the annotation to your queryset
# I'm not actually sure what the related_name or field_name for your user
# profiles are, so I'm pretending that tickets have a profile foreignkey field
# to where the first_name and last_name fields are
user_full_name_expr = Func(Value(' '), F('profile__first_name'), F('profile__last_name'), function='CONCAT_WS')
# The next two lines can be combined as long as the annotation comes first.
tickets = Ticket.objects.annotate(user_full_name=user_full_name_expr)
tickets = tickets.get_active(u, page_type).filter(*q).exclude(*exq).order_by(*order_dash)[cur:cur_length]
For fun, here's a working example based on the User model.
from django.contrib.auth.models import User
from django.db.models import F, Func, Value
User.objects.create(username='john', first_name='John', last_name='Jingleheimer-Schmidt')
User.objects.create(username='mike', first_name='Michael', last_name='Finnigan')
foo = User.objects.annotate(full_name=Func(Value(' '), F('first_name'), F('last_name'), function='CONCAT_WS'))
print(foo.filter(full_name__icontains='john'))
# outputs: [<User: john>]
What you need is, for create search FullText. I recommend use (http://haystacksearch.org/)
See documentation of Django (https://docs.djangoproject.com/en/1.11/ref/contrib/postgres/search/)
In Django, can I re-use an existing Q object on multiple models, without writing the same filters twice?
I was thinking about something along the lines of the pseudo-Django code below, but did not find anything relevant in the documentation :
class Author(Model):
name = TextField()
company_name = TextField()
class Book(Model):
author = ForeignKey(Author)
# Create a Q object for the Author model
q_author = Q(company_name="Books & co.")
# Use it to retrieve Book objects
qs = Book.objects.filter(author__matches=q_author)
If that is not possible, can I extend an existing Q object to work on a related field? Pseudo-example :
# q_book == Q(author__company_name="Books & co.")
q_book = q_author.extend("author")
# Use it to retrieve Book objects
qs = Book.objects.filter(q_book)
The only thing I've found that comes close is using a subquery, which is a bit unwieldy :
qs = Book.objects.filter(author__in=Author.objects.filter(q_author))
From what I can tell by your comment, it just looks like you're trying to pass a set of common arguments to multiple filters, to do that you can just unpack a dictionary
The values in the dictionary can still be q objects if required as if it were a value you would pass in to the filter argument normally
args = { 'author__company_name': "Books & co" }
qs = Book.objects.filter(**args)
args['author_name'] = 'Foo'
qs = Book.objects.filter(**args)
To share this between different models, you'd have to do some dictionary mangling
author_args = { k.lstrip('author__'): v for k, v in args.items }
You can do this
books = Book.objects.filter(author__company_name="Books & co")
I have a model called Book with a Title Field.
class Book(models.Model):
title = models.CharField(null = False,blank=False)
Now I need a filter to get all books with titles starting with a number.
This works but how will I extend the queryset?
b = Book.objects.filter(title__startswith = 1).order_by('title')
This doesn't work
b = b + Book.objects.filter(title__startswith = 2).order_by('title')
or this
b.extend(Book.objects.filter(title__startswith = 2).order_by('title'))
You can use a regular expression as a query parameter to check that the a value starts with a number:
Book.objects.filter(title__regex=r'^\d')