So I'm trying to use smart_query in web2py to find specific values in a db, but the only explanation I can find is in the web2py book and it's not very clear. The example GET request from the book is formatted like this:
def GET(search):
try:
rows = db.smart_query([db.person, db.pet], search).select()
return dict(result=rows)
except:
...
I'm confused as to what values I would put in place of db.person and db.pet. Here is what the book says on it:
The method db.smart_query takes two arguments:
a list of field or table that should be allowed in the query
a string containing the query expressed in natural language
I'm thinking the first value would be the database I'm searching, but then I don't know what the second value would be. The book makes it sound like it should be the string I'm searching for, but I think that that's what the variable search is for.
Could someone please help me understand what exactly each argument is supposed to do?
The first argument to smart_query is a list of DAL Table and/or Field objects (a Table object in the list will simply be expanded to include all of the table's fields). This list determines which fields can be included in the query.
The second argument is the query itself, which can include field names and comparison operators (and their natural language counterparts) as well as "and" and "or" to expression conjunctions and disjunctions. For an idea of what is allowed, you can examine the relevant code here.
The SQLFORM.grid advanced search widget generates queries that are ultimately parsed by smart_query, so to get a better idea of how to generate such queries, try creating a test SQLFORM.grid and play with the search widget in the UI to see the queries it generates.
Related
I have a problem with Django and would like to ask for some advice:
One of my models contains specific indicators with variable values. This is why I am using an Arrayfield for those.
For example one Indicator has 3 values, another only 2 and some only have 1 value.
Now I want to compare values from within the Arrayfield with each other.
So I know there are those fancy F Expressions and I tried to use them, but here I am stuck, because it seems like F doesn't allow an positional-lookup from inside ArrayField.
Does anyone know if i can use F for positional lookup of arrayfield? Or do I have to make a direct SQL-Call from Django or change my db somehow to make one value per field?
Here is the Model:
class Indicators(models.Model):
datetime = models.DateTimeField(default=datetime.now)
name = models.TextField(default="none")
values = ArrayField(models.FloatField(default=0))
This is what I want to achieve:
indicator_compareable = Indicators.objects.filter(name='compareable',
values__0=F('values__1')).values_list('values')
And it raises an Exception..
operator does not exist: double precision = double precision[]
LINE 1: ...areable' AND "indicators"."values"[1] = ("...
As Mikhail suggested (in first comment to my post), here is the translated SQL from my Django filter:
SELECT "indicators"."values"[1] FROM "indicators"
WHERE ("indicators"."name" = 'compareable'
AND "indicators"."values"[1] = ("indicators"."values"))
From the Exception it seems like I am not allowed to give ArrayField position to F Expression (or that Django ignores the position..). And the translated sql-query shows, that it seems like F doesn't handle ArrayField-position...
In postgres a correct query for my needs would be:
SELECT values FROM indicators WHERE name='compareable' and
values[1]=values[2];
During my search I found those links here, but they didn't help..:
Django F field iteration
https://code.djangoproject.com/ticket/24709
So actually after investing the translated postgres query it's clear that django's F Expression is (still) not offering positional lookup (and might not be implemented).
I'll leave this as the answer, maybe it helps someone out there who is stuck at the same point.
I decided to use Django's SQL-RAW for my specific needs. But in general the rest of my code uses Django ORM whenever possible.
Here there has been a Django-issue to this topic, but by now it's 1 and a half years old: https://code.djangoproject.com/ticket/24709
by the way, my django version: Django 2.1 (the mentioned django-issue was with v1.8)
Thanks again to Mikhail for his suggestions.
New to using Python NDB.
I have something like:
class User(ndb.Model):
seen_list = nbd.KeyProperty(kind=Survey, repeated=True)
class Survey(ndb.Model):
same = ndb.StringProperty(required=True)
I want to be able to query for users that have not seen certain surveys.
What I am doing now is:
users = User.query(seen_list != 'survey name').fetch()
This does not work. What would be the proper way to do this? Should I first query the Survey list to get the key of the survey with a certain name? Is the != part correct?
I could not find any examples similar to this.
Thanks.
unfortunately, if your survey is a repeated property, it won't work that way. When you query a repeated property the datastore tries EVERY entry in your list, and if one works, it'll return the item. So when you say "!= survey name 1", if you have at least ONE entry in your list that isn't "survey name 1", it'll come back as positive, even if another result IS "survey name 1".
it's uninstinctive if you come from an SQL background I know.... the only way to go around that is to go programatically and evaluate the ones your query returns. It comes from the fact that, for repeated values, Big Table "flatten" your results, which means it creates one entry for EVERY value in your repeated attribute. so as it scans, it eventually finds one "correct" line with your info, grabs the object key from there, and returns the object.
My use case is that I need to store queries in DB and retrieve them from time to time and evaluate. Thats needed for mailing-app where every user can subscribe to a web-site content selected by individually customized query.
Most basic solution is to store raw SQL and use it with RawQuerySet. But I wonder is there better solutions?
At first glance, it is really dangerous to hand out query building job to others, since they can do anything (even delete all your data in your database or drop entire table etc.)
Even you let them build a specific part of the query, it is still open to Sql Injection. If it is ok for all those dangers, then you may try the following.
This is and old script I used and let users set a specific part of the query. Basics are using string.Template and eval (the evil part)
Define your Model:
class SomeModel(Model):
usr = ForeingKey(User)
ct = ForeignKey(ContentType) # we will choose related DB table with this
extra_params = TextField() # store extra filtering criteria in here
Lets execute all queries belongs to a user. Say we have a User query with extra_params is_staff and 'username__iontains'
usr: somebody
ct: User
extra_params: is_staff=$stff_stat, username__icontains='$uname'
$ defines placeholders in extra_params
from string import Template
for _qry in SomeModel.objects.filter(usr='somebody'): # filter somebody's queries
cts = Template(_qry.extra_params) # take extras with Template
f_cts = cts.substitute(stff_stat=True, uname='Lennon') # sustitute placeholders with real time filtering values
# f_cts is now `is_staff=True, username__icontains='Lennon'`
qry = Template('_qry.ct.model_class().objects.filter($f_cts)') # Now, use Template again to place our extras into a django `filter` query. We also select related model in here with `_qry.ct.model_class()`
exec_qry = qry.substitute(f_cts=f_cts)
# now we have `User.objects.filter(is_staff=True, username__icontains='Lennon')
query = eval(exec_qry) # lets evaluate it!
If you have all relted imports done,then you an use Q or any other query building option in your extra_params. Also You can use other methods to form Create or Update queries.
You can read more about Template form there. But as I said. It is REALLY DANGEROUS to give a such option to other users.
Also you may need to read about Django Content Type
Update: As #GillBates mentioned, you can use a dictonary structure to create the query. In this case, you will not need Template anymore. You can use json for such data transfer (or any other if you wish). Assuming you use json to get the data from an outer source following code is a scratch that uses some variables from the upper code block.
input_data : '{"is_staff"=true, "username__icontains"="Lennon"}'
import json
_data = json.loads(input_data)
result_set = _qry.ct.model_class().objects.filter(**_data)
According to your answer,
User passes some content-specific parameters into a form, then view function, that recieves POST, constructs query
one option is to store parameters (pickle'd or json'ed, or in a model) and reconstruct query with regular django means. This is somewhat more robust solution, since it can handle some datastructure changes.
You could create a new model user_option and store the selections in this table.
From your question, it's hard to determine whether it is a better solution, but it would make your user's choices more explicit in your data structure.
I use Djapian to search for object by keywords, but I want to be able to filter results. It would be nice to use Django's QuerySet API for this, for example:
if query.strip():
results = Model.indexer.search(query).prefetch()
else:
results = Model.objects.all()
results = results.filter(somefield__lt=somevalue)
return results
But Djapian returns a ResultSet of Hit objects, not Model objects. I can of course filter the objects "by hand", in Python, but it's not realistic in case of filtering all objects (when query is empty) - I would have to retrieve the whole table from database.
Am I out of luck with using Djapian for this?
I went through its source and found that Djapian has a filter method that can be applied to its results. I have just tried the below code and it seems to be working.
My indexer is as follows:
class MarketIndexer( djapian.Indexer ):
fields = [ 'name', 'description', 'tags_string', 'state']
tags = [('state', 'state'),]
Here is how I filter results (never mind the first line that does stuff for wildcard usage):
objects = model.indexer.search(q_wc).flags(djapian.resultset.xapian.QueryParser.FLAG_WILDCARD).prefetch()
objects = objects.filter(state=1)
When executed, it now brings Markets that have their state equal to "1".
I dont know Djapian, but i am familiar with xapian. In Xapian you can filter the results with a MatchDecider.
The decision function of the match decider gets called on every document which matches the search criteria so it's not a good idea to do a database query for every document here, but you can of course access the values of the document.
For example at ubuntuusers.de we have a xapian database which contains blog posts, forum posts, planet entries, wiki entries and so on and each document in the xapian database has some additional access information stored as value. After the query, an AuthMatchDecider filters the potential documents and returns the filtered MSet which are then displayed to the user.
If the decision procedure is as simple as somefield < somevalue, you could also simply add the value of somefield to the values of the document (using the sortable_serialize function provided by xapian) and add (using OP_FILTER) an OP_VALUE_RANGE query to the original query.
Given a class:
from django.db import models
class Person(models.Model):
name = models.CharField(max_length=20)
Is it possible, and if so how, to have a QuerySet that filters based on dynamic arguments? For example:
# Instead of:
Person.objects.filter(name__startswith='B')
# ... and:
Person.objects.filter(name__endswith='B')
# ... is there some way, given:
filter_by = '{0}__{1}'.format('name', 'startswith')
filter_value = 'B'
# ... that you can run the equivalent of this?
Person.objects.filter(filter_by=filter_value)
# ... which will throw an exception, since `filter_by` is not
# an attribute of `Person`.
Python's argument expansion may be used to solve this problem:
kwargs = {
'{0}__{1}'.format('name', 'startswith'): 'A',
'{0}__{1}'.format('name', 'endswith'): 'Z'
}
Person.objects.filter(**kwargs)
This is a very common and useful Python idiom.
A simplified example:
In a Django survey app, I wanted an HTML select list showing registered users. But because we have 5000 registered users, I needed a way to filter that list based on query criteria (such as just people who completed a certain workshop). In order for the survey element to be re-usable, I needed for the person creating the survey question to be able to attach those criteria to that question (don't want to hard-code the query into the app).
The solution I came up with isn't 100% user friendly (requires help from a tech person to create the query) but it does solve the problem. When creating the question, the editor can enter a dictionary into a custom field, e.g.:
{'is_staff':True,'last_name__startswith':'A',}
That string is stored in the database. In the view code, it comes back in as self.question.custom_query . The value of that is a string that looks like a dictionary. We turn it back into a real dictionary with eval() and then stuff it into the queryset with **kwargs:
kwargs = eval(self.question.custom_query)
user_list = User.objects.filter(**kwargs).order_by("last_name")
Additionally to extend on previous answer that made some requests for further code elements I am adding some working code that I am using
in my code with Q. Let's say that I in my request it is possible to have or not filter on fields like:
publisher_id
date_from
date_until
Those fields can appear in query but they may also be missed.
This is how I am building filters based on those fields on an aggregated query that cannot be further filtered after the initial queryset execution:
# prepare filters to apply to queryset
filters = {}
if publisher_id:
filters['publisher_id'] = publisher_id
if date_from:
filters['metric_date__gte'] = date_from
if date_until:
filters['metric_date__lte'] = date_until
filter_q = Q(**filters)
queryset = Something.objects.filter(filter_q)...
Hope this helps since I've spent quite some time to dig this up.
Edit:
As an additional benefit, you can use lists too. For previous example, if instead of publisher_id you have a list called publisher_ids, than you could use this piece of code:
if publisher_ids:
filters['publisher_id__in'] = publisher_ids
Django.db.models.Q is exactly what you want in a Django way.
This looks much more understandable to me:
kwargs = {
'name__startswith': 'A',
'name__endswith': 'Z',
***(Add more filters here)***
}
Person.objects.filter(**kwargs)
A really complex search forms usually indicates that a simpler model is trying to dig it's way out.
How, exactly, do you expect to get the values for the column name and operation?
Where do you get the values of 'name' an 'startswith'?
filter_by = '%s__%s' % ('name', 'startswith')
A "search" form? You're going to -- what? -- pick the name from a list of names? Pick the operation from a list of operations? While open-ended, most people find this confusing and hard-to-use.
How many columns have such filters? 6? 12? 18?
A few? A complex pick-list doesn't make sense. A few fields and a few if-statements make sense.
A large number? Your model doesn't sound right. It sounds like the "field" is actually a key to a row in another table, not a column.
Specific filter buttons. Wait... That's the way the Django admin works. Specific filters are turned into buttons. And the same analysis as above applies. A few filters make sense. A large number of filters usually means a kind of first normal form violation.
A lot of similar fields often means there should have been more rows and fewer fields.