My use case is that I need to store queries in DB and retrieve them from time to time and evaluate. Thats needed for mailing-app where every user can subscribe to a web-site content selected by individually customized query.
Most basic solution is to store raw SQL and use it with RawQuerySet. But I wonder is there better solutions?
At first glance, it is really dangerous to hand out query building job to others, since they can do anything (even delete all your data in your database or drop entire table etc.)
Even you let them build a specific part of the query, it is still open to Sql Injection. If it is ok for all those dangers, then you may try the following.
This is and old script I used and let users set a specific part of the query. Basics are using string.Template and eval (the evil part)
Define your Model:
class SomeModel(Model):
usr = ForeingKey(User)
ct = ForeignKey(ContentType) # we will choose related DB table with this
extra_params = TextField() # store extra filtering criteria in here
Lets execute all queries belongs to a user. Say we have a User query with extra_params is_staff and 'username__iontains'
usr: somebody
ct: User
extra_params: is_staff=$stff_stat, username__icontains='$uname'
$ defines placeholders in extra_params
from string import Template
for _qry in SomeModel.objects.filter(usr='somebody'): # filter somebody's queries
cts = Template(_qry.extra_params) # take extras with Template
f_cts = cts.substitute(stff_stat=True, uname='Lennon') # sustitute placeholders with real time filtering values
# f_cts is now `is_staff=True, username__icontains='Lennon'`
qry = Template('_qry.ct.model_class().objects.filter($f_cts)') # Now, use Template again to place our extras into a django `filter` query. We also select related model in here with `_qry.ct.model_class()`
exec_qry = qry.substitute(f_cts=f_cts)
# now we have `User.objects.filter(is_staff=True, username__icontains='Lennon')
query = eval(exec_qry) # lets evaluate it!
If you have all relted imports done,then you an use Q or any other query building option in your extra_params. Also You can use other methods to form Create or Update queries.
You can read more about Template form there. But as I said. It is REALLY DANGEROUS to give a such option to other users.
Also you may need to read about Django Content Type
Update: As #GillBates mentioned, you can use a dictonary structure to create the query. In this case, you will not need Template anymore. You can use json for such data transfer (or any other if you wish). Assuming you use json to get the data from an outer source following code is a scratch that uses some variables from the upper code block.
input_data : '{"is_staff"=true, "username__icontains"="Lennon"}'
import json
_data = json.loads(input_data)
result_set = _qry.ct.model_class().objects.filter(**_data)
According to your answer,
User passes some content-specific parameters into a form, then view function, that recieves POST, constructs query
one option is to store parameters (pickle'd or json'ed, or in a model) and reconstruct query with regular django means. This is somewhat more robust solution, since it can handle some datastructure changes.
You could create a new model user_option and store the selections in this table.
From your question, it's hard to determine whether it is a better solution, but it would make your user's choices more explicit in your data structure.
Related
I want to implement this structural model to store my data on Mongodb with MongoEngine on flask:
skills = [{"asm":"Assembly",
"flag":False,
"date": datetime},
{"java":"Java",
"flag":False,
"date": datetime}]
So I don't know how I can declare and update this kind of structure.
For updating one object I used:
User.objects(skills=form.skills.data).update_one()
However, I don't know how to update more fields in one shot.
I tried with the code below but it doesn’t work.
now = datetime.now()
User.objects(skills=form.skills).update_one(set__skills = ({'ruby':'Ruby'}, {'flag':'true'},{'date':now}))
What kind of fields should I declare on forms.py?
For what I understood, you need a a nested document (skills) into another (who refers to User in this case). For doing something like this you don't have to update atomically a field but append values to the subdocument and the save everything.
Tryin' to follow your example, in your case should do something like this:
user = User.objects(email=current_user.email).get()
To get the BaseQuery that refers to user X through a certain query filter, in my example the email of the current logged user
user.kskills.append(SubDocumentClass(skillName="name_of_the_skill", status=True, date=datetime.now()))
For append a collection to the subdocument list. (I've appended your field)
user.save()
To save everything
Edit: Added solution below. No-one was able to suggest a native Postgres, one-hit search-and-replace so I had to parse the log in Python and then do an update.
Versions: Django==1.10.3 and Postgres 9.6
Is there a way to search within a nested Django JSONField (backed by Postgres jsonb) for a specific key/value state? Ideally this would be a native Django but I can break out into raw sql if necessary.
e.g. search for one or more occurrences of {"status":"running} within the following data:
{"subtask1": {"status":"running"},
"subtask2": {"status":"complete"}}
Background:
I'm using a JSONField to log the current status of long-running subtasks. Each subtask updates it's element of the json field selectively via a nativePostgres jsonb_set() operation on the server.
After each subtask, I want to query the log field to see whether this subtask was the last to complete. If all are complete (ie no occurences for {"status":"running"} within the nested json tree) then I'll update the main .complete field for the Django RunningTask instance.
Sample & simplified model:
class LongRunningTask(models.model):
id = models.AutoField(primary_key=True)
complete = models.BooleanField(default=False)
log = JSONField(null=True, blank=True, default=dict)
Example data for log field:
{"subtask1": {"status":"running"},
"subtask2": {"status":"complete"}}
Thanks in advance for any pointers.
Note:
I've tried Django's built-in contains operator but this won't find the values because they are nested one level down. I got excited by the prospect of the values operator but from my testing it's not implemented on the JSONField (only HStoreField).
Solution:
Couldn't find a native Postgres one-hit search and replace call, so I ended up parsing the log in Python after each sub-task completed to find out if it's the last one. I hope to find a better solution in time.
_current_log = LongRunningTask.objects.get(pk=current_task.id).log
_statuses = [True if _current_log[i]["status"] == "complete"
else False
for i in _current_log]
if all(_statuses):
LongRunningTask.objects.filter(pk=adhoc_task.id).update(complete=True)
Try this way:
import json
all_completed=True
json_dict=json.loads(json_log)
for key,value in json_dict.items():
if json_dict[key]['status']!='complete':
all_completed=False
break
This is just a hint. You can change it according to your requirements.
Try this:-
LongRunningTask.objects.filter(log__values__contains={"status":"running"})
I am trying to query my postgres database from django, the query I'm using is
s = Booking.objects.all().filter(modified_at__range=[last_run, current_time], coupon_code__in=l)
Now I am changing this object of mine in some ways in my script, and not saving it to the database. What I want to know is that, is it possible to query this object now?
say, I changed my variable as
s.modified_at = '2016-02-22'
Is it still possible to query this object as:
s.objects.all()
or something similar?
The QueryManager is Django's interface to the database (ORM). By definition this means you can only query data that has been stored in the database.
So, in short: "no". You cannot do queries on unsaved data.
Thinking about why you are even asking this, especially looking at the example using "modified_at": why do you not want to save your data?
(You might want to use auto_now=True for your "modified_at" field, btw.)
You could do something like this:
bookings = Booking.objects.all().filter(modified_at__range=[last_run, current_time], coupon_code__in=l)
for booking in bookings:
booking.modified_at = 'some value'
booking.save() # now booking object will have the updated value
I use Djapian to search for object by keywords, but I want to be able to filter results. It would be nice to use Django's QuerySet API for this, for example:
if query.strip():
results = Model.indexer.search(query).prefetch()
else:
results = Model.objects.all()
results = results.filter(somefield__lt=somevalue)
return results
But Djapian returns a ResultSet of Hit objects, not Model objects. I can of course filter the objects "by hand", in Python, but it's not realistic in case of filtering all objects (when query is empty) - I would have to retrieve the whole table from database.
Am I out of luck with using Djapian for this?
I went through its source and found that Djapian has a filter method that can be applied to its results. I have just tried the below code and it seems to be working.
My indexer is as follows:
class MarketIndexer( djapian.Indexer ):
fields = [ 'name', 'description', 'tags_string', 'state']
tags = [('state', 'state'),]
Here is how I filter results (never mind the first line that does stuff for wildcard usage):
objects = model.indexer.search(q_wc).flags(djapian.resultset.xapian.QueryParser.FLAG_WILDCARD).prefetch()
objects = objects.filter(state=1)
When executed, it now brings Markets that have their state equal to "1".
I dont know Djapian, but i am familiar with xapian. In Xapian you can filter the results with a MatchDecider.
The decision function of the match decider gets called on every document which matches the search criteria so it's not a good idea to do a database query for every document here, but you can of course access the values of the document.
For example at ubuntuusers.de we have a xapian database which contains blog posts, forum posts, planet entries, wiki entries and so on and each document in the xapian database has some additional access information stored as value. After the query, an AuthMatchDecider filters the potential documents and returns the filtered MSet which are then displayed to the user.
If the decision procedure is as simple as somefield < somevalue, you could also simply add the value of somefield to the values of the document (using the sortable_serialize function provided by xapian) and add (using OP_FILTER) an OP_VALUE_RANGE query to the original query.
Given a class:
from django.db import models
class Person(models.Model):
name = models.CharField(max_length=20)
Is it possible, and if so how, to have a QuerySet that filters based on dynamic arguments? For example:
# Instead of:
Person.objects.filter(name__startswith='B')
# ... and:
Person.objects.filter(name__endswith='B')
# ... is there some way, given:
filter_by = '{0}__{1}'.format('name', 'startswith')
filter_value = 'B'
# ... that you can run the equivalent of this?
Person.objects.filter(filter_by=filter_value)
# ... which will throw an exception, since `filter_by` is not
# an attribute of `Person`.
Python's argument expansion may be used to solve this problem:
kwargs = {
'{0}__{1}'.format('name', 'startswith'): 'A',
'{0}__{1}'.format('name', 'endswith'): 'Z'
}
Person.objects.filter(**kwargs)
This is a very common and useful Python idiom.
A simplified example:
In a Django survey app, I wanted an HTML select list showing registered users. But because we have 5000 registered users, I needed a way to filter that list based on query criteria (such as just people who completed a certain workshop). In order for the survey element to be re-usable, I needed for the person creating the survey question to be able to attach those criteria to that question (don't want to hard-code the query into the app).
The solution I came up with isn't 100% user friendly (requires help from a tech person to create the query) but it does solve the problem. When creating the question, the editor can enter a dictionary into a custom field, e.g.:
{'is_staff':True,'last_name__startswith':'A',}
That string is stored in the database. In the view code, it comes back in as self.question.custom_query . The value of that is a string that looks like a dictionary. We turn it back into a real dictionary with eval() and then stuff it into the queryset with **kwargs:
kwargs = eval(self.question.custom_query)
user_list = User.objects.filter(**kwargs).order_by("last_name")
Additionally to extend on previous answer that made some requests for further code elements I am adding some working code that I am using
in my code with Q. Let's say that I in my request it is possible to have or not filter on fields like:
publisher_id
date_from
date_until
Those fields can appear in query but they may also be missed.
This is how I am building filters based on those fields on an aggregated query that cannot be further filtered after the initial queryset execution:
# prepare filters to apply to queryset
filters = {}
if publisher_id:
filters['publisher_id'] = publisher_id
if date_from:
filters['metric_date__gte'] = date_from
if date_until:
filters['metric_date__lte'] = date_until
filter_q = Q(**filters)
queryset = Something.objects.filter(filter_q)...
Hope this helps since I've spent quite some time to dig this up.
Edit:
As an additional benefit, you can use lists too. For previous example, if instead of publisher_id you have a list called publisher_ids, than you could use this piece of code:
if publisher_ids:
filters['publisher_id__in'] = publisher_ids
Django.db.models.Q is exactly what you want in a Django way.
This looks much more understandable to me:
kwargs = {
'name__startswith': 'A',
'name__endswith': 'Z',
***(Add more filters here)***
}
Person.objects.filter(**kwargs)
A really complex search forms usually indicates that a simpler model is trying to dig it's way out.
How, exactly, do you expect to get the values for the column name and operation?
Where do you get the values of 'name' an 'startswith'?
filter_by = '%s__%s' % ('name', 'startswith')
A "search" form? You're going to -- what? -- pick the name from a list of names? Pick the operation from a list of operations? While open-ended, most people find this confusing and hard-to-use.
How many columns have such filters? 6? 12? 18?
A few? A complex pick-list doesn't make sense. A few fields and a few if-statements make sense.
A large number? Your model doesn't sound right. It sounds like the "field" is actually a key to a row in another table, not a column.
Specific filter buttons. Wait... That's the way the Django admin works. Specific filters are turned into buttons. And the same analysis as above applies. A few filters make sense. A large number of filters usually means a kind of first normal form violation.
A lot of similar fields often means there should have been more rows and fewer fields.