We Keep Coding

django substitute dictionary with a faster JSON serializer

I'm using Django raw queryset to select data from database.
I will need a translation (by using ugettext) on a field before I return this json serialized data to django rest_framework as an API
However I'm having optimization issue as this I found out it takes quite a while to manually append dictionary to a list especially if I have a lot of database rows.
After some searching i found a library ujson that claims can serialize JSON faster. However I'm struggling to use this as I need this raw query to return translated name of a field (fruits)
Anyone have any idea how to replace this dictionary method with other faster method to serialize JSON data?
all_fruits = []
activate ("en")
raw_query = MyObject.objects.raw(" select id, fruits from my_table ")
for each_name in raw_query:
json_obj = dict( id = each_name.id,
fruits= ugettext(each_name.fruits)
)
all_fruits.append(json_obj)

It's better to avoid raw SQL only when you really have no other Solution, Django QuerySet provide very great and full-featured API for database query, the following solution could fit your needs:
all_fruits = []
activate ("en")
my_object_list = MyObject.objects.all()
for obj in my_object_list.values():
all_fruits.append({"id":obj.id, "fruits" : ugettext(obj.fruits)})

Django filter based on retrieved values

I have a model which contains a few parameters like the following
{ "user_name" : "john",
"enter_time" : 1442257970.509184,
"frequency" : 30
}
I need to write a filter functions , preferably using django-filter or any other pythonic-way so that, I could filter out those records where in
def test():
cur_time = time.time()
if cur_time >= enter_time + frequency:
return True
return False
Currently in my views.py I am able to filter based on names and single values.
For example,
records = UserRecord.objects.filter(token=user_token, user_name=name, status='active').distinct()
serializer = RecordSerializer(records, many=True)
I am not sure, how to filter out based on the test condition defined in the test(). One workaround was to get serializer.data and process a list of OrderedDictionaries to filter out the content. But I am looking for django way of doing it.

You can use F expressions in filters. In your case:
UserRecord.objects.annotate(time_plus_freq=F('enter_time')+F('frequency'))\
.filter(time_plus_freq__lt=cur_time)
This way filtering is done in the SQL and you do not need to do any on the Python side.
If Django has trouble defining the type of the annotated expression look at output_field kwarg in annotate.

If you really, really want to do it directly, there is a way but it will be done on application-level, not on database-level. You can simply make generator like this:
records = (record for record in records if test(record))
but this is really bad and you should avoid it.
Much better solution is to rewrite your test condition, so it will use django queryset filtering, like it was mentioned in Ivan's answer.

How to serialize and deserialize Django ORM query (not queryset)?

My use case is that I need to store queries in DB and retrieve them from time to time and evaluate. Thats needed for mailing-app where every user can subscribe to a web-site content selected by individually customized query.
Most basic solution is to store raw SQL and use it with RawQuerySet. But I wonder is there better solutions?

At first glance, it is really dangerous to hand out query building job to others, since they can do anything (even delete all your data in your database or drop entire table etc.)
Even you let them build a specific part of the query, it is still open to Sql Injection. If it is ok for all those dangers, then you may try the following.
This is and old script I used and let users set a specific part of the query. Basics are using string.Template and eval (the evil part)
Define your Model:
class SomeModel(Model):
usr = ForeingKey(User)
ct = ForeignKey(ContentType) # we will choose related DB table with this
extra_params = TextField() # store extra filtering criteria in here
Lets execute all queries belongs to a user. Say we have a User query with extra_params is_staff and 'username__iontains'
usr: somebody
ct: User
extra_params: is_staff=$stff_stat, username__icontains='$uname'
$ defines placeholders in extra_params
from string import Template
for _qry in SomeModel.objects.filter(usr='somebody'): # filter somebody's queries
cts = Template(_qry.extra_params) # take extras with Template
f_cts = cts.substitute(stff_stat=True, uname='Lennon') # sustitute placeholders with real time filtering values
# f_cts is now `is_staff=True, username__icontains='Lennon'`
qry = Template('_qry.ct.model_class().objects.filter($f_cts)') # Now, use Template again to place our extras into a django `filter` query. We also select related model in here with `_qry.ct.model_class()`
exec_qry = qry.substitute(f_cts=f_cts)
# now we have `User.objects.filter(is_staff=True, username__icontains='Lennon')
query = eval(exec_qry) # lets evaluate it!
If you have all relted imports done,then you an use Q or any other query building option in your extra_params. Also You can use other methods to form Create or Update queries.
You can read more about Template form there. But as I said. It is REALLY DANGEROUS to give a such option to other users.
Also you may need to read about Django Content Type
Update: As #GillBates mentioned, you can use a dictonary structure to create the query. In this case, you will not need Template anymore. You can use json for such data transfer (or any other if you wish). Assuming you use json to get the data from an outer source following code is a scratch that uses some variables from the upper code block.
input_data : '{"is_staff"=true, "username__icontains"="Lennon"}'
import json
_data = json.loads(input_data)
result_set = _qry.ct.model_class().objects.filter(**_data)

According to your answer,
User passes some content-specific parameters into a form, then view function, that recieves POST, constructs query
one option is to store parameters (pickle'd or json'ed, or in a model) and reconstruct query with regular django means. This is somewhat more robust solution, since it can handle some datastructure changes.

You could create a new model user_option and store the selections in this table.
From your question, it's hard to determine whether it is a better solution, but it would make your user's choices more explicit in your data structure.

How to fetch the datastore(app engine) record without instantiating a model?

Hi can anyone tell me how it can be done,i am a beginner. I tried using this:
def get_entities(keys):
rpc = datastore.GetRpcFromKwargs({})
keys, multiple = datastore.NormalizeAndTypeCheckKeys(keys)
entities = None
try:
entities = datastore.Get(keys, rpc=rpc)
except datastore_errors.EntityNotFoundError:
assert not multiple
return entities
but unable to get keys without the models use.

do you mean you want datastore.Entity objects instead of Model instances? if so, assuming keys is a list, you should be able to simplify your code to this:
return datastore.Get(keys)
otherwise, if you just want to see which keys have matching entities in the datastore, try this:
return db.GqlQuery('SELECT __key__ FROM <kind> WHERE __key__ IN :1', keys)
replace <kind> with the kind of entity you're querying for.

In Django, how does one filter a QuerySet with dynamic field lookups?

Given a class:
from django.db import models
class Person(models.Model):
name = models.CharField(max_length=20)
Is it possible, and if so how, to have a QuerySet that filters based on dynamic arguments? For example:
# Instead of:
Person.objects.filter(name__startswith='B')
# ... and:
Person.objects.filter(name__endswith='B')
# ... is there some way, given:
filter_by = '{0}__{1}'.format('name', 'startswith')
filter_value = 'B'
# ... that you can run the equivalent of this?
Person.objects.filter(filter_by=filter_value)
# ... which will throw an exception, since `filter_by` is not
# an attribute of `Person`.

Python's argument expansion may be used to solve this problem:
kwargs = {
'{0}__{1}'.format('name', 'startswith'): 'A',
'{0}__{1}'.format('name', 'endswith'): 'Z'
}
Person.objects.filter(**kwargs)
This is a very common and useful Python idiom.

A simplified example:
In a Django survey app, I wanted an HTML select list showing registered users. But because we have 5000 registered users, I needed a way to filter that list based on query criteria (such as just people who completed a certain workshop). In order for the survey element to be re-usable, I needed for the person creating the survey question to be able to attach those criteria to that question (don't want to hard-code the query into the app).
The solution I came up with isn't 100% user friendly (requires help from a tech person to create the query) but it does solve the problem. When creating the question, the editor can enter a dictionary into a custom field, e.g.:
{'is_staff':True,'last_name__startswith':'A',}
That string is stored in the database. In the view code, it comes back in as self.question.custom_query . The value of that is a string that looks like a dictionary. We turn it back into a real dictionary with eval() and then stuff it into the queryset with **kwargs:
kwargs = eval(self.question.custom_query)
user_list = User.objects.filter(**kwargs).order_by("last_name")

Additionally to extend on previous answer that made some requests for further code elements I am adding some working code that I am using
in my code with Q. Let's say that I in my request it is possible to have or not filter on fields like:
publisher_id
date_from
date_until
Those fields can appear in query but they may also be missed.
This is how I am building filters based on those fields on an aggregated query that cannot be further filtered after the initial queryset execution:
# prepare filters to apply to queryset
filters = {}
if publisher_id:
filters['publisher_id'] = publisher_id
if date_from:
filters['metric_date__gte'] = date_from
if date_until:
filters['metric_date__lte'] = date_until
filter_q = Q(**filters)
queryset = Something.objects.filter(filter_q)...
Hope this helps since I've spent quite some time to dig this up.
Edit:
As an additional benefit, you can use lists too. For previous example, if instead of publisher_id you have a list called publisher_ids, than you could use this piece of code:
if publisher_ids:
filters['publisher_id__in'] = publisher_ids

Django.db.models.Q is exactly what you want in a Django way.

This looks much more understandable to me:
kwargs = {
'name__startswith': 'A',
'name__endswith': 'Z',
***(Add more filters here)***
}
Person.objects.filter(**kwargs)

A really complex search forms usually indicates that a simpler model is trying to dig it's way out.
How, exactly, do you expect to get the values for the column name and operation?
Where do you get the values of 'name' an 'startswith'?
filter_by = '%s__%s' % ('name', 'startswith')
A "search" form? You're going to -- what? -- pick the name from a list of names? Pick the operation from a list of operations? While open-ended, most people find this confusing and hard-to-use.
How many columns have such filters? 6? 12? 18?
A few? A complex pick-list doesn't make sense. A few fields and a few if-statements make sense.
A large number? Your model doesn't sound right. It sounds like the "field" is actually a key to a row in another table, not a column.
Specific filter buttons. Wait... That's the way the Django admin works. Specific filters are turned into buttons. And the same analysis as above applies. A few filters make sense. A large number of filters usually means a kind of first normal form violation.
A lot of similar fields often means there should have been more rows and fewer fields.