We Keep Coding

Cannot deserialize properly a response using pymongo

I was using an API that were written in NodeJS, but for some reasons I had to re-write the code in python. The thing is that the database is in MongoDB, and that the response to all the queries (with large results) includes a de-serialized version with$id
as, for example {$oid: "f54h5b4jrhnf}.
this object id representation with the nested $iod
instead of just the plain string that Node use to return, is messing with the front end, and I haven't been able to find a way to get just the string, rather than this nested object (other than iterate in every single document and extract the id string) without also changing the way the front end treat the response
is there a solution to get a json response of the shape [{"_id":"63693f438cdbc3adb5286508", etc...}
?
I tried using pymongo and mongoengine, both seems unable to de-serialize in a simple way

You have several options (more than mentioned below).
MongoDB
In a MongoDB query, you could project/convert all ObjectIds to a string using "$toString".
Python
Iterate, like you mention in your question.
--OR--
You could also define/use a custom pymongo TypeRegistry class with a custom TypeDecoder and use it with a collection's CodecOptions so that every ObjectId read from a collection is automatically decoded as a string.
Here's how I did it with a toy database/collection.
from bson.objectid import ObjectId
from bson.codec_options import TypeDecoder
class myObjectIdDecoder(TypeDecoder):
bson_type = ObjectId
def transform_bson(self, value):
return str(value)
from bson.codec_options import TypeRegistry
type_registry = TypeRegistry([myObjectIdDecoder()])
from bson.codec_options import CodecOptions
codec_options = CodecOptions(type_registry=type_registry)
collection = db.get_collection('geojson', codec_options=codec_options)
# the geojson collection _id field values have type ObjectId
# but because of the custom CodecOptions/TypeRegistry/TypeDecoder
# all ObjectId's are decoded as strings for python
collection.find_one()["_id"]
# returns '62ae621406926107b33b523c' I.e., just a string.

How to query a JSONField comprising a list of values in DJANGO

I am using JSONField to store the configuration parameter(user_types) as a list as follows:
["user_type1", "user_type2", "user_type3"]
How to query to filter elements of type "user_type1"? The following query is not working:
rows=ConfigUserTable.objects.filter(user_types__in=["user_type1"])
Thanks

Use the contains lookup, which is overridden on JSONField. For example, the following may work:
ConfigUserTable.objects.filter(user_types__contains="user_type1")
However, this might depend on how you are storing JSON data in the field. If you are storing the data as a dict, querying on that key will certainly work. I.e. data in this format in the field user_types:
{"types": ["user_type1", "user_type2", "user_type3"]}
could be queried like so:
ConfigUserTable.objects.filter(user_types__types__contains="user_type1")
Reference: https://docs.djangoproject.com/en/dev/topics/db/queries/#std:fieldlookup-jsonfield.contains

To find the _id of document other than during insertion

I have created several documents and inserted into Mongo DB.I am using python for the same . Is there a way where in I can get the _id of a particular record?
I know that we get the _id during insertion. But if i need to use it at a lateral interval is there a way I can get it by say using the find() command?

You can user the projection to get particular field from the document like this
db.collection.find({query},{_id:1}) this will return only _id
http://docs.mongodb.org/manual/reference/method/db.collection.find/

In python, you can use the find_one() method to get the document and access the _id property as following:
def get_id(value):
# Get the document with the record:
document = client.db.collection.find_one({'field': value}, {'_id': True})
return document['_id']

When you insert the record, you can specify the _id value. This has added benefits for when you're using ReplicaSets as well.
from pymongo.objectid import ObjectId
client.db.collection.insert({'_id': ObjectId(), 'key1': value....})
You could store those Ids in a list and use it later on if your requirement for needing the _id occurs immediately after insert.

Override return values of a column in SQLAlchemy; hybrid property or custom type?

I would need a very quick advice. I have a table field which can contain NULL, one or more strings, separated by ';'.
At the moment the column is defined in the model as usual:
aliases = Column(String(255))
I have an hybrid property that splits the strings and returns a list:
def my_aliases(self):
if self.aliases:
return [i.strip() for i in self.aliases.split(';')]
How can change the default behaviour of the model to get rid of the useless 'self.aliases' and always get the list or None of 'self.my_aliases'?
Is it possible to override the attribute?

Using mapper or the declarative API you can create a computed attribute in your class. Options would include:
An attribute computed from a query
Using a descriptor to parse/assemble your semicolon-separated list
And I'm assuming here that you don't have the option of changing the fields of your tables. But if you do, putting lists that you have to parse inside a single column is a "smell". For example, what happens when your list is too long? Better to have a separate table for that data and use a straight-forward join to get your aliases list.

Djapian - filtering results

I use Djapian to search for object by keywords, but I want to be able to filter results. It would be nice to use Django's QuerySet API for this, for example:
if query.strip():
results = Model.indexer.search(query).prefetch()
else:
results = Model.objects.all()
results = results.filter(somefield__lt=somevalue)
return results
But Djapian returns a ResultSet of Hit objects, not Model objects. I can of course filter the objects "by hand", in Python, but it's not realistic in case of filtering all objects (when query is empty) - I would have to retrieve the whole table from database.
Am I out of luck with using Djapian for this?

I went through its source and found that Djapian has a filter method that can be applied to its results. I have just tried the below code and it seems to be working.
My indexer is as follows:
class MarketIndexer( djapian.Indexer ):
fields = [ 'name', 'description', 'tags_string', 'state']
tags = [('state', 'state'),]
Here is how I filter results (never mind the first line that does stuff for wildcard usage):
objects = model.indexer.search(q_wc).flags(djapian.resultset.xapian.QueryParser.FLAG_WILDCARD).prefetch()
objects = objects.filter(state=1)
When executed, it now brings Markets that have their state equal to "1".

I dont know Djapian, but i am familiar with xapian. In Xapian you can filter the results with a MatchDecider.
The decision function of the match decider gets called on every document which matches the search criteria so it's not a good idea to do a database query for every document here, but you can of course access the values of the document.
For example at ubuntuusers.de we have a xapian database which contains blog posts, forum posts, planet entries, wiki entries and so on and each document in the xapian database has some additional access information stored as value. After the query, an AuthMatchDecider filters the potential documents and returns the filtered MSet which are then displayed to the user.
If the decision procedure is as simple as somefield < somevalue, you could also simply add the value of somefield to the values of the document (using the sortable_serialize function provided by xapian) and add (using OP_FILTER) an OP_VALUE_RANGE query to the original query.