We Keep Coding

Cannot deserialize properly a response using pymongo

I was using an API that were written in NodeJS, but for some reasons I had to re-write the code in python. The thing is that the database is in MongoDB, and that the response to all the queries (with large results) includes a de-serialized version with$id
as, for example {$oid: "f54h5b4jrhnf}.
this object id representation with the nested $iod
instead of just the plain string that Node use to return, is messing with the front end, and I haven't been able to find a way to get just the string, rather than this nested object (other than iterate in every single document and extract the id string) without also changing the way the front end treat the response
is there a solution to get a json response of the shape [{"_id":"63693f438cdbc3adb5286508", etc...}
?
I tried using pymongo and mongoengine, both seems unable to de-serialize in a simple way

You have several options (more than mentioned below).
MongoDB
In a MongoDB query, you could project/convert all ObjectIds to a string using "$toString".
Python
Iterate, like you mention in your question.
--OR--
You could also define/use a custom pymongo TypeRegistry class with a custom TypeDecoder and use it with a collection's CodecOptions so that every ObjectId read from a collection is automatically decoded as a string.
Here's how I did it with a toy database/collection.
from bson.objectid import ObjectId
from bson.codec_options import TypeDecoder
class myObjectIdDecoder(TypeDecoder):
bson_type = ObjectId
def transform_bson(self, value):
return str(value)
from bson.codec_options import TypeRegistry
type_registry = TypeRegistry([myObjectIdDecoder()])
from bson.codec_options import CodecOptions
codec_options = CodecOptions(type_registry=type_registry)
collection = db.get_collection('geojson', codec_options=codec_options)
# the geojson collection _id field values have type ObjectId
# but because of the custom CodecOptions/TypeRegistry/TypeDecoder
# all ObjectId's are decoded as strings for python
collection.find_one()["_id"]
# returns '62ae621406926107b33b523c' I.e., just a string.

JSON issue with MongoDB ObjectId

As you know MongoDB documents has at least one ObjectId named _id. It's not possible to convert a document contains an ObjectId to JSON. currently I have two solutions to convert this document to JSON:
del doc['_id']
or create a new document with a string instance of that field.
What it just works when I know which field contains ObjectId. What to do if I have multiple ObjectId and I don't know what are they?

MongoDB returns a BSON (not a JSON) document, so actually you want to convert a BSON document into JSON document.
Try to take a look into this artickle: https://technobeans.com/2012/09/10/mongodb-convert-bson-to-json/

PyMongo find_one() returns nothing when passed _id as query parameter

I have a single document in my Mongo database:
{"_id" : ObjectId("569bbe3a65193cde93ce7092"),
"categories" : [{_id: 0, "category": "Groceries"},
{_id: 1, "category": "Bills"}, . . .]}
Using PyMongo in my project, I get this result calling find_one():
x = db.collection.find_one({"_id": "ObjectId(\"569bbe3a65193cde93ce7092\")"})
print(x)
// None
Whenever I perform this same query in the Mongo shell, it returns my document. I cannot for the life of me figure out why this is not working. Using find({}) returns the document, so I know PyMongo can see it.
I can call find_one({"categories": {"$exists": True}}) to retrieve the document, and since this will be the only document containing "categories", this will work; however, now I'm just baffled as to why accessing the document by _id is giving me such trouble. Neither escaping the quotes nor quote-wrapping 569bbe3a65193cde93ce7092 has made any difference.

To add to the #Simulant answer, you need to import the ObjectId from the bson.objectid:
from bson.objectid import ObjectId
x = db.collection.find_one({"_id": ObjectId("569bbe3a65193cde93ce7092")})

pass it without the quotes on the content of _id you also need to import ObjectId.
from bson.objectid import ObjectId
{"_id": ObjectId("569bbe3a65193cde93ce7092")}
If you pass it with quotes you are searching for an Object with the String ObjectId("569bbe3a65193cde93ce7092") as ID. But in MongoDB the ID is an Object and not a String. Thats a difference.

how can I get a valid object id of mongodb from a string?

I want to make my mongodb OjectId to a field:
class ZhinengStats(Document):
zhineng_id = ObjectIdField(db_field="_id", primary_key=True, required=True, help_text="job id")
but a valid ObjectId must be a 12-byte input of type 'str' or a 24-character hex string, so if I do this:
ZhinengStats.objects(zhineng_id="programmer").first()
I will get an error. I thought a hash is a good way:
hash("programmer") # 7354308922443094682
but as you can see, hash seems not ok.
So how can I get a valid object id?
UPDATE:
the main problem is because this is field is a chinese, so something like this:ObjectId('兼职'.decode("utf-8")) can not work, so how can I make this support utf-8?

Is there any reason why you must generate the ObjectId from a string? If you simply wants an unique id for your mongodb document, you can generate one automatically like so
from bson.objectid import ObjectId
_id = ObjectId()
the ObjectId generated will be based on the machine's hardware signature, and the current time

Python mongodb documentation has examples ready for you:
class bson.objectid.ObjectId(oid=None)
Initialize a new ObjectId.
ObjectId(b'foo-bar-quux')
ObjectId('666f6f2d6261722d71757578')

Assuming the string is in variable rndId, do the following:
from bson.objectid import ObjectId
rndId="1234"
collection.find_one({"_id": ObjectId(rndId)
*** Pre-install the bson package dependency :
> pip install bson

How to search document by oid in mongoengine

I need get documents from db by oid, like:
Docs.objects(_id='4f4381f4e779897a2c000009')
But how to do it, if _id requires ObjectId object and even I try to set ObjectId from pymongo it doesn't work.
Docs.objects(_id=pymongo.objectid.ObjectId('4f4381f4e779897a2c000009'))
return empty list

This should work:
Docs.objects(pk='4f4381f4e779897a2c000009')

How about just using the raw string:
Docs.objects.get(id='4f4381f4e779897a2c000009')
That is probably the easiest way ... right ?

Came to this question because I had a lot of trouble with this myself. It seems like PyMongo changed this and objectid is no longer inside pymongo and is now instead:
import bson
Doc.objects.get(id=bson.objectid.ObjectId('4f4381f4e779897a2c000009'))
Also, Mongoengine uses the name 'id' for the ObjectID field.

This thread is old, but in case someone looks at it around 2022:
This works fine with MongoDB Atlas + Mongoengine == 0.23.1
from bson.objectid import ObjectId
Doc.objects(_id=ObjectId("85a2c854002c893dd7756b5g"))