We Keep Coding

How to iterate over a mongo document with an embedded doc using pymongo?

I have MongoDB data that looks like this:
{
"_id" : ObjectId("602ad096f7449063397d41bd"),
"name" : "1234/1236 Main St",
"city" : "Indianapolis",
"state" : "IN",
"zip" : "46208",
"total_units" : 2,
"units" : [
{
"unit_num" : 1,
"street" : "1234 Main",
"bedrooms" : 3,
"bath" : 1,
"sqft" : 1225,
"monthly_rent" : 800,
"lease_expiration" : "2021-06-30"
},
{
"unit_num" : 2,
"street" : "1236 Main",
"bedrooms" : 3,
"bath" : 1,
"sqft" : 1225,
"monthly_rent" : 800,
"lease_expiration" : "2021-07-31"
}
]
}]}
Using python and PyMongo, I'm trying to iterate over all the entries, and not all the entries have multiple units, and return the monthly_rent and lease_expiration value. This is what I have in the shell:
db.property.find({active: true}, {_id: 0, name: 1, "street": 1,"units.monthly_rent": 1, "units.lease_expiration": 1}).sort({"units.lease_expiration": 1})
and it returns this:
{
"name" : "1234/1236 Main St",
"units" : [
{
"street" : "1234 Main St",
"monthly_rent" : 800,
"lease_expiration" : "2021-06-30"
},
{
"street" : "1236 Main St",
"monthly_rent" : 800,
"lease_expiration" : "2021-07-31"
}
]
}
Within the python script, I want to iterate over each property and unit and print out name,"units.street","units.monthly_rent", "units.lease_expiration", but I can't seem to get it to traverse the units array. I'm testing this:
for prop in list(db.mycol.find({})):
print(prop)
print(prop["units.monthly_rent"])
The print(prop) prints out all the data as expected, but the other print statement gives an error: KeyError: 'units.monthly_rent'
Can someone point me in the right direction?
Adding aggregation query from comments below:
db.property.aggregate([
{ $project: {
"units": {
$filter: { input: "$units", as: "u",
cond: { $gte: [ "$$u.unit_num", 0 ] }
} } }
},
{ $unwind: "$units" },
{ $project: {
"street": "$units.street", "
monthly_rent": "$units.monthly_rent",
"lease_expiration": "$units.lease_expiration" }
}
])

You've to loop ver the units and print unit.monthly_rent
for prop in list(db.mycol.find({})):
print(prop)
for unit in list(prop['units']):
print(unit['monthly_rent'])
You can't do array['monthly_rent'] instead you've to do array[0]['monthly_rent'] and so on
You've get each element and then you get access to the object inside the array

Matching / Mapping lists with elasticsearch

There is a list in mongodb,
eg:
db_name = "Test"
collection_name = "Map"
db.Map.findOne()
{
"_id" : ObjectId(...),
"Id" : "576",
"FirstName" : "xyz",
"LastName" : "abc",
"skills" : [
"C++",
"Java",
"Python",
"MongoDB",
]
}
There is a list in elastcisearch index (I am using kibana to execute queries)
GET /user/_search
{
"took" : 31,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 7,
"max_score" : 1.0,
"hits" : [
{
"_index" : "customer",
"_type" : "_doc",
"_id" : "5",
"_score" : 1.0,
"_source" : {
"name" : "xyz abc"
"Age" : 21,
"skills" : [
"C++",
"Java",
"Python",
"MongoDB",
]
}
},
]
}
}
Can anyone help with the elasticsearch query that will match both the records based on skills.
I am using python to write the code
If a match is found, I am trying to get the first name and last name of that user
First name : "xyz"
Last name : "abc"

Assuming you are indexing all the document in elastic and of these you want to match documents where skills has both java and mongodb the query will be as:
{
"query": {
"bool": {
"filter": [
{
"term": {
"skills": "mongodb"
}
},
{
"term": {
"skills": "java"
}
}
]
}
}
}

Updating an object inside an array with PyMongo

I am wondering how do you update a nested array with PyMongo/MongoDB by selecting a document(row) and then going into the nested array and selecting a specific object.
{
"_id" : "12345",
"name" : "John Doe,
"mylist" : [
{
"nested_id" : "1",
"data1" : "lorem ipsum",
"data2" : "stackoverflow",
"data3" : "james bond"
},
{
"nested_id" : "2",
"data1" : "lorem ipsum",
"data2" : "stackoverflow",
"data3" : "james bond"
},
{
....
}
]
}
and then lets say you pass a discretionary with the elements you want to update. In this example only update data1 and data3
data = {
"data1" : "new lorem",
"data3" : "goldeneye"
}
I have tried with the following syntax, but with no success.
db.testing.find_and_modify(
query={"_id": "12345", 'mylist.nested_id' : "1"},
update={"$set": {'mylist' : data}})
what it should look like after the update
{
"_id" : "12345",
"name" : "John Doe,
"mylist" : [
{
"nested_id" : "1",
"data1" : "new lorem",
"data2" : "stackoverflow",
"data3" : "goldeneye"
},
{
"nested_id" : "2",
"data1" : "lorem ipsum",
"data2" : "stackoverflow",
"data3" : "james bond"
},
{
....
}
]
}

Use "dot notation" and the positional operator in the update portion. Also transform your input to match the "dot notation" form for the key representation:
# Transform to "dot notation" on explicit field
for key in data:
data["mylist.$." + key] = data[key]
del data[key]
# Basically makes
# {
# "mylist.$.data1": "new lorem",
# "mylist.$.data3": "goldeneye"
# }
db.testing.find_and_modify(
query = {"_id": "12345", 'mylist.nested_id' : "1"},
update = { "$set": data }
)
So that will transpose $ to the actual matched element position from the query portion of the update. The matched array element will be updated and using "dot notation" only the mentioned fields will be affected.
Have no idea what "service" is supposed to mean in this context and I am just treating it as a "transcribing error" since you are clearly trying to match an array element in position.
That could be cleaner, but this should give you the general idea.

pymongo extract objectID from find

hello i have the following mongodb collection:
> db.attributes.find().pretty()
{
"_id" : ObjectId("53a4445fd901f278f8685b91"),
"values" : [
{
"code" : "AQ",
"pmsCode" : "638c",
"name" : {
"en-UK" : "Aqua"
},
"tcxCode" : "16-4529 TCX",
"hexCode" : "#00aed8",
"images" : [
"AQ.jpg"
],
"_id" : ObjectId("53a4445fd901f278f8685b17")
},
{
"code" : "AQ",
"pmsCode" : "3115c",
"name" : {
"en-UK" : "Aqua"
},
"tcxCode" : "",
"hexCode" : "#00c4db",
"images" : [
"AQ.jpg"
],
"_id" : ObjectId("53a4445fd901f278f8685b18")
}],
"name" : {
"en-UK" : "Colour"
}
}
{
"_id" : ObjectId("53a4445fd901f278f8685bac"),
"values" : [
{
"code" : 0,
"_id" : ObjectId("53a4445fd901f278f8685b92"),
"name" : {
"en-UK" : "0-3 MTHS"
}
}, {
"code" : 0,
"_id" : ObjectId("53a4445fd901f278f8685b93"),
"name" : {
"en-UK" : "ONE SIZE"
}
}
,
"name" : {
"en-UK" : "Size"
}
}
basically a collection that has two object Colour and Size which have sub-objects called values
what is the correct way to find the ObjectId for specific Colour values code using pymongo?
I have this attribute_id = attributes.find({"values.code": product_color_code}) but how do i extract the actual ObjectID from this?
any advise much appreciated.

u can try using select _id from table_name GROUP BY _id HAVING some_condition_on_colout methods of SQL
in mongodb using python, you can do the following
ideas.aggregate([
{"$match": {'colour':'some_Color_of_ur_choice' }},
{'$group':{'_id': "$_id",'count':{"$sum": 1 } }}
])
this will help u even count no. of occurrences of colours according to the ObjectIds

MongoDB: Remove duplicate records from Projection

How can I remove duplicate records from mongoDB projection ?
Lets say I have My mongo documents in following form -
{"_id":"55555454", "From":"Bob", "To":"Alice", "subject":"Hi", "date":"04102011"}
{"_id":"55555455", "From":"Bob", "To":"Dave", "subject":"Hello", "date":"04102014"}
{"_id":"55555456", "From":"Bob", "To":"Alice", "subject":"Bye", "date":"04112013"}
When I do a simple projection
db.col.find({}, {"From":1, "To":1, "_id"=0})
which will obviously give me all three records like this.
{"From":"Bob", "To":"Alice"} {"From":"Bob","To":"Dave"} {"From":"Bob",
"To":"Alice"}
However What I want is only two records, this way -
{"From":"Bob", "To":"Alice"} {"From":"Bob","To":"Dave"}
As My application is in python currently (using pymongo), what I am doing is that I am removing duplicate in the application from the list of records using
result = [dict(tupleized) for tupleized in set(tuple(item.items()) for item in l)]
Is there any DB method which I can apply to the projection and gives me only two records.

You can't do a reduction and eliminate duplicate documents using just find with MongoDB and a projection.
The find commands won't work as you need remember that it's returning a cursor to the client and as such, can't reduce the results to only those documents that are unique without a secondary pass.
Using this as test data (removed the _id):
> db.test.find()
{ "From" : "Bob", "To" : "Alice", "subject" : "Hi", "date" : "04102011" }
{ "From" : "Bob", "To" : "Dave", "subject" : "Hello", "date" : "04102014" }
{ "From" : "Bob", "To" : "Alice", "subject" : "Bye", "date" : "04112013" }
{ "From" : "Bob", "To" : "Alice", "subject" : "Hi", "date" : "04102011" }
{ "From" : "Bob", "To" : "Dave", "subject" : "Hello", "date" : "04102014" }
{ "From" : "Bob", "To" : "Alice", "subject" : "Bye", "date" : "04112013" }
{ "From" : "Bob", "To" : "Dave", "subject" : "Hello", "date" : "04102014" }
{ "From" : "Bob", "To" : "Alice", "subject" : "Bye", "date" : "04112013" }
{ "From" : "George", "To" : "Carl", "subject" : "Bye", "date" : "04112013" }
{ "From" : "David", "To" : "Carl", "subject" : "Bye", "date" : "04112013" }
You could use aggregation:
> db.test.aggregate({ $group: { _id: { "From": "$From", "To": "$To" }}})
Results:
{
"result" : [
{
"_id" : {
"From" : "David",
"To" : "Carl"
}
},
{
"_id" : {
"From" : "George",
"To" : "Carl"
}
},
{
"_id" : {
"From" : "Bob",
"To" : "Dave"
}
},
{
"_id" : {
"From" : "Bob",
"To" : "Alice"
}
}
],
"ok" : 1
}
The Python code should look very similar to the aggregation pipeline suggested above.

Projection only defines which fields you want to appear in the result. It is much like the statement starting with:
SELECT From, To
as opposed to the basic form of
SELECT *
So what you actually wanted to do was the equivalent of this:
db.collection.find(
{ "From": "Bob", "To": "Alice" },
{ "From": 1, "To": 1 }
)
Which actually selects the records that you want and is much the same form as:
SELECT From, To
FROM collection
WHERE
From = "Bob"
AND To = "Alice"
Should that actually somehow produce "duplicate" results the you can remove this with use of aggregate:
db.collection.aggregate([
{ "$match": {
"From": "Bob", "To": "Alice"
}}
{ "$group": {
"_id": {
"From": "$From", "To": "$To"
}
}}
])