We Keep Coding

Select mongo documents whose subdoc array has duplicate field?

With a schema like this
{
"doc1": {
"items": [
{
"item_id": 1
},
{
"item_id": 2
},
{
"item_id": 3
},
]
},
"doc2": {
"items": [
{
"item_id": 1
},
{
"item_id": 2
},
{
"item_id": 1
},
]
}
}
I want to query for documents that contain a duplicate item in their items array field. A duplicate means items with the same item_id field.
So the result for the example above should return doc2 only, because it has two items with the same item_id
Something like this?
qry = {
"items": {
"$size": {
"$ne": {
"items.unique_count" # obviously this doesn't exist, not sure how to do it
}
}
}
}
result = MyDocument.find(qry)

One option similar to #rickhg12hs and your suggestions is:
db.collection.aggregate([
{$match: {
$expr: {
$ne: [
{$size: "$items"},
{$size: {
$reduce: {
input: "$items",
initialValue: [],
in: {$setUnion: ["$$value", ["$$this.item_id"]]}
}
}
}
]
}
}
}
])
See how it works on the playground example

Removing a list entry in a list in pyMongo

I have a database collection that has objects like this:
{
"_id": ObjectId("something"),
"name_lower": "total",
"name": "Total",
"mounts": [
[
"mount1",
"instance1"
],
[
"mount2",
"instance1"
],
[
"mount1",
"instance2"
],
[
"mount2",
"instance2"
]
]
}
Say I want to remove every mount that has the instance instance2, How would I go about doing that? I have been searching for quite a while.

You can do something like this
[
{
$unwind: "$mounts"
},
{
$match: {
"mounts": {
$ne: "instance2"
}
}
},
{
$group: {
_id: "$_id",
name: {
$first: "$name"
},
mounts: {
$push: "$mounts"
}
}
}
]
Working Mongo playground

This answer is based on #varman answer but more pythonic and efficient.
The first stage should be a $match condition to filter out documents that don't need to be updated.
Since the mounts key consists of a nested array, we have to $unwind it, so that we can remove array elements that need to be removed.
We have to apply the $match condition again to filter out the element that has to be removed.
Finally, we have to $group the pipeline by _id key, so that the documents which got $unwind in the previous stage will be groupped into a single document.
from pymongo import MongoClient
client = MongoClient("<URI-String>")
col = client["<DB-Name"]["<Collection-Name>"]
count = 0
for cursor in col.aggregate([
{
"$match": {
"mounts": {"$ne": "instance2"}
}
},
{
"$unwind": "$mounts"
},
{
"$match": {
"mounts": {"$ne": "instance2"}
}
},
{
"$group": {
"_id": "$_id",
"newMounts": {
"$push": "$mounts"
}
}
},
]):
# print(cursor)
col.update_one({
"_id": cursor["_id"]
}, {
"$set": {
"mounts": cursor["newMounts"]
}
})
count += 1
print("\r", count, end="")
print("\n\nDone!!!")

Elasticsearch DSL query with specific Output

I have a few objects in the database:
object 1, object 2, object 3, .., object n
Now I am doing filter like this
MyDocument.search().filter("match", is_active=True).sort('id').execute()
Output:
searchDocumentobject 1, searchDocumentobject 2,searchDocumentobject 3, ....
Now I need to searchDocumentobject 2 in last from the list.
Need the output like this:
searchDocumentobject 1,searchDocumentobject 3, .... , searchDocumentobject 2
Thank you

In MyModel, add a new method, which returns 0 if you want to keep that document at last, else it'll return 1.
class MyModel(models.Model):
# Add new method here
def get_rank(self):
if self.id == 2: # your condition here
return 0 # return 0, if you want to keep it at last
return 1
Now, you can utilize this method in MyDocument. Add a new field in MyDocument, which we'll use for sorting.
class MyDocument(Document):
# Add new field here
rank = fields.IntegerField(attr='get_rank')
Now, you can query like this,
MyDocument.search().filter("match", is_active=True).sort('-rank', 'id').execute()

You can achieve this behavior in your search request using function score, the idea is that each other documents you'll have score of 1 (default) and with document 2 you give it lower score then sort by "_score", "id". Here is the DSL, try construct the query from your python API:
{
"_source": ["id"],
"query": {
"function_score": {
"query": {
"bool": {
//add your query here
"must": [
{
"terms": {
"id": [1, 2, 3, 70]
}
}
]
}
},
"functions": [
{
"filter": {
"term": {
"id": 2
}
},
"weight": 0.5
}
]
}
},
"sort": [
"_score",
{
"id": {
"order": "asc"
}
}
]
}
Also as Yifeng stated in comment section, you can re-sort the results after you query from ES.

Workaround for preserveNullAndEmptyArrays in MongoDB 2.6

I am using a Python script to query a MongoDB collection. The collection contains embedded documents with varying structures.
I am trying to simply "$unwind" an array contained in several documents. However, the array is not in ALL documents.
That means only the documents that contain the field are returned, the others are ignored. I am using PyMongo 2.6 so I am unable to use preserveNullAndEmptyArrays as mentioned in the documentation because it is new in MongoDB 3.2
Is there a workaround to this? Something along the lines of "if the field path exists, unwind".
The structure of documents and code in question is outlined in detail in this separate but related question I asked earlier.
ISSUE:
I am trying to "$unwind" the value of $hostnames.name. However, since the path doesn't exist in all documents, this results in several ignored documents.
Structure 1 Hostname stored as $hostnames.name
{
"_id" : "192.168.1.1",
"addresses" : {
"ipv4" : "192.168.1.1"
},
"hostnames" : [
{
"type" : "PTR",
"name" : "example.hostname.com"
}
]
}
Structure 2 Hostname stored as $hostname
{
"_id" : "192.168.2.1",
"addresses" : {
"ipv4" : "192.168.2.1"
},
"hostname" : "helloworld.com",
}
Script
cmp = db['computers'].aggregate([
{"$project": {
"u_hostname": {
"$ifNull": [
"$hostnames.name",
{ "$map": {
"input": {"$literal": ["A"]},
"as": "el",
"in": "$hostname"
}}
]
},
"_id": 0,
"u_ipv4": "$addresses.ipv4"
}},
{"$unwind": "$u_hostname"}
])
I am missing all documents that have an empty array for "hostnames".
Here is the structure of the documents that are still missing.
Structure 3
{
"_id" : "192.168.1.1",
"addresses" : { "ipv4" : "192.168.1.1" },
"hostnames" : [], }
}

We can still preserve all the documents where the array field is missing by playing with the $ifNull operator and use a logical $condition processing to assign a value to the newly computed field.
The condition here is $eq which returns True if the field is [None] or False when the condition expression evaluates to false.
cmp = db['computers'].aggregate(
[
{"$project":{
"u_ipv4": "$addresses.ipv4",
"u_hostname": {
"$let": {
"vars": {
"hostnameName": {
"$cond": [
{"$eq": ["$hostnames", []]},
[None],
{"$ifNull": ["$hostnames.name", [None]]}
]
},
"hostname": {"$ifNull": ["$hostname", None]}
},
"in": {
"$cond": [
{"$eq": ["$$hostnameName", [None]]},
{"$map": {
"input": {"$literal": [None]},
"as": "el",
"in": "$$hostname"
}},
"$$hostnameName"
]
}
}
}
}},
{ "$unwind": "$u_hostname" }
]
)

how to find values that are common for different fields [pymongo aggregate]

Let's say we have mongodb documents:
{'shop':'yes'}
{'shop':'ice_cream'}
{'shop':'grocery'}
{'amenity':'yes'}
{'amenity':'hotel'}
How do I write an aggregate query in pymongo which would return the values that are common for both keys? In that example it should return 'yes'.

Your aggregation pipeline would make use of the $setIntersection in the $project operator stage. This takes two or more arrays and returns an array that contains the elements that appear in every input array. Another aggregation operator that is useful is the $addToSet array operator which is used in creating the distinct list of values for each grouped field that can then be compared later on.
In mongoshell, inserting the documents
db.collection.insert([
{'shop':'yes'},
{'shop':'ice_cream'},
{'shop':'grocery'},
{'amenity':'yes'},
{'amenity':'hotel'}
])
You could try the following aggregation pipeline:
db.collection.aggregate([
{
"$group": {
"_id": null,
"shops": {
"$addToSet": "$shop"
},
"amenities": {
"$addToSet": "$amenity"
}
}
},
{
"$project": {
"_id": 0,
"commonToBoth": { "$setIntersection": [ "$shops", "$amenities" ] }
}
}
]);
Output:
/* 0 */
{
"result" : [
{
"commonToBoth" : [
"yes"
]
}
],
"ok" : 1
}
Pymongo:
>>> pipe = [
... {"$group": { "_id": None, "shops": {"$addToSet": "$shop"}, "amenities": {"$addToSet": "$amenity"}}},
... { "$project": {"_id": 0, "commonToBoth":{"$setIntersection": ["$shops", "$amenities"]}}}
... ]
>>>
>>> for doc in collection.aggregate(pipe):
... print(doc)
...
{u'commonToBoth': [u'yes']}