Excuse my ignorance, I am new in MongoDB. I am having tree collections, where the one is a superset of the other two whose elements are not overlapped. Each item is distinguish by a unique string id. What I want is to get the items of the superset that are not included in the other two collections. Could you please provide me some hint on how do do this efficiently?
Thanks.
EDIT:
Superset structure:
{ "_id" : 1, "str_id" : "ABC1fd3fsewer", "date": "a day" }
Subset 1 structure: { "_id" : 1, "str_id" : "ABre1fd3fsewer", "description" : "product" }
Subset 2 structure: { "_id" : 1, "str_id" : "ABC1fd3fsewfe"}
Each collection has a different structure but all have a common filed, the str_id.
EDIT Improved by #Neel suggestion
I have following format:
parent = [{'str_id':'a', 'tag1':'parent_random', 'tag2': 'parent_random', 'tag3':'parent_random'},{'str_id':'b',...},{'str_id':'c',...},{'str_id':'d',...}...]
child1 = [{'str_id':'a', 'tag2': child1_random'},{'str_id':'b', 'tag2': 'child1_random'}]
child2 = [{'str_id':'c', 'tag1':'child2_random'}]
and I want
outcome = [{'str_id':'c', 'tag1':'parent_random', 'tag2': 'parent_random', 'tag3':'parent_random'},{'str_id':'d', 'tag1':'parent_random', 'tag2': 'parent_random', 'tag3':'parent_random'}]
It sounds like you'll need an aggregate operation.
This document might help you:
Lookup in an array
You can do multiple lookups with one aggregate operation so you can check both the subset collections.
I am going to assume you are working with a REST API and that the client is sending a request for a subset of documents from the superset collection. You can send the array of documents you want to check from superset from the client then:
1 - match all the documents in superset to the array of documents you're sending
2 - unwind your superset document array
3 - lookup the subset collections on "str_id" field and set to a field, like "subset_one_results".
4 - do a match operation on both subset results that returns an empty array on, say, "subset_one_results"... this will match all superset documents that are not contained in subset1 for example.
$match({ $and : { "subset_one_results" : { $eq : [] } }, { "subset_two_results" : { $eq : [] } } })
5 - group them in a new array if you want to return them as an array to the client.
To increase the performance of your operations, you have to determine how often this request will be made. If it risks being often, be sure to create an index on the field that will be solicited if it's not an ObjectId field. I can't tell from your code if you are using a custom string field or an ObjectId, which is why I'm bringing up this point.
I don't know what you're using for making your queries (pure MongoDB query language, driver, etc.) so I am not sure how to answer with code hence delineating the steps up above.
I am using a public API at www.gpcontract.co.uk to populate a large variably nested dictionary representing a hierarchy of UK health organisations.
Some background information
The top level of the hierarchy is the four UK countries (England, Scotland, Wales and Northern Ireland), then regional organisations all the way down to individual clinics. The depth of the hierarchy is different for each of the countries and can change depending on the year. Each organisation has a name, orgcode and dictionary listing its child organisations.
Unfortunately, the full nested hierarchy is not available from the API, but calls to http://www.gpcontract.co.uk/api/children/[organisation code]/[year] will return the immediate child organisations of any other.
So that the hierarchy can be easily navigated in my app, I want to generate an offline dictionary of this full hierarchy (on a per year basis) which will be saved using pickle and bundled with the app.
Getting this means a lot of API calls, and I am having trouble converting the returned JSON into the dictionary object I require.
Here is an example of one tiny part of the hierarchy (I have only shown a single child organisation as an example).
JSON hierarchy example
{
"eng": {
"name": "England",
"orgcode": "eng",
"children": {}
},
"sco": {
"name": "Scotland",
"orgcode": "sco",
"children": {}
},
"wal": {
"name": "Wales",
"orgcode": "wal",
"children": {}
},
"nir": {
"name": "Northern Ireland",
"orgcode": "nir",
"children": {
"blcg": {
"name": "Belfast Local Commissioning Group",
"orgcode": "blcg",
"children": {
"abc": {
"name": "Random Clinic",
"orgcode": "abc",
"children": {}
}
}
}
}
}
}
Here’s the script I’m using to make the API calls and populate the dictionary:
My script
import json, pickle, urllib.request, urllib.error, urllib.parse
# Organisation hierarchy may vary between years. Set the year here.
year = 2017
# This function returns a list containing a dictionary for each child organisation with keys for name and orgcode
def get_child_orgs(orgcode, year):
orgcode = str(orgcode)
year = str(year)
# Correct 4-digit year to 2-digit
if len(year) > 2:
year = year[2:]
try:
child_data = json.loads(urllib.request.urlopen('http://www.gpcontract.co.uk/api/children/' + str(orgcode) + '/' + year).read())
output = []
if child_data != []:
for item in child_data['children']:
output.append({'name' : item['name'], 'orgcode' : str(item['orgcode']).lower(), 'children' : {}})
return output
except urllib.error.HTTPError:
print('HTTP error!')
except:
print('Other error!')
# I start with a template of the top level of the hierarchy and then populate it
hierarchy = {'eng' : {'name' : 'England', 'orgcode' : 'eng', 'children' : {}}, 'nir' : {'name' : 'Northern Ireland', 'orgcode' : 'nir', 'children' : {}}, 'sco' : {'name' : 'Scotland', 'orgcode' : 'sco', 'children' : {}}, 'wal' : {'name' : 'Wales', 'orgcode' : 'wal', 'children' : {}}}
print('Loading data...\n')
# Here I use nested for loops to make API calls and populate the dictionary down the levels of the hierarchy. The bottom level contains the most items.
for country in ('eng', 'nir', 'sco', 'wal'):
for item1 in get_child_orgs(country, year):
hierarchy[country]['children'][item1['orgcode']] = item1
for item2 in get_child_orgs(item1['orgcode'], year):
hierarchy[country]['children'][item1['orgcode']]['children'][item2['orgcode']] = item2
# Only England and Wales hierarchies go deeper than this
if country in ('eng', 'wal'):
level3 = get_child_orgs(item2['orgcode'], year)
# Check not empty array
if level3 != []:
for item3 in level3:
hierarchy[country]['children'][item1['orgcode']]['children'][item2['orgcode']]['children'][item3['orgcode']] = item3
level4 = get_child_orgs(item3['orgcode'], year)
# Check not empty array
if level4 != []:
for item4 in level4:
hierarchy[country]['children'][item1['orgcode']]['children'][item2['orgcode']]['children'][item3['orgcode']]['children'][item4['orgcode']] = item4
# Save the completed hierarchy with pickle
file_name = 'hierarchy_' + str(year) + '.dat'
with open(file_name, 'wb') as out_file:
pickle.dump(hierarchy, out_file)
print('Success!')
The problem
This seems to work most of the time, but it feels hacky and sometimes crashes when a nested for loop returns a "NoneType is not iterable error". I realise this is making a lot of API calls and takes several minutes to run, but I cannot see a way around this, as I want the completed hierarchy available offline for the user to make the data searchable quickly. I will then use the API in a slightly different way to get the actual healthcare data for the chosen organisation.
My question
Is there a cleaner and more flexible way to do this that would accommodate the variable nesting of the organisation hierarchy?
Is there a way to do this significantly more quickly?
I am relatively inexperienced with JSON so any help would be appreciated.
I think this question may be better suited over on the Code Review Stack Exchange, but as you mention that your code sometimes crashes and returns NoneType errors I'll give it the benefit of the doubt.
Looking at your description, this is what stands out to me
Each organisation has a name, orgcode and dictionary listing its child organisations. [API calls] will return the immediate child organisations of any other.
So, what this suggests to me (and how it looks in your sample data) is that all your data is exactly equivalent; the hierarchy only exists due to the nesting of the data and is not enforced by the format of any particular node.
This, consequently, means that you should be able to have a single piece of code which handles the nesting of an infinitely (or arbitrarily, if you prefer) deep tree. Obviously, you do this for the API call itself (get_child_orgs()), so just replicate that for constructing the tree.
def populate_hierarchy(organization,year):
""" Recursively Populate the Organization Hierarchy
organization should be a dict with an "orgcode" key with a string value
and "children" key with a dict value.
year should be a 2-4 character string representing a year.
"""
orgcode = organization['orgcode']
## get_child_orgs returns a list of organizations
children = get_child_orgs(orgcode,year)
## get_child_orgs returns None on Errors
if children:
for child in children:
## Add child to the current organization's children, using
## orgcode as its key
organization['children'][child['orgcode']] = child
## Recursively populate the child's sub-hierarchy
populate_hierarchy(child,year)
## Technically, the way this is written, returning organization is
## pointless because we're modifying organization in place, but I'm
## doing it anyway to explicitly denote the end of the function
return organization
for country in hierarchy.values():
populate_hierarchy(country,year)
It's worth noting (since you were checking for empty lists prior to iterating in your original code) that for x in y still functions correctly if y is an empty list, so you don't need to check.
The NoneType Error likely arises because you catch the Error in get_child_orgs and then implicitly return None. Therefore, for example level3 = get_child_orgs[etc...] results in level3 = None; this leads to if None != []: in the next line being True, and then you try to iterate over None with for item3 in None: which would raise the error. As noted in the code above, this is why I check the truthiness of children.
As for whether this can be done more quickly, you can try working with the threading/multiprocessing modules. I just don't know how profitable either of those will be for three reasons:
I haven't tried out the API, so I don't know how much time you have to gain from implementing multiple threads/processes
I have seen API's which timeout requests from IP Addresses when you query too quickly/too often (which would make the implementation pointless)
You say you're only running this process once per year, so runtime in perspective of a full year seems pretty insignificant (obviously, unless the current API calls are taking literal days to complete)
Finally, I would simply question whether pickle is the appropriate method of storing the information, or if you wouldn't just be better off using json.dump/load (for the record, the json module doesn't care if you change the extension to .dat if you're partial to that extension name).
In my MongoDB, I have a student collection with 10 records having fields name and roll. One record of this collection is:
{
"_id" : ObjectId("53d9feff55d6b4dd1171dd9e"),
"name" : "Swati",
"roll" : "80",
}
I want to retrieve the field roll only for all 10 records in the collection as we would do in traditional database by using:
SELECT roll FROM student
I went through many blogs but all are resulting in a query which must have WHERE clause in it, for example:
db.students.find({ "roll": { $gt: 70 })
The query is equivalent to:
SELECT * FROM student WHERE roll > 70
My requirement is to find a single key only without any condition. So, what is the query operation for that.
From the MongoDB docs:
A projection can explicitly include several fields. In the following operation, find() method returns all documents that match the query. In the result set, only the item and qty fields and, by default, the _id field return in the matching documents.
db.inventory.find( { type: 'food' }, { item: 1, qty: 1 } )
In this example from the folks at Mongo, the returned documents will contain only the fields of item, qty, and _id.
Thus, you should be able to issue a statement such as:
db.students.find({}, {roll:1, _id:0})
The above statement will select all documents in the students collection, and the returned document will return only the roll field (and exclude the _id).
If we don't mention _id:0 the fields returned will be roll and _id. The '_id' field is always displayed by default. So we need to explicitly mention _id:0 along with roll.
get all data from table
db.student.find({})
SELECT * FROM student
get all data from table without _id
db.student.find({}, {_id:0})
SELECT name, roll FROM student
get all data from one field with _id
db.student.find({}, {roll:1})
SELECT id, roll FROM student
get all data from one field without _id
db.student.find({}, {roll:1, _id:0})
SELECT roll FROM student
find specified data using where clause
db.student.find({roll: 80})
SELECT * FROM students WHERE roll = '80'
find a data using where clause and greater than condition
db.student.find({ "roll": { $gt: 70 }}) // $gt is greater than
SELECT * FROM student WHERE roll > '70'
find a data using where clause and greater than or equal to condition
db.student.find({ "roll": { $gte: 70 }}) // $gte is greater than or equal
SELECT * FROM student WHERE roll >= '70'
find a data using where clause and less than or equal to condition
db.student.find({ "roll": { $lte: 70 }}) // $lte is less than or equal
SELECT * FROM student WHERE roll <= '70'
find a data using where clause and less than to condition
db.student.find({ "roll": { $lt: 70 }}) // $lt is less than
SELECT * FROM student WHERE roll < '70'
I think mattingly890 has the correct answer , here is another example along with the pattern/commmand
db.collection.find( {}, {your_key:1, _id:0})
> db.mycollection.find().pretty();
{
"_id": ObjectId("54ffca63cea5644e7cda8e1a"),
"host": "google",
"ip": "1.1.192.1"
}
db.mycollection.find({},{ "_id": 0, "host": 1 }).pretty();
Here you go , 3 ways of doing , Shortest to boring :
db.student.find({}, 'roll _id'); // <--- Just multiple fields name space separated
// OR
db.student.find({}).select('roll _id'); // <--- Just multiple fields name space separated
// OR
db.student.find({}, {'roll' : 1 , '_id' : 1 ); // <---- Old lengthy boring way
To remove specific field use - operator :
db.student.find({}).select('roll -_id') // <--- Will remove id from result
While gowtham's answer is complete, it is worth noting that those commands may differ from on API to another (for those not using mongo's shell).
Please refer to documentation link for detailed info.
Nodejs, for instance, have a method called `projection that you would append to your find function in order to project.
Following the same example set, commands like the following can be used with Node:
db.student.find({}).project({roll:1})
SELECT _id, roll FROM student
Or
db.student.find({}).project({roll:1, _id: 0})
SELECT roll FROM student
and so on.
Again for nodejs users, do not forget (what you should already be familiar with if you used this API before) to use toArray in order to append your .then command.
Try the following query:
db.student.find({}, {roll: 1, _id: 0});
And if you are using console you can add pretty() for making it easy to read.
db.student.find({}, {roll: 1, _id: 0}).pretty();
Hope this helps!!
Just for educational purposes you could also do it with any of the following ways:
1.
var query = {"roll": {$gt: 70};
var cursor = db.student.find(query);
cursor.project({"roll":1, "_id":0});
2.
var query = {"roll": {$gt: 70};
var projection = {"roll":1, "_id":0};
var cursor = db.student.find(query,projection);
`
db.<collection>.find({}, {field1: <value>, field2: <value> ...})
In your example, you can do something like:
db.students.find({}, {"roll":true, "_id":false})
Projection
The projection parameter determines which fields are returned in the
matching documents. The projection parameter takes a document of the
following form:
{ field1: <value>, field2: <value> ... }
The <value> can be any of the following:
1 or true to include the field in the return documents.
0 or false to exclude the field.
NOTE
For the _id field, you do not have to explicitly specify _id: 1 to
return the _id field. The find() method always returns the _id field
unless you specify _id: 0 to suppress the field.
READ MORE
For better understanding I have written similar MySQL query.
Selecting specific fields
MongoDB : db.collection_name.find({},{name:true,email:true,phone:true});
MySQL : SELECT name,email,phone FROM table_name;
Selecting specific fields with where clause
MongoDB : db.collection_name.find({email:'you#email.com'},{name:true,email:true,phone:true});
MySQL : SELECT name,email,phone FROM table_name WHERE email = 'you#email.com';
This works for me,
db.student.find({},{"roll":1})
no condition in where clause i.e., inside first curly braces.
inside next curly braces: list of projection field names to be needed in the result and 1 indicates particular field is the part of the query result
getting name of the student
student-details = db.students.find({{ "roll": {$gt: 70} },{"name": 1, "_id": False})
getting name & roll of the student
student-details = db.students.find({{ "roll": {$gt: 70}},{"name": 1,"roll":1,"_id": False})
I just want to add to the answers that if you want to display a field that is nested in another object, you can use the following syntax
db.collection.find( {}, {{'object.key': true}})
Here key is present inside the object named object
{ "_id" : ObjectId("5d2ef0702385"), "object" : { "key" : "value" } }
var collection = db.collection('appuser');
collection.aggregate(
{ $project : { firstName : 1, lastName : 1 } },function(err, res){
res.toArray(function(err, realRes){
console.log("response roo==>",realRes);
});
});
it's working
Use the Query like this in the shell:
1. Use database_name
e.g: use database_name
2. Which returns only assets particular field information when matched , _id:0 specifies not to display ID in the result
db.collection_name.find( { "Search_Field": "value" },
{ "Field_to_display": 1,_id:0 } )
If u want to retrieve the field "roll" only for all 10 records in the collections.
Then try this.
In MongoDb :
db.students.find( { } , { " roll " : { " $roll " })
In Sql :
select roll from students
The query for MongoDB here fees is collection and description is a field.
db.getCollection('fees').find({},{description:1,_id:0})
Apart from what people have already mentioned I am just introducing indexes to the mix.
So imagine a large collection, with let's say over 1 million documents and you have to run a query like this.
The WiredTiger Internal cache will have to keep all that data in the cache if you have to run this query on it, if not that data will be fed into the WT Internal Cache either from FS Cache or Disk before the retrieval from DB is done (in batches if being called for from a driver connected to database & given that 1 million documents are not returned in 1 go, cursor comes into play)
Covered query can be an alternative. Copying the text from docs directly.
When an index covers a query, MongoDB can both match the query conditions and return the results using only the index keys; i.e. MongoDB does not need to examine documents from the collection to return the results.
When an index covers a query, the explain result has an IXSCAN stage that is not a descendant of a FETCH stage, and in the executionStats, the totalDocsExamined is 0.
Query : db.getCollection('qaa').find({roll_no : {$gte : 0}},{_id : 0, roll_no : 1})
Index : db.getCollection('qaa').createIndex({roll_no : 1})
If the index here is in WT Internal Cache then it would be a straight forward process to get the values. An index has impact on the write performance of the system thus this would make more sense if the reads are a plenty compared to the writes.
If you are using the MongoDB driver in NodeJs then the above-mentioned answers might not work for you. You will have to do something like this to get only selected properties as a response.
import { MongoClient } from "mongodb";
// Replace the uri string with your MongoDB deployment's connection string.
const uri = "<connection string uri>";
const client = new MongoClient(uri);
async function run() {
try {
await client.connect();
const database = client.db("sample_mflix");
const movies = database.collection("movies");
// Query for a movie that has the title 'The Room'
const query = { title: "The Room" };
const options = {
// sort matched documents in descending order by rating
sort: { "imdb.rating": -1 },
// Include only the `title` and `imdb` fields in the returned document
projection: { _id: 0, title: 1, imdb: 1 },
};
const movie = await movies.findOne(query, options);
/** since this method returns the matched document, not a cursor,
* print it directly
*/
console.log(movie);
} finally {
await client.close();
}
}
run().catch(console.dir);
This code is copied from the actual MongoDB doc you can check here.
https://docs.mongodb.com/drivers/node/current/usage-examples/findOne/
db.student.find({}, {"roll":1, "_id":0})
This is equivalent to -
Select roll from student
db.student.find({}, {"roll":1, "name":1, "_id":0})
This is equivalent to -
Select roll, name from student
In mongodb 3.4 we can use below logic, i am not sure about previous versions
select roll from student ==> db.student.find(!{}, {roll:1})
the above logic helps to define some columns (if they are less)
Using Studio 3T for MongoDB, if I use .find({}, { _id: 0, roll: true }) it still return an array of objects with an empty _id property.
Using JavaScript map helped me to only retrieve the desired roll property as an array of string:
var rolls = db.student
.find({ roll: { $gt: 70 } }) // query where role > 70
.map(x => x.roll); // return an array of role
Not sure this answers the question but I believe it's worth mentioning here.
There is one more way for selecting single field (and not multiple) using db.collection_name.distinct();
e.g.,db.student.distinct('roll',{});
Or, 2nd way: Using db.collection_name.find().forEach(); (multiple fields can be selected here by concatenation)
e.g., db.collection_name.find().forEach(function(c1){print(c1.roll);});
_id = "123321"; _user = await likes.find({liker_id: _id},{liked_id:"$liked_id"}); ;
let suppose you have liker_id and liked_id field in the document so by putting "$liked_id" it will return _id and liked_id only.
For Single Update :
db.collection_name.update({ field_name_1: ("value")}, { $set: { field_name_2 : "new_value" }});
For MultiUpdate :
db.collection_name.updateMany({ field_name_1: ("value")}, { $set: {field_name_2 : "new_value" }});
Make sure indexes are proper.
I have a mongo collection with multiple documents, suppose the following (assume Tom had two teachers for History in 2012 for whatever reason)
{
"name" : "Tom"
"year" : 2012
"class" : "History"
"Teacher" : "Forester"
}
{
"name" : "Tom"
"year" : 2011
"class" : "Math"
"Teacher" : "Sumpra"
}
{
"name" : "Tom",
"year" : 2012,
"class" : "History",
"Teacher" : "Reiser"
}
I want to be able to query for all the distinct classes "Tom" has ever had, even though Tom has had multiple "History" classes with multiple teachers, I just want the query to get the minimal number of documents such that Tom is in all of them, and "History" shows up one time, as opposed to having a query result that contains multiple documents with "History" repeated.
I took a look at:
http://mongoengine-odm.readthedocs.org/en/latest/guide/querying.html
and want to be able to try something like:
student_users = Students.objects(name = "Tom", class = "some way to say distinct?")
Though it does not appear to be documented. If this is not the syntactically correct way to do it, is this possible in mongoengine, or is there some way to accomplish with some other library like pymongo? Or do i have to query for all documents with Tom then do some post-processing to get to unique values? Syntax would be appreciated for any case.
First of all, it's only possible to get distinct values on some field (only one field) as explained in MongoDB documentation on Distinct.
Mongoengine's QuerySet class does support distinct() method to do the job.
So you might try something like this to get results:
Students.objects(name="Tom").distinct(field="class")
This query results in one BSON-document containing list of classes Tom attends.
Attention Note that returned value is a single document, so if it exceeds max document size (16 MB), you'll get error and in that case you have to switch to map/reduce approach to solve such kind of problems.
import pymongo
posts = pymongo.MongoClient('localhost', 27017)['db']['colection']
res = posts.find({ "geography": { "$regex": '/europe/', "$options": 'i'}}).distinct('geography')
print type(res)
res.sort()
for line in res:
print line
refer to http://docs.mongodb.org/manual/reference/method/db.collection.distinct/
distinct returns a list , will be printed on print type(res) , you can sort a list with res.sort() , after that it will print the values of the sorted list.
Also you can query posts before select distinct values .
student_users = Students.objects(name = "Tom").distinct('class')