We Keep Coding

How to scroll through elastic query results, python

I'm querying my elastic search server and limiting it to 100 results, but there could be a potential of 5000+ results, but for speed I don't want to overload the users connection trying to send it all in bulk.
data = es.search(index=case_to_view, size=100,body={
"query": {
"range" : {
"someRandomFIeld" : {
"gte" : 1,
}
}
}
})
This is doing two things, getting me results that have the field type and only getting the results where that field type exists if its value is greater than equal to 1.
data['hits']['total'] # 5089
How do I let the user get the next lot of results from the same query, ie. The next 100, previous 100, etc

You'll want to utilize the "from" and "size" properties.
You can see it here in the 7.0 documentation.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html
ex :
{
"from" : 0, "size" : 10,
"query" : {
"term" : { "user" : "kimchy" }
}
}

How to use find() nested documents for two levels or more?

Here is my sample mongodb database
database image for one object
The above is a database with an array of articles. I fetched only one object for simplicity purposes.
database image for multiple objects ( max 20 as it's the size limit )
I have about 18k such entries.
I have to extract the description and title tags present inside the (articles and 0) subsections.
The find() method is the question here.. i have tried this :
for i in db.ncollec.find({'status':"ok"}, { 'articles.0.title' : 1 , 'articles.0.description' : 1}):
for j in i:
save.write(j)
After executing the code, the file save has this :
_id
articles
_id
articles
and it goes on and on..
Any help on how to print what i stated above?
My entire code for reference :
import json
import newsapi
from newsapi import NewsApiClient
import pymongo
from pymongo import MongoClient
client = MongoClient()
db = client.dbasenews
ncollec = db.ncollec
newsapi = NewsApiClient(api_key='**********')
source = open('TextsExtractedTemp.txt', 'r')
destination = open('NewsExtracteddict.txt', "w")
for word in source:
if word == '\n':
continue
all_articles = newsapi.get_everything(q=word, language='en', page_size=1)
print(all_articles)
json.dump(all_articles, destination)
destination.write("\n")
try:
ncollec.insert(all_articles)
except:
pass

Okay, so I checked a little to update my rusty memory of pymongo, and here is what I found.
The correct query should be :
db.ncollec.find({ 'status':"ok",
'articles.title' : { '$exists' : 'True' },
'articles.description' : { '$exists' : 'True' } })
Now, if you do this :
query = { 'status' : "ok",
'articles.title' : { '$exists' : 'True' },
'articles.description' : { '$exists' : 'True' } }
for item in db.ncollect.find(query):
print item
And that it doesn't show anything, the query is correct, but you don't have the right database, or the right tree, or whatever.
But I assure you, that with the database you showed me, that if you do...
query = { 'status' : "ok",
'articles.title' : { '$exists' : 'True' },
'articles.description' : { '$exists' : 'True' } }
for item in db.ncollect.find(query):
save.write(item[0]['title'])
save.write(item[0]['description'])
It'll do what you wished to do in the first place.
Now, the key item[0] might not be good, but for this, I can't really be of any help since it is was you are showing on the screen. :)
Okay, now. I have found something for you that is a bit more complicated, but is cool :)
But I'm not sure if it'll work for you. I suspect you're giving us a wrong tree, since when you do .find( {'status' : 'ok'} ), it doesn't return anything, and it should return all the documents with a 'status' : 'ok', and since you have lots...
Anyways, here is the query, that you should use with .aggregate() method, instead of .find() :
elem = { '$match' : { 'status' : 'ok', 'articles.title' : { '$exists' : 'True'}, 'articles.description' : { '$exists' : 'True'}} }
[ elem, { '$unwind' : '$articles' }, elem ]
If you want an explanation as to how this works, I invite you to read this page.
This query will return ONLY the elements in your array that have a title, and a description, with a status OK. If an element doesn't have a title, or a description, it will be ignored.

pymongo date field converted to unknown date format

I am getting data from one collection using python and I will be processing it and storing it in another collection. In the processed collection some of the date fields looks different like Date(-61833715200000).
I use below code to get data and processing it and then I bulk insert the values to new collection.
fleet_managers = taximongo.users.aggregate([{ "$match": { "role" : "fleet_manager"}}])
fleet_managers = pd.DataFrame(list(fleet_managers))
fleet_managers['city_id'] = fleet_managers['region_id'].map({'57ff2e84f39e0f0444000004':'Chennai','57ff2e08f39e0f0444000003':'Hyderabad'})
pros_fleet_managers.insert_many(fleet_managers.to_dict('records'))
The collection looks like this:
{
"_id" : ObjectId("58006678ee5e0e29c5000009"),
"deleted_at" : NaN,
"region_id" : "57ff2e84f39e0f0444000004",
"reset_password_sent_at" : Date(-61833715200000),
"current_sign_in_at" : ISODate("2016-10-14T06:07:55.568Z"),
"last_sign_in_at" : ISODate("2016-10-14T06:07:45.574Z"),
"remember_created_at" : Date(-61833715200000)
}
What did do wrong here. Thanks already.
I have found the solution by using the $ifNull while projecting the fields.
fleet_managers = taximongo.users.aggregate([{ "$match": { "role" : "fleet_manager"}},{"$project":{'_id':1,'deleted_at':{ "$ifNull": [ "$deleted_at", "null" ] },
'reset_password_sent_at':{ "$ifNull": [ "$reset_password_sent_at", "null" ] }, 'region_id':1,'current_sign_in_at':1,'last_sign_in_at':1,'remember_created_at':{ "$ifNull": [ "$remember_created_at", "null" ] }}}])
fleet_managers = pd.DataFrame(list(fleet_managers))
fleet_managers['city_id'] = fleet_managers['region_id'].map({'57ff2e84f39e0f0444000004':'Chennai','57ff2e08f39e0f0444000003':'Hyderabad'})
pros_fleet_managers.insert_many(fleet_managers.to_dict('records'))
The above code gives the solution but I need to handle the null or non existence dynamically i.e., when fetching it from the source collection.
Help me out on this.

Database error from mongoengine max_distance query

Hi i have a problem with queries data using mongoengine. I try to do this same like in documentation but i have a problem.
I create two models
class Alarm(mongoengine.Document):
added = mongoengine.DateTimeField()
title = mongoengine.StringField()
tracks = mongoengine.ListField(mongoengine.EmbeddedDocumentField(Track))
meta = {
'indexes': [[("tracks.location", "2dsphere")]]
}
class Track(mongoengine.EmbeddedDocument):
created_on = mongoengine.DateTimeField()
location = mongoengine.PointField()
from django shell i add one row :
db.alarm.insert({"title": "Warszawa", "tracks": [{"location": {"type": "Point", "coordinates": [21.01666, 52.233333]}}]})
I connect to mongodb and from shell i try to find my new location using $near:
>db.alarm.find({'tracks.location': {$near: {$geometry: {"type": "Point", coordinates: [18.068611, 59.329444]}, $maxDistance: 810997}}})
>
> db.alarm.find({'tracks.location': {$near: {$geometry: {"type": "Point", coordinates: [18.068611, 59.329444]}, $maxDistance: 810998}}})
{
"_id" : ObjectId("53ef89626dda06655a57a342"), "title" : "Warszawa",
"tracks" : [ { "location" : { "type" : "Point", "coordinates" : [
21.01666, 52.233333 ] } } ] }
Returned result is that what i expected. First query return none second find my location.
But i cannot receive this same result using mongoengine
Alarm.objects(tracks__location__near = {"coordinates":[ 21.01666, 52.233333 ] , "type": "Point"}, tracks__location__max_distance=810998)
i get:
<repr(<mongoengine.queryset.queryset.QuerySet at 0x7ff3a50e8a10>) failed: pymongo.errors.OperationFailure: database error: Can't canonicalize query: BadValue geo near accepts just one argument when querying for a GeoJSON point. Extra field found: $maxDistance: 810998>

This is more like a hint, but:
$near operator in MongoDB has two subclauses: $geometry and $maxDistance (that's how you call it in the Mongo shell). But it looks like your custom object tries to instantiate $near clause with only $geometry as parameter.
So it's not tracks__location__max_distance, but rather something like tracks__location__near__max_distance (i.e. max_distance should be inside $near clause).

How to $set sub-sub-array value in MongoDB by pymongo

Data:
{
"_id" : ObjectId("50cda9741d41c81da6000002"),
"template_name" : "common_MH",
"role" : "MH",
"options" : [
{
"sections" : [
{
"tpl_option_name" : "test321",
"tpl_option_type" : "string",
"tpl_default_value" : "test321"
}
],
"tpl_section_name" : "Test"
}
]
}
could I modify tpl_default_value in options.$.section.$.tpl_option_name = 'test321'?
I already try too times, but I can't solve.
please assist me, thanks.

This is a bad schema for doing these kinda of updates, there is a JIRA for multi-level positional operator however it is not yet done: https://jira.mongodb.org/browse/SERVER-831
Ideally you either have to update this client side and then atomically set that section of the array:
$section = {
"tpl_option_name" : "test321",
"tpl_option_type" : "string",
"tpl_default_value" : "test321"
};
db.col.update({}, {$set: {options.$.sections.1: $section}})
Or you need to change your schema. Does the sections really need to be embedded? I noticed that you have a tpl_section_name in the top level but then you are nesting sections within that, it sounds more logical that only one section should be there.
That document would be easier to update.