We Keep Coding

How to scroll through elastic query results, python

I'm querying my elastic search server and limiting it to 100 results, but there could be a potential of 5000+ results, but for speed I don't want to overload the users connection trying to send it all in bulk.
data = es.search(index=case_to_view, size=100,body={
"query": {
"range" : {
"someRandomFIeld" : {
"gte" : 1,
}
}
}
})
This is doing two things, getting me results that have the field type and only getting the results where that field type exists if its value is greater than equal to 1.
data['hits']['total'] # 5089
How do I let the user get the next lot of results from the same query, ie. The next 100, previous 100, etc

You'll want to utilize the "from" and "size" properties.
You can see it here in the 7.0 documentation.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html
ex :
{
"from" : 0, "size" : 10,
"query" : {
"term" : { "user" : "kimchy" }
}
}

Use $cond within $match in mongoDB aggregation

i've tried to use $cond within $match in one of the stages of an aggregation as shown below :
{ "$match" : {
"field1" : {
"$cond" : {
"if" : { "$eq" : [ "$id" , 1206]},
"then" : 0,
"else" : 1545001200
}
},
"field2" : value2 }}
But i got this error :
Error:
Assert: command failed: {
"ok" : 0,
"errmsg" : "unknown operator: $cond",
"code" : 2,
"codeName" : "BadValue"
} : aggregate failed
The mongodb version is 3.4.4.
Any idea about this issue?

You just have to reword the logic a little bit.
{ $match: { $expr: {
$or: [
{ $and: [
{ $eq: [ "$id", 1206 ] },
{ $eq: [ "$field1", 0 ] }
]},
{ $and: [
{ $ne: [ "$id", 1206 ] },
{ $eq: [ "$field1", 1545001200 ] }
]},
],
}}}
Logically, the two statements are equivalent:
Match the document by checking field1 == 0 if id == 1206, otherwise match the document by checking field1 == 1545001200
Match the document if either (id == 1206 and field1 == 0) or (id != 1206 and field1 == 1545001200).

For those coming across this later on down the road:
This won't work for 3.4.4. But in MongoDB 3.6 they introduced the $expr operator that lets you use $cond and other operations within a $match query.
https://docs.mongodb.com/manual/reference/operator/aggregation/match/
For an example see iamfrank's answer.
Also as mentioned in the comments you could do this later down the pipeline. But ideally you'll want to filter out results as early on in the pipeline using $match in order to improve processing times.

Unless important details were left out of the question, I think you are complicating something simple. Filtering during a $match aggregation step is the natural expected thing it should do.
For this particular example, there are only two simple scenarios to match a document. There is no need to use any other operators, just define the two different mutually exclusive queries and put them in an $or logical operator:
{'$match': {'$or': [
{'id': 1206, 'field1': 0},
{'id': {'$ne': 1206}, 'field1': 1545001200},
]}}

How to query PointField - MongoEngine

I was trying to update PointField in my flask app with upsert_one. But it always inserts new document. I know the problem is with the query which I'm passing.
Below is my model.
class Location(db.Document):
location_name = db.StringField(required=True)
geoCoords = db.PointField()
And the update query.
Location.objects(geoCoords=loc["geoCoords"]).upsert_one(location_name=loc["location_name"], geoCoords=loc["geoCoords"])
#loc["geoCoords"] = [77.6309395,12.9539974]
I also tried running get. But I'm getting the error message "Location matching query does not exist." for the below query.
loc = Location.objects(geoCoords=[77.6309395,12.9539974]).get()
I have following entries in my location collection.
> db.location.find()
{ "_id" : ObjectId("59c5019727bae70ad3259e67"), "geoCoords" : { "type" : "Point", "coordinates" : [ 77.6309395, 12.9539974 ] }, "location_name" : "Bengaluru" }
{ "_id" : ObjectId("59c5022d27bae70ad3259ea2"), "geoCoords" : { "type" : "Point", "coordinates" : [ 77.6309395, 12.9539974 ] }, "location_name" : "Bengaluru" }
>
I couldn't find any related information on querying the PointFiled.

To answer to my question. I think there is no way to get the exact points like I have mentioned in the question.
The nearest method works here is to use __near selector. This accepts the range in meters. So, you can give closest range query as per your requirement.
In my case, I gave 100 meters. Which is fine for me.
Example:
Location.objects(geoCoords__near=thelocation["geoCoords"], geoCoords__max_distance=100).upsert_one(location_name=thelocation["location_name"], geoCoords=thelocation["geoCoords"])

Try this:
Location.objects(geoCoords="...").update(location_name=loc["location_name"], geoCoords=loc["geoCoords"])

pymongo date field converted to unknown date format

I am getting data from one collection using python and I will be processing it and storing it in another collection. In the processed collection some of the date fields looks different like Date(-61833715200000).
I use below code to get data and processing it and then I bulk insert the values to new collection.
fleet_managers = taximongo.users.aggregate([{ "$match": { "role" : "fleet_manager"}}])
fleet_managers = pd.DataFrame(list(fleet_managers))
fleet_managers['city_id'] = fleet_managers['region_id'].map({'57ff2e84f39e0f0444000004':'Chennai','57ff2e08f39e0f0444000003':'Hyderabad'})
pros_fleet_managers.insert_many(fleet_managers.to_dict('records'))
The collection looks like this:
{
"_id" : ObjectId("58006678ee5e0e29c5000009"),
"deleted_at" : NaN,
"region_id" : "57ff2e84f39e0f0444000004",
"reset_password_sent_at" : Date(-61833715200000),
"current_sign_in_at" : ISODate("2016-10-14T06:07:55.568Z"),
"last_sign_in_at" : ISODate("2016-10-14T06:07:45.574Z"),
"remember_created_at" : Date(-61833715200000)
}
What did do wrong here. Thanks already.
I have found the solution by using the $ifNull while projecting the fields.
fleet_managers = taximongo.users.aggregate([{ "$match": { "role" : "fleet_manager"}},{"$project":{'_id':1,'deleted_at':{ "$ifNull": [ "$deleted_at", "null" ] },
'reset_password_sent_at':{ "$ifNull": [ "$reset_password_sent_at", "null" ] }, 'region_id':1,'current_sign_in_at':1,'last_sign_in_at':1,'remember_created_at':{ "$ifNull": [ "$remember_created_at", "null" ] }}}])
fleet_managers = pd.DataFrame(list(fleet_managers))
fleet_managers['city_id'] = fleet_managers['region_id'].map({'57ff2e84f39e0f0444000004':'Chennai','57ff2e08f39e0f0444000003':'Hyderabad'})
pros_fleet_managers.insert_many(fleet_managers.to_dict('records'))
The above code gives the solution but I need to handle the null or non existence dynamically i.e., when fetching it from the source collection.
Help me out on this.

How do I delete values from this document in MongoDB using Python

I am having a document which is structured like this
{
"_id" : ObjectId("564c0cb748f9fa2c8cdeb20f"),
"username" : "blah",
"useremail" : "blah#blahblah.com",
"groupTypeCustomer" : true,
"addedpartners" : [
"562f1a629410d3271ba74f74",
"562f1a6f9410d3271ba74f83"
],
"groupName" : "Mojito",
"groupTypeSupplier" : false,
"groupDescription" : "A group for fashion designers"
}
Now I want to delete one of the values from this 'addedpartners' array and update the document.
I want to just delete 562f1a6f9410d3271ba74f83 from the addedpartners array
This is what I had tried earlier.
db.myCollection.update({'_id':'564c0cb748f9fa2c8cdeb20f'},{'$pull':{'addedpartners':'562f1a6f9410d3271ba74f83'}})

db.myCollection.update(
{ _id: ObjectId(id) },
{ $pull: { 'addedpartners': '562f1a629410d3271ba74f74' } }
);
Try with this

db.myCollection.update({}, {$unset : {"addedpartners.1" : 1 }})
db.myCollection.update({}, {$pull : {"addedpartners" : null}})
No way to delete array directly, i think this is going to work, i haven't tried yet.