We Keep Coding

How I can find closest date by single request using pymongo? Date should be closest to Current Date

Dataset :
{"_id":{"$oid":"61e038a052124accf41cb5e4"},"event_date":{"$date":{"$numberLong":"1642204800000"}},
"name":"UFC Fight Night","event_url":"https://www.tapology.com/fightcenter/events/82805-ufc-fight-night","location":"Las Vegas, NV"}
{"_id":{"$oid":"61e038a252124accf41cb5e5"},"event_date":{"$date":{"$numberLong":"1642809600000"}},"name":"UFC 270","event_url":"https://www.tapology.com/fightcenter/events/82993-ufc-270","location":"Anaheim, CA"}
{"_id":{"$oid":"61e038a252124accf41cb5e6"},"event_date":{"$date":{"$numberLong":"1644019200000"}},"name":"UFC Fight Night","event_url":"https://www.tapology.com/fightcenter/events/83125-ufc-fight-night","location":"Las Vegas, NV"}
I'm using python, and that means I have no way to use $commands in my code for Mongo DB to find collection I need. Question is how can find object which should have datetime value closest to current date. As I understand I have to use python's datetime.now() to set current date and compare it ($currentDate doesn't work for Python). But but in order to compare values I have to deserialize the object and this looks very heavy. By default Mongo uses ISO datetime type.
Can you help me? At least direction to put me on a right way?

Here is a solution with an aggregate
db.collection.aggregate([
{
"$addFields": {
"dateDiff": {
"$abs": {
"$subtract": [
"$event_date",
{
"$literal": new Date()
}
]
}
}
}
},
{
"$sort": {
"dateDiff": 1
}
}
])
I use $subtract to have a difference between event_date and today.
$abs is to have the absolute value. To have the closest in the future or in the past.
And then we just have to sort by dateDiff
In your code you have to replace new Date()
Try it here

Pymongo query on datetime key returns mongo query operator and huge int

I am writing this query;
end = dt.datetime.today()
start = end - dt.timedelta(7)
query = collection.find({'date' : {'$gt': start, '$lt' : end}},
{'_id' : 1, 'name' : 1, 'date_created' : 1})
docs = json.loads(dumps(query))
I am querying using pymongo on a datetime object and converting from bytes to json using bson. What I get back is a document that instead of having a datetime object, it looks like this;
{'_id' : 0, 'name' : 'Brand CN Homepage (Desktop)', 'date' : {'$date' : 1590537600000}}
When I try to convert that huge integer, I get;
OverflowError: Python int too large to convert to C long
the date key, value should look like this;
{'date' : datetime.datetime(2020, 4, 3, 0, 0)}
Is there a way I can get back a datetime object and not a massive integer?

'date' : {'$date' : 1590537600000}
This is an extended json representation of a timestamp, documented here. To load extended json, use json_util rather than the standard library json module.

Python Search JSON values for Date

Ok so I'm looking for advice and suggestions on the best way to comb through json data to look for today's date/time and return the right value.
Here is a sample of what the json data looks like:
[
{
"startDateTime": "2018-04-11T14:17:00-05:00",
"endDateTime": "2018-04-11T14:18:00-05:00",
"oncallMember": [
"username1"
],
"shiftType": "historical"
},
{
"startDateTime": "2018-04-11T14:18:00-05:00",
"endDateTime": "2018-04-16T08:00:00-05:00",
"oncallMember": [
"username2"
],
"shiftType": null
},
{
"startDateTime": "2018-04-16T08:00:00-05:00",
"endDateTime": "2018-04-23T08:00:00-05:00",
"oncallMember": [
"username1"
],
"shiftType": null
},
{
"startDateTime": "2018-04-23T08:00:00-05:00",
"endDateTime": "2018-04-30T08:00:00-05:00",
"oncallMember": [
"username2"
],
"shiftType": null
},
......continues on for the year
The start/end dates are set to be weekly rotations between the members, however when exceptions are set or changed, the start/end dates could vary daily or any other amount of time. What I want to do is check for today's date and find the current "oncallMember". I'm not sure how to search between the start and end times for today's date.
Any help is appreciated.

the module arrow maybe helpful.
first,get the today's timestamp range
import arrow
today = arrow.now()
(day_start,day_end) = today.span('day')
day_start_timestamp = day_start.timestamp
day_end_timestamp = day_end.timestamp
and then you need parse detail data into timestamp,but your raw data looks like a duration of time like "2018-04-16T08:00:00-05:00",maybe you need slice part of it like "2018-04-16T08:00:00",and using arrow parse it into timestamp,like
raw = "2018-04-16T08:00:00"
FORMAT = "YYYY-MM-DDTHH:mm:SS"
obj = arrow.get(raw,FORMAT)
obj_ts = obj.timestamp
and then you need judge whether obj_ts is in range between day_start_timestamp and day_end_timestamp
but if you need your code running for days,the timestamp range need to be changed everyday

json and datetime libraries
Usejson library for reading json by json.loads and converting it into dictionary
and for str to datetime conversion use datetime and dateutil.parser.parse
import json
from dateutil.parser import parse
from datetime import datetime
dict_ = json.loads(json_str)
# json str is the json you mentioned
startDate = dict_[0]['startDateTime']
# '2018-04-11T14:17:00-05:00'
date = parse(startDate)
# datetime.datetime(2018, 4, 11, 14, 17, tzinfo=tzoffset(None, -18000))
Once you get the date-time do further coding for start end date of today's comparison and return oncallMember

How to convert and over write MongoDB date string into ISODate using Python PyMongo?

I'm trying to convert a number of document strings in the format "YYYY-MM-DD" into ISODates in MongoDB and have successfully written this for the console which has the desired effect:
db.address.find({ "date" : { $type : 2 } } ).forEach(function(element){ element.date = ISODate(element.date); db.address.save(element);})
I'm trying to do the same in Python something like this:
client = MongoClient(my_mongodb_ip)
db = client.address
result = db.address.find( { "date" : { "$type" : 2 } } );
for r in result:
print(r['date'])
r["date"] = datetime.strptime(r["date"], "%Y-%m-%d")
print(r['date'])
db.address.update_one({"company": r['company']},
{"$set": {"date" : r['date']}})
Which I'd like to deliver this:
"date": {
"$date": "2017-06-28T00:00:00.000Z"
},
I don't get any updates to the DB.

Try to use:
import dateutil.parser
dateutil.parser.parse(<your time String>)

You can achieve this goal using the arrow module in python. All You need to do is, just create a small function that can take your date as parameter and convert it into the ISO format.
This is how you can do it:
import arrow
def convert_to_ISO_Format(self, value):
date = arrow.get(value)
date = date.format("YYYY-MM-DDTHH:mm:ss")
date = date + "Z"
self.converted_date_iso = date
Or If you know the region of the state and you want to convert it accordingly then you can do like this;
def convert_to_ISO_Region(self,value):
date = arrow.get(value)
date = date.to("Asia/Singapore")
date = date.format("YYYY-MM-DD HH:mm:ss")
date = date + "Z"
self.converted_date_iso = date
OR
If you want to convert the present date and time into ISO, it's pretty simple like the below one statement.
arrow.utcnow().datetime
I hope this can be helpful to your problem.

I figured out what was going wrong here. No need to import arrow or parser, datetime works just fine:
newest_date = datetime.strptime(r["date"], "%Y-%m-%d")
This creates the new datetime object from my "date" item in my collection assuming the format YYYY-MM-DD. Once I figured that out I had also been incorrectly putting .isoformat() to the end of this creating a string again, no idea why I did that.
With the incorrect isoformat() removed I can now run:
db.address.update_one({"address": "1234", {"$set": {"date" : newest_date}})
And the address will update correctly to type 9, i.e. a date object not a string. I checked this with:
db.address.find({ "date" : { $type : 9 } } )

how to create datetime from a negative epoch in Python

First timer on StackExchange.
I am working with ArcGIS Server and Python. While trying to execute a query using the REST endpoint to a map service, I am getting the values for a field that is esriFieldTypeDate in negative epoch in the JSON response.
The JSON response looks like this:
{
"feature" :
{
"attributes" : {
"OBJECTID" : 11,
"BASIN" : "North Atlantic",
"TRACK_DATE" : -3739996800000,
}
,
"geometry" :
{
"paths" :
[
[
[-99.9999999999999, 30.0000000000001],
[-100.1, 30.5000000000001]
]
]
}
}
}
The field I am referring to is "TRACK_DATE" in the above JSON. The values returned by ArcGIS Server are always in milliseconds since epoch. ArcGIS Server also provides a HTML response and the TRACK_DATE field for the same query is displayed as "TRACK_DATE: 1851/06/27 00:00:00 UTC".
So, the date is pre 1900 and I understand the Python in-built datetime module is not able to handle dates before 1900. I am using 32-bit Python v2.6. I am trying to convert it to a datetime by using
datetime.datetime.utcfromtimestamp(float(-3739996800000)/1000)
However, this fails with
ValueError: timestamp out of range for platform localtime()/gmtime() function
How does one work with epochs that are negative and pre 1900 in Python 2.6? I have looked at similar posts, but could not find one that explains working with negative epochs.

This works for me:
datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=(-3739996800000/1000))
→ datetime.datetime(1851, 6, 27, 0, 0)
This would have been better asked on StackOverflow since it is more Python specific than it is GIS-specific.

if timestamp < 0:
return datetime(1970, 1, 1) + timedelta(seconds=timestamp)
else:
return datetime.utcfromtimestamp(timestamp)

You can accomplish this using the datetime module's datetime and timedelta functions.
The other answers divide the timestamp by 1000 to convert milliseconds to seconds. This is unnecessary, since the timedelta function can take milliseconds directly as a parameter. It might therefore be cleaner to do something like this:
datetime.datetime(1970, 1, 1) + datetime.timedelta(milliseconds=-3739996800000)
which gives datetime.datetime(1851, 6, 27, 0, 0), as you'd expect.

The fromtimestamp() and the utcfromtimestamp() methods have been updated now to handle negative timestamps.
You can directly use the fromtimestamp() method in the datetime module to convert the epoch to datetime. Don't forget to convert the milliseconds to seconds. Remember, the fromtimestamp() method will give a datetime object according to your timezone.
To calculate the datetime object in UTC time, you can use the utcfromtimestamp() method.