We Keep Coding

How I can find closest date by single request using pymongo? Date should be closest to Current Date

Dataset :
{"_id":{"$oid":"61e038a052124accf41cb5e4"},"event_date":{"$date":{"$numberLong":"1642204800000"}},
"name":"UFC Fight Night","event_url":"https://www.tapology.com/fightcenter/events/82805-ufc-fight-night","location":"Las Vegas, NV"}
{"_id":{"$oid":"61e038a252124accf41cb5e5"},"event_date":{"$date":{"$numberLong":"1642809600000"}},"name":"UFC 270","event_url":"https://www.tapology.com/fightcenter/events/82993-ufc-270","location":"Anaheim, CA"}
{"_id":{"$oid":"61e038a252124accf41cb5e6"},"event_date":{"$date":{"$numberLong":"1644019200000"}},"name":"UFC Fight Night","event_url":"https://www.tapology.com/fightcenter/events/83125-ufc-fight-night","location":"Las Vegas, NV"}
I'm using python, and that means I have no way to use $commands in my code for Mongo DB to find collection I need. Question is how can find object which should have datetime value closest to current date. As I understand I have to use python's datetime.now() to set current date and compare it ($currentDate doesn't work for Python). But but in order to compare values I have to deserialize the object and this looks very heavy. By default Mongo uses ISO datetime type.
Can you help me? At least direction to put me on a right way?

Here is a solution with an aggregate
db.collection.aggregate([
{
"$addFields": {
"dateDiff": {
"$abs": {
"$subtract": [
"$event_date",
{
"$literal": new Date()
}
]
}
}
}
},
{
"$sort": {
"dateDiff": 1
}
}
])
I use $subtract to have a difference between event_date and today.
$abs is to have the absolute value. To have the closest in the future or in the past.
And then we just have to sort by dateDiff
In your code you have to replace new Date()
Try it here

Python 3.9.4 how to find text within a large string [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
I am using a scraper to get my daily lunch menu (school) for a discord bot. One problem I have is that the menu i am getting is for all week. How can i locate the specific date, then the specific meal (json printed as a string in python)
Ex:
"days":[
{
"date": "05-02"
"meal": "pasta"
}
{
"date": "05-03"
"meal": "Burger"
How does one efficiently and simply locate the "meal" for each day (without searching for just "meal" within the document) Also sorry i'm not sending the file, I don't want to dox myself :/

I will try to give you an answer based on the information you provided.
As you said you have your data as a stringified json. First start with decoding it. You can do it with the json builtin package) :
import json
json_as_str = '{"days": [{"date": "05-02", "meal": "Pasta"}, {"date": "05-03", "meal": "Burger"}, {"date": "05-04", "meal": "Pizza"}]}'
json_as_dict = json.loads(json_as_str)
Once you have your data as an iterable object (i.e a dict), get today's date for filtering (from the datetime builtin package) and get the corresponding string in the format of your data (as you provided them) :
from datetime import date
today = date.today()
today_as_formatted_str = f"{today:%m-%d}"
If you want any other day, you can get it with the timedelta object from the datetime package :
from datetime import timedelta
yesterday = today - timedelta(days=1) # Also today + timedelta(days=-1)
tomorrow = today + timedelta(days=1)
day_after_tomorrow = today + timedelta(days=2)
...
You can now filter the days list of your json object, based on the date key of each of the list's elements :
def filter_records_on_date(record):
return record["date"] == today_as_formatted_str
# Extract all the elements of the list that correspond to today's date
today_records = filter(
filter_records_on_date,
json_as_dict["days"],
)
You can achieve the same thing with :
today_records = filter(
lambda record: record["date"] == today_as_formatted_str,
json_as_dict["days"],
)
Now that you have the list of all meal records for today's date, just extract the corresponding meal :
def extract_meal_from_record(record):
return record["meal"]
today_meals = list(map(extract_meal_from_record, today_records))
Once again you can achieve the same result with :
today_meals = list(map(lambda record: record["meal"], today_records))
If you are sure there is only one record per day, you can extract the only meal with :
today_meal = today_meals[0]
This is a simple yet efficient solution based on the few information you provided on your data structure.

Python Search JSON values for Date

Ok so I'm looking for advice and suggestions on the best way to comb through json data to look for today's date/time and return the right value.
Here is a sample of what the json data looks like:
[
{
"startDateTime": "2018-04-11T14:17:00-05:00",
"endDateTime": "2018-04-11T14:18:00-05:00",
"oncallMember": [
"username1"
],
"shiftType": "historical"
},
{
"startDateTime": "2018-04-11T14:18:00-05:00",
"endDateTime": "2018-04-16T08:00:00-05:00",
"oncallMember": [
"username2"
],
"shiftType": null
},
{
"startDateTime": "2018-04-16T08:00:00-05:00",
"endDateTime": "2018-04-23T08:00:00-05:00",
"oncallMember": [
"username1"
],
"shiftType": null
},
{
"startDateTime": "2018-04-23T08:00:00-05:00",
"endDateTime": "2018-04-30T08:00:00-05:00",
"oncallMember": [
"username2"
],
"shiftType": null
},
......continues on for the year
The start/end dates are set to be weekly rotations between the members, however when exceptions are set or changed, the start/end dates could vary daily or any other amount of time. What I want to do is check for today's date and find the current "oncallMember". I'm not sure how to search between the start and end times for today's date.
Any help is appreciated.

the module arrow maybe helpful.
first,get the today's timestamp range
import arrow
today = arrow.now()
(day_start,day_end) = today.span('day')
day_start_timestamp = day_start.timestamp
day_end_timestamp = day_end.timestamp
and then you need parse detail data into timestamp,but your raw data looks like a duration of time like "2018-04-16T08:00:00-05:00",maybe you need slice part of it like "2018-04-16T08:00:00",and using arrow parse it into timestamp,like
raw = "2018-04-16T08:00:00"
FORMAT = "YYYY-MM-DDTHH:mm:SS"
obj = arrow.get(raw,FORMAT)
obj_ts = obj.timestamp
and then you need judge whether obj_ts is in range between day_start_timestamp and day_end_timestamp
but if you need your code running for days,the timestamp range need to be changed everyday

json and datetime libraries
Usejson library for reading json by json.loads and converting it into dictionary
and for str to datetime conversion use datetime and dateutil.parser.parse
import json
from dateutil.parser import parse
from datetime import datetime
dict_ = json.loads(json_str)
# json str is the json you mentioned
startDate = dict_[0]['startDateTime']
# '2018-04-11T14:17:00-05:00'
date = parse(startDate)
# datetime.datetime(2018, 4, 11, 14, 17, tzinfo=tzoffset(None, -18000))
Once you get the date-time do further coding for start end date of today's comparison and return oncallMember

How to convert and over write MongoDB date string into ISODate using Python PyMongo?

I'm trying to convert a number of document strings in the format "YYYY-MM-DD" into ISODates in MongoDB and have successfully written this for the console which has the desired effect:
db.address.find({ "date" : { $type : 2 } } ).forEach(function(element){ element.date = ISODate(element.date); db.address.save(element);})
I'm trying to do the same in Python something like this:
client = MongoClient(my_mongodb_ip)
db = client.address
result = db.address.find( { "date" : { "$type" : 2 } } );
for r in result:
print(r['date'])
r["date"] = datetime.strptime(r["date"], "%Y-%m-%d")
print(r['date'])
db.address.update_one({"company": r['company']},
{"$set": {"date" : r['date']}})
Which I'd like to deliver this:
"date": {
"$date": "2017-06-28T00:00:00.000Z"
},
I don't get any updates to the DB.

Try to use:
import dateutil.parser
dateutil.parser.parse(<your time String>)

You can achieve this goal using the arrow module in python. All You need to do is, just create a small function that can take your date as parameter and convert it into the ISO format.
This is how you can do it:
import arrow
def convert_to_ISO_Format(self, value):
date = arrow.get(value)
date = date.format("YYYY-MM-DDTHH:mm:ss")
date = date + "Z"
self.converted_date_iso = date
Or If you know the region of the state and you want to convert it accordingly then you can do like this;
def convert_to_ISO_Region(self,value):
date = arrow.get(value)
date = date.to("Asia/Singapore")
date = date.format("YYYY-MM-DD HH:mm:ss")
date = date + "Z"
self.converted_date_iso = date
OR
If you want to convert the present date and time into ISO, it's pretty simple like the below one statement.
arrow.utcnow().datetime
I hope this can be helpful to your problem.

I figured out what was going wrong here. No need to import arrow or parser, datetime works just fine:
newest_date = datetime.strptime(r["date"], "%Y-%m-%d")
This creates the new datetime object from my "date" item in my collection assuming the format YYYY-MM-DD. Once I figured that out I had also been incorrectly putting .isoformat() to the end of this creating a string again, no idea why I did that.
With the incorrect isoformat() removed I can now run:
db.address.update_one({"address": "1234", {"$set": {"date" : newest_date}})
And the address will update correctly to type 9, i.e. a date object not a string. I checked this with:
db.address.find({ "date" : { $type : 9 } } )

How can I count the number of times an item appears in a JSON object? Django

I developed the following code to extract data from a web API and I am trying to manipulate it based on the following condition: Count the number of times that users appear between dates initial date which is current time -
6 days and final date is current time.
The JSON object that I receive has the following structure:
[{
id: 1003384,
user_id : 0001
CreatedOn: "2017-02-16 15:54:48",
Problem: "AVAILABILILTY",
VIP: "YES",
Vendor_CODE: "XYZ12345",
Week_id: "07",
},
{
id: 1003338,
user_id: 0002
CreatedOn: "2017-02-15 13:49:16",
Problem: "AVAILABILILTY",
VIP: "NO",
Vendor_CODE: "XYZ67890",
Week_id: "09",
},
{
id: 1553338,
user_id: 0002
CreatedOn: "2017-03-15 09:30:36",
Problem: "AVAILABILILTY",
VIP: "YES",
Vendor_CODE: "ACE67890",
Week_id: "13",
}]
Now, when I execute the following code I cannot count the number of times the users appears between given two dates because I get the error keys should be integers not strings and I do not know where should I implement the initial and final dates.
from datetime import datetime, timedelta
from rest_framework.response import Response
import json, urllib.request
from collections import Counter
#Request a response to the Web API
def get_data(request, *args, **kwargs):
# YYYY-MM-DD
start_date = datetime.now() - timedelta(days=7)
end_date = datetime.now() - timedelta(days=1)
with urllib.request.urlopen("http://10.61.202.98:8081/T/ansdb/api/rows/triage/ect/tickets",timeout=15) as url:
response_data = json.loads(url.read().decode())
#verify that you receive data in your terminal
print(response_data[0])
# The JSON object should be manipulated as a dictionary but this is not the case
user_times = Counter(k['user_id'] for k in response_data if k.get('user_id'))
return JsonResponse(response_data, safe=False)
In order to avoid the error keys should be integers not string I use print(response_data[0]) to get the output:
{
id: 1003384,
user_id : 0001
CreatedOn: "2017-02-16 15:54:48",
Problem: "AVAILABILILTY",
VIP: "YES",
Vendor_CODE: "XYZ12345",
Week_id: "07",
}
My questions are:
Is my JSON data a dictionary once I parsed it with json.loads(url.read().decode())? If so, why I cannot retrieve data as response_data['user_id'] to see all the users?
What is missing to the code to count the number of times that user appears in order to be working?
Where should I implement the dates code to retrieve the users by given dates?
Thanks in advance, please feel free to suggest any ideas to this post.
Updates
June 06, 2017
The JSON data that I parsed is a dictionary and I can validate it because I used print(isinstance(response_data[0],dict)) and the result is True.
I tried to print the keys from this dictionary with print(response_data.keys()) but I encountered this error: list object has no attribute 'keys'. How is this possible if Python states that this is a dictionary but I cannot print the keys?

First: You cannot do response_data['user_id'] because response_data is a list. So, you have to do response_data[0]['user_id'].
Second: Your code should be inside a list comprehension:
Counter([k['user_id'] for k in response_data if k.get('user_id')])
Third: You should implement it in your list comprehension which will look something like this:
Counter([k['user_id'] for k in response_data if k.get('user_id') and start_date < dateutil.parser.parse(k.get('CreatedOn')) < end_date])
You should install dateutil for the third to work. Or use any other way to convert string to datetime.
Hope it helps!