Django aggregation by the name of day - python

I've tried make 'day of week aggregation', I have this code:
IN:
MyMode.objects.values('day').annotate(Sum('visits'))
OUT:
[{'visits__sum': 44, 'day': datetime.datetime(2015, 4, 5, 0, 0)},
{'visits__sum': 699, 'day': datetime.datetime(2015, 9, 6, 0, 0)},
{'visits__sum': 3, 'day': datetime.datetime(2015, 9, 3, 0, 0)},
{'visits__sum': 12, 'day': datetime.datetime(2011, 4, 5, 0, 0)}]
But I want to aggregate by the name of day, not the number.
I need totally visits in Monday, Tuesday, etc. Monday from 2015.08 should be in the same 'bag' where Monday from 2015.06 or 2012.02.

For Django 1.8+:
from django.db.models import Func
class DayOfWeek(Func):
""" Sunday == 0, ..., Saturday == 6 """
template = 'extract(dow from %(expressions)s)'
>>> (
MyMode.objects.annotate(dow=DayOfWeek(F('day')))
.values('dow')
.annotate(c=Count('dow'))
)
[{'c': 185, 'dow': 0.0}, {'c': 178, 'dow': 5.0}]
For Django 1.7-, I believe you need to do a raw query.

I don't think this is the solution you are expecting for, but maybe can help you:
for Sundays:
MyMode.objects.values('day').annotate(Sum('visits')).filter(day__week_day=1)
Change the filter value for the other weekend days. Days starts in Sunday=1, Monday=2, etc.

Related

Iterable Datetime - How to get continuos datetime objects with day name in Python?

I want to create a mapping of the everyday of the week with its datetime object. So my dictionary should be having keys as "Monday", "Tuesday", .. (so on) so that I can get a datetime object for every day on the next(!) week.
At the moment I have a dictionary with these values:
DAYS_DATETIME_RELATIONS = {
"today": datetime.datetime.now(),
"tomorrow": datetime.datetime.now() + datetime.timedelta(days=1),
"after_tomorrow": datetime.datetime.now() + datetime.timedelta(days=2)
}
Unfortunately I cannot find any algorithmic solution for this and hope anyone of you could help me.
This can be achieved by using 2 dictionaries in the following manner:
import calendar
import datetime
days = {i: calendar.day_name[i-1] for i in range(7)}
today = datetime.datetime.now()
# using i % 7 so an arbitrary range can be used, for example
# range(7, 15) to get the week after the next week
next_week = {days[i % 7]: (today + datetime.timedelta(days=i)).date()
for i in range(7)}
print(next_week)
# {'Tuesday': datetime.date(2018, 1, 9), 'Sunday': datetime.date(2018, 1, 7),
# 'Monday': datetime.date(2018, 1, 8), 'Thursday': datetime.date(2018, 1, 11),
# 'Wednesday': datetime.date(2018, 1, 10), 'Friday': datetime.date(2018, 1, 12),
# 'Saturday': datetime.date(2018, 1, 13)}
print(next_week['Saturday'])
# 2018-01-13
Here is another way to solve your question using datetime and timedelta from datetime module:
from datetime import datetime, timedelta
def generate_dict_relation(_time, _days=0):
keys = {'Yesterday': -1, 'Today': 0, 'Tomorow': 1, 'After_tomorrow': 2}
if not _days:
return {key: _time + timedelta(days=keys.get(key, 0)) for key in keys}
else:
return {(_time + timedelta(days=_days+k)).strftime('%A'): _time + timedelta(days=_days+k)
for k in range(0, 7)}
_date_now = datetime.now()
DAYS_DATETIME_RELATIONS = {}
# get dates: yesterday, today, tomorrow and after tomorrow
DAYS_DATETIME_RELATIONS.update(generate_dict_relation(_date_now, 0))
# get dates after 7 days = 1 week
DAYS_DATETIME_RELATIONS.update(generate_dict_relation(_date_now, 7))
next_tuesday = DAYS_DATETIME_RELATIONS.get('Tuesday')
next_monday = DAYS_DATETIME_RELATIONS.get('Monday')
yesterday = DAYS_DATETIME_RELATIONS.get('Yesterday')
print('{0:%d/%m/%Y %H:%M:%S:%s} \t {1}'.format(next_tuesday, repr(next_tuesday)))
print('{0:%d/%m/%Y %H:%M:%S:%s} \t {1}'.format(next_monday, repr(next_monday)))
print('{0:%d/%m/%Y %H:%M:%S:%s} \t {1}'.format(yesterday, repr(yesterday)))
Output:
16/01/2018 10:56:26:1516096586 datetime.datetime(2018, 1, 16, 10, 56, 26, 659949)
15/01/2018 10:56:26:1516010186 datetime.datetime(2018, 1, 15, 10, 56, 26, 659949)
06/01/2018 10:56:26:1515232586 datetime.datetime(2018, 1, 6, 10, 56, 26, 659949)
One very generic way will be to create a custom iterator to return you the continuos datetime objects as:
from datetime import datetime, timedelta
class RepetetiveDate(object):
def __init__(self, day_range=7, datetime_obj=datetime.now(), jump_days=1):
self.day_range = day_range
self.day_counter = 0
self.datetime_obj = datetime_obj
self.jump_days = jump_days
self.time_deltadiff = timedelta(days=self.jump_days)
def __iter__(self):
return self
# If you are on Python 2.7
# define this function as `next(self)`
def __next__(self):
if self.day_counter >= self.day_range:
raise StopIteration
if self.day_counter != 0: # don't update for the first iteration
self.datetime_obj += self.time_deltadiff
self.day_counter += 1
return self.datetime_obj
Here, this iterator returns continuos datetime object starting from the datetime object you'll initially pass (default starts from current date).
It is using 3 optional params which you may customize as per your need:
day_range: Maximum allowed iteration for the RepetetiveDate iterator. Default value is 7.
jump_days: Integer value for jumping the number of days for the datetime object in next iteration. That means, if jump_days is equal to "2", will return datetime objects of every alternate date. To get the datetime objects of past, pass this value as negative. Default value is 1.
datetime_obj: Accepts the datetime from which date you want to start your iteration. Default value is current date.
If you are new to iterators, take a look at:
What exactly are Python's iterator, iterable, and iteration protocols?
Difference between Python's Generators and Iterators
Sample Run for upcoming dates:
>>> x = RepetetiveDate()
>>> next(x)
datetime.datetime(2018, 1, 8, 15, 55, 39, 124654)
>>> next(x)
datetime.datetime(2018, 1, 9, 15, 55, 39, 124654)
>>> next(x)
datetime.datetime(2018, 1, 10, 15, 55, 39, 124654)
Sample Run for previous dates:
>>> x = RepetetiveDate(jump_days=-1)
>>> next(x)
datetime.datetime(2018, 1, 6, 15, 55, 39, 124654)
>>> next(x)
datetime.datetime(2018, 1, 5, 15, 55, 39, 124654)
>>> next(x)
datetime.datetime(2018, 1, 4, 15, 55, 39, 124654)
How to get your desired dictionary?
Using this, you may create your dictionary using the dict comprehension as:
Dictionary of all days of week
>>> {d.strftime("%A"): d for d in RepetetiveDate(day_range=7)}
{
'Monday': datetime.datetime(2018, 1, 8, 15, 23, 16, 926364),
'Tuesday': datetime.datetime(2018, 1, 9, 15, 23, 16, 926364),
'Wednesday': datetime.datetime(2018, 1, 10, 15, 23, 16, 926364),
'Thursday': datetime.datetime(2018, 1, 11, 15, 23, 16, 926364),
'Friday': datetime.datetime(2018, 1, 12, 15, 23, 16, 926364),
'Saturday': datetime.datetime(2018, 1, 13, 15, 23, 16, 926364),
'Sunday': datetime.datetime(2018, 1, 14, 15, 23, 16, 926364)
}
Here I am using d.strftime("%A") to extract day name from the datetime object.
List of current days for next 4 weeks
>>> [d for d in RepetetiveDate(jump_days=7, day_range=4))]
[
datetime.datetime(2018, 1, 7, 16, 17, 45, 45005),
datetime.datetime(2018, 1, 14, 16, 17, 45, 45005),
datetime.datetime(2018, 1, 21, 16, 17, 45, 45005),
datetime.datetime(2018, 1, 28, 16, 17, 45, 45005)
]
One very clean way to implement this is using rrule from the dateutil library. For example:
>>> from dateutil.rrule import rrule, DAILY
>>> from datetime import datetime
>>> start_date = datetime.now()
>>> {d.strftime("%A"): d for d in rrule(freq=DAILY, count=7, dtstart=start_date)}
which will return your desired dict object:
{
'Sunday': datetime.datetime(2018, 1, 7, 17, 2, 30),
'Monday': datetime.datetime(2018, 1, 8, 17, 2, 30),
'Tuesday': datetime.datetime(2018, 1, 9, 17, 2, 30),
'Wednesday': datetime.datetime(2018, 1, 10, 17, 2, 30),
'Thursday': datetime.datetime(2018, 1, 11, 17, 2, 30),
'Friday': datetime.datetime(2018, 1, 12, 17, 2, 30),
'Saturday': datetime.datetime(2018, 1, 13, 17, 2, 30)
}
(Special thanks to Jon Clements for telling me about rrule)

How to merge list of dicts based on dict attribute uniquely

I have a two Django Querysets that I want to merge based on its date atrribute. Well it is not really Django question, but I try to explain as clearly as I can.
I need to group entries based on two data attributes. Lets say I have a model:
class User(models.Model):
start_date = models.DateField(blank=True, null=True)
end_date = models.DateField(blank=True, null=True)
...
Now I need to group these entries by month (how many users started on May 2010 etc):
truncate_start_date = connection.ops.date_trunc_sql('month', 'start_date')
report_start = User.objects.exclude(start_date__isnull=True)\
.extra({'month': truncate_start_date}).values('month')\
.annotate(start_count=Count('pk')).order_by('-month')
and I have same query for end_date:
truncate_end_date = connection.ops.date_trunc_sql('month', 'end_date')
report_end = Employee.objects.exclude(end_date__isnull=True)\
.extra({'month': truncate_end_date}).values('month')\
.annotate(end_count=Count('pk')).order_by('-month')
Now this is what report_start looks like:
[{'start_count': 33, 'month': datetime.datetime(2016, 5, 1, 0, 0, tzinfo=<UTC>)},
{'start_count': 79, 'month': datetime.datetime(2016, 4, 1, 0, 0, tzinfo=<UTC>)},
{'start_count': 72, 'month': datetime.datetime(2016, 3, 1, 0, 0, tzinfo=<UTC>)},
... ]
Now, how do I merge these two lists of dicts to one based on month? I tried chain, but there were duplicate month records.
I want to get:
[{'start_count': 33, 'end_count': None, 'month': datetime.datetime(2016, 5, 1, 0, 0, tzinfo=<UTC>)},
{'start_count': 79, 'end_count': 2, 'month': datetime.datetime(2016, 4, 1, 0, 0, tzinfo=<UTC>)},
{'start_count': 72, 'end_count': 8, 'month': datetime.datetime(2016, 3, 1, 0, 0, tzinfo=<UTC>)},
... ]
What I was able to come up with was to change it to dict and then back to list of dicts. But I believe this is not very elegant solution and there must be a better way to write this pythonic way.
Any ideas? Here is my ugly code:
d = dict()
for end in report_end:
d[end['month']] = {"end_count": end['end_count']}
for start in report_start:
if start['month'] in d.keys():
d[start['month']]["start_count"] = start['start_count']
else:
d[start['month']] = {"start_count": start['start_count']}
result = []
for key, i in d.items():
result.append({'month': key,
'start_count': i['start_count'] if 'start_count' in i.keys() else None,
'end_count': i['end_count'] if 'end_count' in i.keys() else None})
datetime is hashable, so you can store it as a key to a dict and merge easily. Here is a bit terser solution using itemgetter. This assumes that your timestamps are unique within each list of dicts.
from operator import itemgetter
import datetime
starts = [
{'start_count': 33, 'month': datetime.datetime(2016, 5, 1, 0, 0)},
{'start_count': 79, 'month': datetime.datetime(2016, 4, 1, 0, 0)},
{'start_count': 72, 'month': datetime.datetime(2016, 3, 1, 0, 0)}
]
# dummy data
ends = [
{'end_count': 122, 'month': datetime.datetime(2016, 5, 1, 0, 0)},
{'end_count': 213, 'month': datetime.datetime(2016, 4, 1, 0, 0)},
{'end_count': 121, 'month': datetime.datetime(2016, 3, 1, 0, 0)}
]
starts = dict(map(itemgetter('month', 'start_count'), starts))
ends = dict(map(itemgetter('month', 'end_count'), ends))
joined = [{'month': m, 'start_count': s, 'end_count': ends.get(m, None)}
for m, s in starts.items()]

Django Queryset Datetime Filter

I have a column formatted as such in one of my models:
TEMP_START = models.DateTimeField(null=True)
And I am attempting to do an exact lookup using queryset syntax such as
x.filter(TEMP_START=my_datetime_object) # x can be thought of as y.objects.all()
This returns no objects, when it should do more than one. However,
x.filter(TEMP_START__date=my_datetime_object.date()).filter(TEMP_START__hour=my_datetime_object.hour)
Does return the proper objects (they're hourly). Are direct datetime filters not supported, and thus keywords must be used?
====== Edit with bad results:
Searching for: {'TEMP_START': datetime.datetime(2016, 3, 31, 2, 0)}
Values in column: [{'TEMP_START': datetime.datetime(2016, 3, 29, 8, 0)}, {'TEMP_START': datetime.datetime(2016, 3, 29, 14, 0)}, {'TEMP_START': datetime.datetime(2016, 3, 30, 2, 0)}, {'TEMP_START': datetime.datetime(2016, 3, 29, 20, 0)}, {'TEMP_START': datetime.datetime(2016, 3, 30, 8, 0)}, {'TEMP_START': datetime.datetime(2016, 3, 30, 20, 0)}, {'TEMP_START': datetime.datetime(2016, 3, 31, 2, 0)}, {'TEMP_START': datetime.datetime(2016, 3, 30, 14, 0)}]
Values being returned: []
Code:
args_timeframe_start = {variables.temp_start: self.ranked_timeframes[rank][variables.temp_start]}
print(args_timeframe_start)
print(self.query_data.values(variables.temp_start).distinct())
query_data = self.query_data.filter(**args_timeframe_start)
print(query_data.values(variables.temp_start).distinct())
You need to find out what is my_datetime_object but most likely because DateTime fields contain python datetime.datetime objects, datetime.datetime objects is composed of year, month, date, hour, minute, second, microsecond. So if you merely compare date and hour, sure you could get results, but you couldn't guarantee that my_datetime_object matches one of the records in your database that has the same minute, second and microsecond.
Try this quickly and you could see what does datetime look like, also django doc about DateTimeField:
from datetime import datetime
date = datetime.now()
print date

How to calculate running time from status and time using python

I have a circulation pump that I check wither it's on or off on and this is not by any fixed interval what so ever. For a single day that could give me a dataset looking like this where 'value' represents the pump being on or off.
data=(
{'value': 0, 'time': datetime.datetime(2011, 1, 18, 7, 58, 25)},
{'value': 1, 'time': datetime.datetime(2011, 1, 18, 8, 0, 3)},
{'value': 0, 'time': datetime.datetime(2011, 1, 18, 8, 32, 10)},
{'value': 0, 'time': datetime.datetime(2011, 1, 18, 9, 22, 7)},
{'value': 1, 'time': datetime.datetime(2011, 1, 18, 9, 30, 58)},
{'value': 1, 'time': datetime.datetime(2011, 1, 18, 12, 2, 23)},
{'value': 0, 'time': datetime.datetime(2011, 1, 18, 15, 43, 11)},
{'value': 1, 'time': datetime.datetime(2011, 1, 18, 20, 14, 55)})
The format is not that important and can be changed.
What I do want to know is how to calculate how many minutes ( or timespan or whatever) the 'value' has been 0 or 1 (or ON or OFF)?
This is just a small sample of the data, it stretches over several years so there could be a lot.
I have been using numpy/mathplotlib for plotting some graphs and there might be something in numpy to do this but I'm not good enough at it.
Edit
What I would like to see as an output to this would be a sum of the time in the different states. Something like...
0 04:42:13
1 07:34:17
It really depends on how you're going to treat this data points, are they representative of what? Generally, to know when switch occur you could use itertools.groupby like this:
>>> from itertools import groupby
>>> for i, grp in groupby(data, key=lambda x: x['value']):
lst = [x['time'] for x in grp]
print(i, max(lst) - min(lst))
0 0:00:00
1 0:00:00
0 0:49:57
1 2:31:25
0 0:00:00
1 0:00:00
This is the example of minimal time you can be sure your system was up or down (assuming no interruptions between measurement).
Once you decide how to treat your points, modification to this algorithm would be trivial.
EDIT: since you only need sums of up/down-time, here is the simpler version:
>>> sums = {0:datetime.timedelta(0), 1:datetime.timedelta(0)}
>>> for cur, nex in zip(data, data[1:]):
sums[cur['value']] += nex['time'] - cur['time']
>>> for i, j in sums.items():
print(i, j)
0 5:32:10
1 6:44:20
If you expect long-periods of continuous up/down-time, you might still benefit of the itertools.groupby. This is py3k version, so it won't be particularly efficient in py2k.

How can I specify date and time in Python?

What is the object used in Python to specify date (and time) in Python?
For instance, to create an object that holds a given date and time, (let's say '05/10/09 18:00').
As per S.Lott's request, so far I have:
class Some:
date =
I stop there. After the "=" sign for, I realize I didn't knew what the right object was ;)
Simple example:
>>> import datetime
# 05/10/09 18:00
>>> d = datetime.datetime(2009, 10, 5, 18, 00)
>>> print d.year, d.month, d.day, d.hour, d.second
2009 10 5 18 0
>>> print d.isoformat(' ')
2009-10-05 18:00:00
>>>
Nick D has the official way of handling your problem. If you want to pass in a string like you did in your question, the dateutil module (http://labix.org/python-dateutil) has excellent support for that kind of thing.
For examples, I'm going to copy and paste from another answer I gave a while back now:
Simple example:
>>> parse("Thu Sep 25 2003")
datetime.datetime(2003, 9, 25, 0, 0)
>>> parse("Sep 25 2003")
datetime.datetime(2003, 9, 25, 0, 0)
>>> parse("Sep 2003", default=DEFAULT)
datetime.datetime(2003, 9, 25, 0, 0)
>>> parse("Sep", default=DEFAULT)
datetime.datetime(2003, 9, 25, 0, 0)
>>> parse("2003", default=DEFAULT)
datetime.datetime(2003, 9, 25, 0, 0)
To ambigous:
>>> parse("10-09-2003")
datetime.datetime(2003, 10, 9, 0, 0)
>>> parse("10-09-2003", dayfirst=True)
datetime.datetime(2003, 9, 10, 0, 0)
>>> parse("10-09-03")
datetime.datetime(2003, 10, 9, 0, 0)
>>> parse("10-09-03", yearfirst=True)
datetime.datetime(2010, 9, 3, 0, 0)
To all over the board:
>>> parse("Wed, July 10, '96")
datetime.datetime(1996, 7, 10, 0, 0)
>>> parse("1996.07.10 AD at 15:08:56 PDT", ignoretz=True)
datetime.datetime(1996, 7, 10, 15, 8, 56)
>>> parse("Tuesday, April 12, 1952 AD 3:30:42pm PST", ignoretz=True)
datetime.datetime(1952, 4, 12, 15, 30, 42)
>>> parse("November 5, 1994, 8:15:30 am EST", ignoretz=True)
datetime.datetime(1994, 11, 5, 8, 15, 30)
>>> parse("3rd of May 2001")
datetime.datetime(2001, 5, 3, 0, 0)
>>> parse("5:50 A.M. on June 13, 1990")
datetime.datetime(1990, 6, 13, 5, 50)
Take a look at the documentation for it here:
http://labix.org/python-dateutil#head-c0e81a473b647dfa787dc11e8c69557ec2c3ecd2
Look at the datetime module; there are datetime, date and timedelta class definitions.
>>> import datetime
>>> datetime.datetime.strptime('05/10/09 18:00', '%d/%m/%y %H:%M')
datetime.datetime(2009, 10, 5, 18, 0)
>>> datetime.datetime.today()
datetime.datetime(2009, 10, 5, 21, 3, 55, 827787)
So, you can either use format string to convert to datetime.datetime object or if you're particularly looking at today's date could use today() function.

Categories