Related
I wrote a generator in python that yields a new day of data at each call from a pandas DataFrame. My DataFrame is unix timestamp indexed. My first attempt at a code worked as follows (df is the dataframe, tz is a pytz.timezone (Europe/Amsterdam in my case):
def interval_generator(df, tz):
today = datetime.datetime.fromtimestamp(df.index.min(), tz)
last_day = datetime.datetime.fromtimestamp(df.index.max(), tz)
while today <= last_day:
tomorrow = today + datetime.timedelta(days=1)
yield df.loc[tz.localize(today).timestamp():tz.localize(tomorrow).timestamp() - 1]
today = tomorrow
However when running my code I noticed that the DateTime object has the weird behaviour of really sticking with the timezone it was initially attached to (especially the incremented hour). Example of the (in my eyes) weird behaviour:
import datetime
import pytz
tz = pytz.timezone('Europe/Amsterdam')
# This is when daylight saving times stops in the Netherlands in 2015.
t1 = datetime.datetime(2015, 10, 25, 0, 0)
t2 = t1 + datetime.timedelta(days=1)
t1_localized = tz.localize(t1)
t2_localized = tz.localize(t2)
t2_loc_incremented = t1_localized + datetime.timedelta(days=1)
When printing the output of these final three variables you get:
>>> t1_localized
datetime.datetime(2015, 10, 25, 0, 0, tzinfo=<DstTzInfo 'Europe/Amsterdam' CEST+2:00:00 DST>)
>>> t2_localized
datetime.datetime(2015, 10, 26, 0, 0, tzinfo=<DstTzInfo 'Europe/Amsterdam' CET+1:00:00 STD>)
>>> t2_loc_incremented
datetime.datetime(2015, 10, 26, 0, 0, tzinfo=<DstTzInfo 'Europe/Amsterdam' CEST+2:00:00 DST>)
More importantly for my code, the timestamp for both versions of t2 is different:
>>> t2_localized.timestamp()
1445814000.0
>>> t2_loc_incremented.timestamp()
1445810400.0
I solved this in my generator function with the following workaround:
def interval_generator(df, tz):
today = datetime.datetime.fromtimestamp(df.index.min(), tz=tz).strftime('%Y-%m-%d')
today = datetime.datetime.strptime(today, '%Y-%m-%d')
last_day = datetime.datetime.fromtimestamp(df.index.max(), tz=tz).strftime('%Y-%m-%d')
last_day = datetime.datetime.strptime(last_day, '%Y-%m-%d')
while today <= last_day:
tomorrow = today + datetime.timedelta(days=1)
yield df.loc[tz.localize(today).timestamp():tz.localize(tomorrow).timestamp() - 1]
today = tomorrow
Which basically gets me the desired functionality, but I can't help but wonder whether there isn't a better way to deal with this issue. Is there any good alternative for my problem? Is this considered a bug of the datetime module? (I am using python 3.4) I tried googling, but could not find anything
In Python, I can find the Unix time stamp of a local time, knowing the time zone, like this (using pytz):
>>> import datetime as DT
>>> import pytz
>>> mtl = pytz.timezone('America/Montreal')
>>> naive_time3 = DT.datetime.strptime('2013/11/03', '%Y/%m/%d')
>>> naive_time3
datetime.datetime(2013, 11, 3, 0, 0)
>>> localized_time3 = mtl.localize(naive_time3)
>>> localized_time3
datetime.datetime(2013, 11, 3, 0, 0, tzinfo=<DstTzInfo 'America/Montreal' EDT-1 day, 20:00:00 DST>)
>>> localized_time3.timestamp()
1383451200.0
So far, so good. naive_time is not aware of the time zone, whereas localized_time knows its midnight on 2013/11/03 in Montréal, so the (UTC) Unix time stamp is good. This time zone is also my local time zone and this time stamp seems right:
$ date -d #1383451200
Sun Nov 3 00:00:00 EDT 2013
Now, clocks were adjusted one hour backward November 3rd at 2:00 here in Montréal, so we gained an extra hour that day. This means that there were, here, 25 hours between 2013/11/03 and 2013/11/04. This shows it:
>>> naive_time4 = DT.datetime.strptime('2013/11/04', '%Y/%m/%d')
>>> localized_time4 = mtl.localize(naive_time4)
>>> localized_time4
datetime.datetime(2013, 11, 4, 0, 0, tzinfo=<DstTzInfo 'America/Montreal' EST-1 day, 19:00:00 STD>)
>>> (localized_time4.timestamp() - localized_time3.timestamp()) / 3600
25.0
Now, I'm looking for an easy way to get the localized_time4 object from localized_time3, knowing I want to get the next localized day at the same hour (here, midnight). I tried timedelta, but I believe it's not aware of time zones or DST:
>>> localized_time4td = localized_time3 + DT.timedelta(1)
>>> localized_time4td
datetime.datetime(2013, 11, 4, 0, 0, tzinfo=<DstTzInfo 'America/Montreal' EDT-1 day, 20:00:00 DST>)
>>> (localized_time4td.timestamp() - localized_time3.timestamp()) / 3600
24.0
My purpose is to get informations about log entries that are stored with their Unix timestamp for each local day. Of course, if I use localized_time3.timestamp() and add 24 * 3600 here (which will be the same as localized_time4td.timestamp()), I will miss all log entries that happened between localized_time4td.timestamp() and localized_time4td.timestamp() + 3600.
In other words, the function or method I'm looking for should know when to add 25 hours, 24 hours or 23 hours sometimes to a Unix time stamp, depending on when DST shifts happen.
Without using a new package:
def add_day(x):
d = x.date()+DT.timedelta(1)
return mtl.localize(x.replace(year=d.year, month=d.month, day=d.day, tzinfo=None))
Full script:
import datetime as DT
import pytz
import calendar
mtl = pytz.timezone('America/Montreal')
naive_time3 = DT.datetime.strptime('2013/11/03', '%Y/%m/%d')
print repr(naive_time3)
#datetime.datetime(2013, 11, 3, 0, 0)
localized_time3 = mtl.localize(naive_time3)
print repr(localized_time3)
#datetime.datetime(2013, 11, 3, 0, 0, tzinfo=<DstTzInfo 'America/Montreal' EDT-1 day, 20:00:00 DST>)
print calendar.timegm(localized_time3.utctimetuple())
#1383451200.0
def add_day(x):
d = x.date()+DT.timedelta(1)
return mtl.localize(x.replace(year=d.year, month=d.month, day=d.day, tzinfo=None))
print repr(add_day(localized_time3))
#datetime.datetime(2013, 11, 4, 0, 0, tzinfo=<DstTzInfo 'America/Montreal' EST-1 day, 19:00:00 STD>)
(calendar is for Python2.)
I gradually provide several solutions with the most robust solution at the very end of this answer that tries to handle the following issues:
utc offset due to DST
past dates when the local timezone might have had different utc offset due to reason unrelated to DST. dateutil and stdlib solutions fail here on some systems, notably Windows
ambiguous times during DST (don't know whether Arrow provides interface to handle it)
non-existent times during DST (the same)
To find POSIX timestamp for tomorrow's midnight (or other fixed hour) in a given timezone, you could use code from How do I get the UTC time of “midnight” for a given timezone?:
from datetime import datetime, time, timedelta
import pytz
DAY = timedelta(1)
tz = pytz.timezone('America/Montreal')
tomorrow = datetime(2013, 11, 3).date() + DAY
midnight = tz.localize(datetime.combine(tomorrow, time(0, 0)), is_dst=None)
timestamp = (midnight - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
dt.date() method returns the same naive date for both naive and timezone-aware dt objects.
The explicit formula for timestamp is used to support Python version before Python 3.3. Otherwise .timestamp() method could be used in Python 3.3+.
To avoid ambiguity in parsing input dates during DST transitions that are unavoidable for .localize() method unless you know is_dst parameter, you could use Unix timestamps stored with the dates:
from datetime import datetime, time, timedelta
import pytz
DAY = timedelta(1)
tz = pytz.timezone('America/Montreal')
local_dt = datetime.fromtimestamp(timestamp_from_the_log, tz)
tomorrow = local_dt.date() + DAY
midnight = tz.localize(datetime.combine(tomorrow, time(0, 0)), is_dst=None)
timestamp = (midnight - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
To support other fixed hours (not only midnight):
tomorrow = local_dt.replace(tzinfo=None) + DAY # tomorrow, same time
dt_plus_day = tz.localize(tomorrow, is_dst=None)
timestamp = dt_plus_day.timestamp() # use the explicit formula before Python 3.3
is_dst=None raises an exception if the result date is ambiguous or non-existent. To avoid exception, you could choose the time that is closest to the previous date from yesterday (same DST state i.e., is_dst=local_dt.dst()):
from datetime import datetime, time, timedelta
import pytz
DAY = timedelta(1)
tz = pytz.timezone('America/Montreal')
local_dt = datetime.fromtimestamp(timestamp_from_the_log, tz)
tomorrow = local_dt.replace(tzinfo=None) + DAY
dt_plus_day = tz.localize(tomorrow, is_dst=local_dt.dst())
dt_plus_day = tz.normalize(dt_plus_day) # to detect non-existent times
timestamp = (dt_plus_day - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
.localize() respects given time even if it is non-existent, therefore .normalize() is required to fix the time. You could raise an exception here if normalize() method changes its input (non-existent time detected in this case) for consistency with other code examples.
(Thanks to #rdodev for pointing me to Arrow).
Using Arrow, this operation becomes easy:
>>> import arrow
>>> import datetime as DT
>>> lt3 = arrow.get(DT.datetime(2013, 11, 3), 'America/Montreal')
>>> lt3
<Arrow [2013-11-03T00:00:00-04:00]>
>>> lt4 = arrow.get(DT.datetime(2013, 11, 4), 'America/Montreal')
>>> lt4
<Arrow [2013-11-04T00:00:00-05:00]>
>>> lt4.timestamp - (lt3.replace(days=1).timestamp)
0
>>> (lt3.replace(days=1).timestamp - lt3.timestamp) / 3600
25.0
Using Arrow's replace method, singular unit names replace that property while plural adds to it. So lt3.replace(days=1) is November 4th, 2013 while lt3.replace(day=1) is November 1st, 2013.
Here an alternative based on dateutil:
>>> # In Spain we changed DST 10/26/2013
>>> import datetime
>>> import dateutil.tz
>>> # tzlocal gets the timezone of the computer
>>> dt1 = datetime.datetime(2013, 10, 26, 14, 00).replace(tzinfo=dateutil.tz.tzlocal())
>>> print dt1
2013-10-26 14:00:00+02:00
>>> dt2 = dt1 + datetime.timedelta(1)
>>> print dt2
2013-10-27 14:00:00+01:00
# see if we hace 25 hours of difference
>>> import time
>>> (time.mktime(dt2.timetuple()) - time.mktime(dt1.timetuple())) / 3600.0
25.0
>>> (float(dt2.strftime('%s')) - float(dt1.strftime('%s'))) / 3600 # the same
25.0
I am using Django.
in settings:
TIME_ZONE = 'Europe/Copenhagen'
USE_TZ = True
Due to DST, clock skips an hour on 2013-3-31. 01:59 goes to 03:00
I views:
The date and time are given in local time. I want these to be inserted as utc.
Code below correctly saves as UTC, but gives RuntimeWarning: DateTimeField received a naive datetime
the_date = datetime.datetime(2013, 3, 31, 1, 59)
hit = hits(date= the_date); hit.save(); # Correctly saved as 00:59:00
the_date = datetime.datetime(2013, 3, 31, 3, 1)
hit = hits(date= the_date); hit.save(); # Correctly saved as 01:01:00
I thought I could avoid the warning by making the datetime aware. It does avoid the warning, but the conversion is now wrong.
tz = timezone(settings.TIME_ZONE)
the_date = datetime.datetime(2013, 3, 31, 3, 1, tzinfo = tz)
hit = hits(date= the_date); hit.save(); # Incorrectly saved as 02:01:00
The following works, with no runtime error:
I have installed pytz.
the_date = local_tz.localize(datetime.datetime(2013, 3, 31, 3, 1))
Getting to my question:
I get that tzinfo doesn't work because it doesn't account for daylight savings time. Fine, I won't use it. But then I was confused when the following seemed to work:
the_date = datetime.datetime.now(local_tz)
This correctly inserted as utc both in wintertime (where it subtracted 1 hour to get utc) and when I changed my computer systime to a date in the summer (where it subtracted 2 hours to get utc).
My question:
Does .now(local_tz) work or did I test it wrong? Why is this different than tzinfo = tz? Or am I using tzinfo wrong?
I recommend converting to UTC as soon as possible and only use UTC internally. Save for cases where you move across timezones (for example a marine vessel) and really need to save timezone information, for the constant timezone case, it's much simpler to just use local time for input/output and convert it to UTC at the user interface.
To convert from localtime to UTC, you need to use the pytz.timezone.normalize method which handles daylight savings time and other timezone transition. See this section of the pytz documentation. In your case, to convert the local datetime to UTC you need the following:
from pytz import timezone, utc
local_tz = timezone(settings.TIME_ZONE)
local_dt = datetime.datetime(2013, 3, 31, 3, 1, tzinfo = local_tz)
utc_dt = utc.normalize(local_dt.astimezone(utc))
I am writing a program which deals a lot with timezones and crossing them. The two things I deal with most are creating a datetime object from "now" and then localizing a naive datetime object.
To create a datetime object from now in the pacific timezone, I am currently doing this (python 2.7.2+)
from datetime import datetime
import pytz
la = pytz.timezone("America/Los_Angeles")
now = datetime.now(la)
Is this correct with regards to DST? If not, I suppose I should be doing:
now2 = la.localize(datetime.now())
My question is why? Can anyone show me a case where the first is wrong and the seconds is right?
As for my seconds question, suppose I had a naive date and time from some user input for 9/1/2012 at 8:00am in Los Angeles, CA. Is the right way to make the datetime like this:
la.localize(datetime(2012, 9, 1, 8, 0))
If not, how should I be building these datetimes?
From the pytz documentation:
The preferred way of dealing with times is to always work in UTC, converting to localtime only when generating output to be read by humans.
So ideally you should be using utcnow instead of now.
Assuming for some reason that your hands are tied and you need to work with local times, you can still run into a problem with trying to localize the current time if you're doing it during the daylight saving transition window. The same datetime might occur twice, once during daylight time and again during standard time, and the localize method doesn't know how to settle the conflict unless you tell it explicitly with the is_dst parameter.
So to get the current UTC time:
utc = pytz.timezone('UTC')
now = utc.localize(datetime.datetime.utcnow())
And to convert it to your local time (but only when you must):
la = pytz.timezone('America/Los_Angeles')
local_time = now.astimezone(la)
Edit: as pointed out in the comments by #J.F. Sebastian, your first example using datetime.now(tz) will work in all cases. Your second example fails during the fall transition as I outlined above. I still advocate using UTC instead of local time for everything except display.
The first solution is correct with regards to DST, and the second solution is bad.
I'll give an example. Here in Europe, when running this code:
from datetime import datetime
import pytz # $ pip install pytz
la = pytz.timezone("America/Los_Angeles")
fmt = '%Y-%m-%d %H:%M:%S %Z%z'
now = datetime.now(la)
now2 = la.localize(datetime.now())
now3 = datetime.now()
print(now.strftime(fmt))
print(now2.strftime(fmt))
print(now3.strftime(fmt))
I get the following:
2012-08-30 12:34:06 PDT-0700
2012-08-30 21:34:06 PDT-0700
2012-08-30 21:34:06
datetime.now(la) creates a datetime with the current time in LA, plus the timezone information for LA.
la.localize(datetime.now()) adds timezone information to the naive datetime, but does no timezone conversion; it just assumes the time was already in this timezone.
datetime.now() creates a naive datetime (without timezone information) with the local time.
As long as you are in LA, you will not see the difference, but if your code ever runs somewhere else, it will probably not do what you wanted.
Apart from that, if you ever need to deal seriously with timezones, it is better to have all your times in UTC, saving yourself a lot of trouble with DST.
This works:
# naive datetime
d = datetime.datetime(2016, 11, 5, 16, 43, 45)
utc = pytz.UTC # UTC timezone
pst = pytz.timezone('America/Los_Angeles') # LA timezone
# Convert to UTC timezone aware datetime
d = utc.localize(d)
>>> datetime.datetime(2016, 11, 5, 16, 43, 45, tzinfo=<UTC>)
# show as in LA time zone (not converting here)
d.astimezone(pst)
>>> datetime.datetime(2016, 11, 5, 9, 43, 45,
tzinfo=<DstTzInfo 'America/Los_Angeles' PDT-1 day, 17:00:00 DST>)
# we get Pacific Daylight Time: PDT
# add 1 day to UTC date
d = d + datetime.timedelta(days=1)
>>> datetime.datetime(2016, 11, 6, 16, 43, 45, tzinfo=<UTC>)
d.astimezone(pst) # now cast to LA time zone
>>> datetime.datetime(2016, 11, 6, 8, 43, 45,
tzinfo=<DstTzInfo 'America/Los_Angeles' PST-1 day, 16:00:00 STD>)
# Daylight saving is applied -> we get Pacific Standard Time PST
This DOES NOT work:
# naive datetime
d = datetime.datetime(2016, 11, 5, 16, 43, 45)
utc = pytz.UTC # UTC timezone
pst = pytz.timezone('America/Los_Angeles') # LA timezone
# convert to UTC timezone aware datetime
d = utc.localize(d)
>>> datetime.datetime(2016, 11, 5, 16, 43, 45, tzinfo=<UTC>)
# convert to 'America/Los_Angeles' timezone: DON'T DO THIS
d = d.astimezone(pst)
>>> datetime.datetime(2016, 11, 5, 9, 43, 45,
tzinfo=<DstTzInfo 'America/Los_Angeles' PDT-1 day, 17:00:00 DST>)
# we are in Pacific Daylight Time PDT
# add 1 day to LA local date: DON'T DO THAT
d = d + datetime.timedelta(days=1)
>>> datetime.datetime(2016, 11, 6, 9, 43, 45,
tzinfo=<DstTzInfo 'America/Los_Angeles' PDT-1 day, 17:00:00 DST>)
# Daylight Saving is NOT respected, we are still in PDT time, not PST
Conclusion:
datetime.timedelta() DOES NOT account for daylight saving.
Do your time add/subtract in UTC timezone ALWAYS.
Cast to local time only for output / display.
The pytz website says:
Unfortunately using the tzinfo argument of the standard datetime
constructors ‘’does not work’’ with pytz for many timezones.
So you should not use datetime.now(la). I don't know the specifics, but some timezones operate on more exotic rules then we are used to, and python's datetime code can't handle them. By using pytz's code, they should be handled correctly since that's pytz's purpose. It may also have issues for the times that occur twice thanks to jumping times in daylight savings.
As for the second question, that's exactly what the documentation shows, so you should be good.
What is the cleanest and most Pythonic way to get tomorrow's date? There must be a better way than to add one to the day, handle days at the end of the month, etc.
datetime.date.today() + datetime.timedelta(days=1) should do the trick
timedelta can handle adding days, seconds, microseconds, milliseconds, minutes, hours, or weeks.
>>> import datetime
>>> today = datetime.date.today()
>>> today
datetime.date(2009, 10, 1)
>>> today + datetime.timedelta(days=1)
datetime.date(2009, 10, 2)
>>> datetime.date(2009,10,31) + datetime.timedelta(hours=24)
datetime.date(2009, 11, 1)
As asked in a comment, leap days pose no problem:
>>> datetime.date(2004, 2, 28) + datetime.timedelta(days=1)
datetime.date(2004, 2, 29)
>>> datetime.date(2004, 2, 28) + datetime.timedelta(days=2)
datetime.date(2004, 3, 1)
>>> datetime.date(2005, 2, 28) + datetime.timedelta(days=1)
datetime.date(2005, 3, 1)
No handling of leap seconds tho:
>>> from datetime import datetime, timedelta
>>> dt = datetime(2008,12,31,23,59,59)
>>> str(dt)
'2008-12-31 23:59:59'
>>> # leap second was added at the end of 2008,
>>> # adding one second should create a datetime
>>> # of '2008-12-31 23:59:60'
>>> str(dt+timedelta(0,1))
'2009-01-01 00:00:00'
>>> str(dt+timedelta(0,2))
'2009-01-01 00:00:01'
darn.
EDIT - #Mark: The docs say "yes", but the code says "not so much":
>>> time.strptime("2008-12-31 23:59:60","%Y-%m-%d %H:%M:%S")
(2008, 12, 31, 23, 59, 60, 2, 366, -1)
>>> time.mktime(time.strptime("2008-12-31 23:59:60","%Y-%m-%d %H:%M:%S"))
1230789600.0
>>> time.gmtime(time.mktime(time.strptime("2008-12-31 23:59:60","%Y-%m-%d %H:%M:%S")))
(2009, 1, 1, 6, 0, 0, 3, 1, 0)
>>> time.localtime(time.mktime(time.strptime("2008-12-31 23:59:60","%Y-%m-%d %H:%M:%S")))
(2009, 1, 1, 0, 0, 0, 3, 1, 0)
I would think that gmtime or localtime would take the value returned by mktime and given me back the original tuple, with 60 as the number of seconds. And this test shows that these leap seconds can just fade away...
>>> a = time.mktime(time.strptime("2008-12-31 23:59:60","%Y-%m-%d %H:%M:%S"))
>>> b = time.mktime(time.strptime("2009-01-01 00:00:00","%Y-%m-%d %H:%M:%S"))
>>> a,b
(1230789600.0, 1230789600.0)
>>> b-a
0.0
Even the basic time module can handle this:
import time
time.localtime(time.time() + 24*3600)
For people who are dealing with servers Time Stamp
To get yesterday Time Stamp:
yesterdaytimestamp = datetime.datetime.today() + datetime.timedelta(days=-1)
To get Today Time Stamp:
currenttimestamp = datetime.datetime.now().timestamp()
To get Tomorrow Time Stamp:
tomorrowtimestamp = datetime.datetime.today() + datetime.timedelta(days=1)
To print:
print('\n Yesterday TimeStamp is : ', yesterdaytimestamp.timestamp(),
'\n Today TimeStamp is :', currenttimestamp,
'\n Tomorrow TimeStamp is: ', tomorrowtimestamp.timestamp())
The output:
Yesterday TimeStamp is : 1632842904.110993
Today TimeStamp is : 1632929304.111022
Tomorrow TimeStamp is : 1633015704.11103
There's nothing at all wrong with using today() as shown in the selected answer if that is the extent of your needs.
datetime.date.today() + datetime.timedelta(days=1)
Alternatively, if you or someone else working with your code might need more precision in handling tomorrow's date, consider using datetime.now() instead of today(). This will certainly allow for simpler, more readable code:
datetime.datetime.now() + datetime.timedelta(days=1)
This returns something like:
datetime.datetime(2022, 2, 17, 19, 50, 19, 984925)
The advantage is that you can now work with datetime attributes in a concise, human readable way:
class datetime.datetime
A combination of a date and a time. Attributes: year, month, day, hour, minute, second, microsecond, and tzinfo.
Examples
You can easily convert this to a date object withdate():
import datetime
tomorrow = datetime.datetime.now() + datetime.timedelta(days=1)
print(f"Tomorrow's date is {tomorrow.date()}")
tomorrow.date() is easy to use and it is very clear to anyone reading your code that it is returning the date for tomorrow. The output for the above looks like so:
Tomorrow's date is 2022-02-17
If later in your code you only need the date number for the day, you can now use tomorrow.day:
print(f"Tomorrow is the {tomorrow.day}rd")
Which will return something like:
Tomorrow is the 17rd
That's a silly example, but you can see how having access to these attributes can be useful and keep your code readable as well. It can be easily understood that tomorrow.day returns the day number.
Need to work with the exact time tomorrow's date begins? You can now replace the hours, minutes, seconds, and microseconds:
# Replace all attributes except day with 0.
midnight = tomorrow.replace(
hour=0,
minute=0,
second=0,
microsecond=0)
# Print midnight as the beginning of tomorrow's date.
print(f"{midnight}")
Reading the above code, it should be apparent which attributes of tomorrow are being replaced. When midnight is printed, it will output:
2022-02-17 00:00:00
Need to know the time left until tomorrow's date? Now something like that is possible, simple, and readable:
print(f"{midnight - datetime.datetime.now()}")
The output is the time to the microsecond that tomorrow's date begins:
3:14:28.158331
There are many ways people might wish to handle tomorrow's date. By ensuring these attributes are available from the beginning, you can write more readable code and avoid unnecessary work later.
For the case you only want to calculate the timestamp
import time
tomorrow = (int(time.time() / 86400) + 1) * 86400