We Keep Coding

Converting time in python with UTC Offset hour only [duplicate]

This question already has answers here:
Weird timezone issue with pytz
(3 answers)
Python datetime object show wrong timezone offset
(2 answers)
Closed 12 days ago.
Consider the following:
from datetime import datetime
import pytz
new_years_in_new_york = datetime(
year=2020,
month=1,
day=1,
hour=0,
minute=0,
tzinfo = pytz.timezone('US/Eastern'))
I now I have a datetime object representing January 1, midnight, in New York. Oddly, if I use pytz to convert this to UTC, I'll get an odd datetime off by several minutes:
new_years_in_new_york.astimezone(pytz.utc)
# datetime.datetime(2020, 1, 1, 4, 56, tzinfo=<UTC>)
Notice that midnight in New York, in pytz, is 4:56 in UTC. Elsewhere on Stack Overflow, I learned that's because pytz uses your /usr/share/zoneinfo data, which uses local mean time to account for timezones before standardization. This can be shown here:
pytz.timezone('US/Eastern')
# <DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>
See that LMK-1 day, 19:04:00 STD? That's a local mean time offset, not the offset I want, which is US/Eastern not during daylight savings time.
Is there a way I can force pytz to use what is currently the standard set of offsets based on a current date? On New Years 2020, it should just be UTC-5. If the date I supplied were during daylight savings time, I would want UTC-4. I'm confused as to why pytz would use a LMT-based offset for a 2020 date.

>>> new_years_in_new_york
datetime.datetime(2020, 1, 1, 0, 0, tzinfo=<DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>)
Notice the odd offset in that datetime. You're not creating this datetime correctly.
This library only supports two ways of building a localized time. The
first is to use the localize() method provided by the pytz library.
This is used to localize a naive datetime (datetime with no timezone
information):
>>> loc_dt = eastern.localize(datetime(2002, 10, 27, 6, 0, 0))
>>> print(loc_dt.strftime(fmt))
2002-10-27 06:00:00 EST-0500
The second way of building a localized time is by converting an
existing localized time using the standard astimezone() method:
>>> ams_dt = loc_dt.astimezone(amsterdam)
>>> ams_dt.strftime(fmt)
'2002-10-27 12:00:00 CET+0100'
Unfortunately using the tzinfo argument of the standard datetime
constructors ‘’does not work’’ with pytz for many timezones.
>>> datetime(2002, 10, 27, 12, 0, 0, tzinfo=amsterdam).strftime(fmt)
'2002-10-27 12:00:00 LMT+0020'
http://pytz.sourceforge.net/#localized-times-and-date-arithmetic

How to find next day's Unix timestamp for same hour, including DST, in Python?

In Python, I can find the Unix time stamp of a local time, knowing the time zone, like this (using pytz):
>>> import datetime as DT
>>> import pytz
>>> mtl = pytz.timezone('America/Montreal')
>>> naive_time3 = DT.datetime.strptime('2013/11/03', '%Y/%m/%d')
>>> naive_time3
datetime.datetime(2013, 11, 3, 0, 0)
>>> localized_time3 = mtl.localize(naive_time3)
>>> localized_time3
datetime.datetime(2013, 11, 3, 0, 0, tzinfo=<DstTzInfo 'America/Montreal' EDT-1 day, 20:00:00 DST>)
>>> localized_time3.timestamp()
1383451200.0
So far, so good. naive_time is not aware of the time zone, whereas localized_time knows its midnight on 2013/11/03 in Montréal, so the (UTC) Unix time stamp is good. This time zone is also my local time zone and this time stamp seems right:
$ date -d #1383451200
Sun Nov 3 00:00:00 EDT 2013
Now, clocks were adjusted one hour backward November 3rd at 2:00 here in Montréal, so we gained an extra hour that day. This means that there were, here, 25 hours between 2013/11/03 and 2013/11/04. This shows it:
>>> naive_time4 = DT.datetime.strptime('2013/11/04', '%Y/%m/%d')
>>> localized_time4 = mtl.localize(naive_time4)
>>> localized_time4
datetime.datetime(2013, 11, 4, 0, 0, tzinfo=<DstTzInfo 'America/Montreal' EST-1 day, 19:00:00 STD>)
>>> (localized_time4.timestamp() - localized_time3.timestamp()) / 3600
25.0
Now, I'm looking for an easy way to get the localized_time4 object from localized_time3, knowing I want to get the next localized day at the same hour (here, midnight). I tried timedelta, but I believe it's not aware of time zones or DST:
>>> localized_time4td = localized_time3 + DT.timedelta(1)
>>> localized_time4td
datetime.datetime(2013, 11, 4, 0, 0, tzinfo=<DstTzInfo 'America/Montreal' EDT-1 day, 20:00:00 DST>)
>>> (localized_time4td.timestamp() - localized_time3.timestamp()) / 3600
24.0
My purpose is to get informations about log entries that are stored with their Unix timestamp for each local day. Of course, if I use localized_time3.timestamp() and add 24 * 3600 here (which will be the same as localized_time4td.timestamp()), I will miss all log entries that happened between localized_time4td.timestamp() and localized_time4td.timestamp() + 3600.
In other words, the function or method I'm looking for should know when to add 25 hours, 24 hours or 23 hours sometimes to a Unix time stamp, depending on when DST shifts happen.

Without using a new package:
def add_day(x):
d = x.date()+DT.timedelta(1)
return mtl.localize(x.replace(year=d.year, month=d.month, day=d.day, tzinfo=None))
Full script:
import datetime as DT
import pytz
import calendar
mtl = pytz.timezone('America/Montreal')
naive_time3 = DT.datetime.strptime('2013/11/03', '%Y/%m/%d')
print repr(naive_time3)
#datetime.datetime(2013, 11, 3, 0, 0)
localized_time3 = mtl.localize(naive_time3)
print repr(localized_time3)
#datetime.datetime(2013, 11, 3, 0, 0, tzinfo=<DstTzInfo 'America/Montreal' EDT-1 day, 20:00:00 DST>)
print calendar.timegm(localized_time3.utctimetuple())
#1383451200.0
def add_day(x):
d = x.date()+DT.timedelta(1)
return mtl.localize(x.replace(year=d.year, month=d.month, day=d.day, tzinfo=None))
print repr(add_day(localized_time3))
#datetime.datetime(2013, 11, 4, 0, 0, tzinfo=<DstTzInfo 'America/Montreal' EST-1 day, 19:00:00 STD>)
(calendar is for Python2.)

I gradually provide several solutions with the most robust solution at the very end of this answer that tries to handle the following issues:
utc offset due to DST
past dates when the local timezone might have had different utc offset due to reason unrelated to DST. dateutil and stdlib solutions fail here on some systems, notably Windows
ambiguous times during DST (don't know whether Arrow provides interface to handle it)
non-existent times during DST (the same)
To find POSIX timestamp for tomorrow's midnight (or other fixed hour) in a given timezone, you could use code from How do I get the UTC time of “midnight” for a given timezone?:
from datetime import datetime, time, timedelta
import pytz
DAY = timedelta(1)
tz = pytz.timezone('America/Montreal')
tomorrow = datetime(2013, 11, 3).date() + DAY
midnight = tz.localize(datetime.combine(tomorrow, time(0, 0)), is_dst=None)
timestamp = (midnight - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
dt.date() method returns the same naive date for both naive and timezone-aware dt objects.
The explicit formula for timestamp is used to support Python version before Python 3.3. Otherwise .timestamp() method could be used in Python 3.3+.
To avoid ambiguity in parsing input dates during DST transitions that are unavoidable for .localize() method unless you know is_dst parameter, you could use Unix timestamps stored with the dates:
from datetime import datetime, time, timedelta
import pytz
DAY = timedelta(1)
tz = pytz.timezone('America/Montreal')
local_dt = datetime.fromtimestamp(timestamp_from_the_log, tz)
tomorrow = local_dt.date() + DAY
midnight = tz.localize(datetime.combine(tomorrow, time(0, 0)), is_dst=None)
timestamp = (midnight - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
To support other fixed hours (not only midnight):
tomorrow = local_dt.replace(tzinfo=None) + DAY # tomorrow, same time
dt_plus_day = tz.localize(tomorrow, is_dst=None)
timestamp = dt_plus_day.timestamp() # use the explicit formula before Python 3.3
is_dst=None raises an exception if the result date is ambiguous or non-existent. To avoid exception, you could choose the time that is closest to the previous date from yesterday (same DST state i.e., is_dst=local_dt.dst()):
from datetime import datetime, time, timedelta
import pytz
DAY = timedelta(1)
tz = pytz.timezone('America/Montreal')
local_dt = datetime.fromtimestamp(timestamp_from_the_log, tz)
tomorrow = local_dt.replace(tzinfo=None) + DAY
dt_plus_day = tz.localize(tomorrow, is_dst=local_dt.dst())
dt_plus_day = tz.normalize(dt_plus_day) # to detect non-existent times
timestamp = (dt_plus_day - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
.localize() respects given time even if it is non-existent, therefore .normalize() is required to fix the time. You could raise an exception here if normalize() method changes its input (non-existent time detected in this case) for consistency with other code examples.

(Thanks to #rdodev for pointing me to Arrow).
Using Arrow, this operation becomes easy:
>>> import arrow
>>> import datetime as DT
>>> lt3 = arrow.get(DT.datetime(2013, 11, 3), 'America/Montreal')
>>> lt3
<Arrow [2013-11-03T00:00:00-04:00]>
>>> lt4 = arrow.get(DT.datetime(2013, 11, 4), 'America/Montreal')
>>> lt4
<Arrow [2013-11-04T00:00:00-05:00]>
>>> lt4.timestamp - (lt3.replace(days=1).timestamp)
0
>>> (lt3.replace(days=1).timestamp - lt3.timestamp) / 3600
25.0
Using Arrow's replace method, singular unit names replace that property while plural adds to it. So lt3.replace(days=1) is November 4th, 2013 while lt3.replace(day=1) is November 1st, 2013.

Here an alternative based on dateutil:
>>> # In Spain we changed DST 10/26/2013
>>> import datetime
>>> import dateutil.tz
>>> # tzlocal gets the timezone of the computer
>>> dt1 = datetime.datetime(2013, 10, 26, 14, 00).replace(tzinfo=dateutil.tz.tzlocal())
>>> print dt1
2013-10-26 14:00:00+02:00
>>> dt2 = dt1 + datetime.timedelta(1)
>>> print dt2
2013-10-27 14:00:00+01:00
# see if we hace 25 hours of difference
>>> import time
>>> (time.mktime(dt2.timetuple()) - time.mktime(dt1.timetuple())) / 3600.0
25.0
>>> (float(dt2.strftime('%s')) - float(dt1.strftime('%s'))) / 3600 # the same
25.0

Python Django: tzinfo doesn't work for DB insertion. But why does .now(local_tz) work?

I am using Django.
in settings:
TIME_ZONE = 'Europe/Copenhagen'
USE_TZ = True
Due to DST, clock skips an hour on 2013-3-31. 01:59 goes to 03:00
I views:
The date and time are given in local time. I want these to be inserted as utc.
Code below correctly saves as UTC, but gives RuntimeWarning: DateTimeField received a naive datetime
the_date = datetime.datetime(2013, 3, 31, 1, 59)
hit = hits(date= the_date); hit.save(); # Correctly saved as 00:59:00
the_date = datetime.datetime(2013, 3, 31, 3, 1)
hit = hits(date= the_date); hit.save(); # Correctly saved as 01:01:00
I thought I could avoid the warning by making the datetime aware. It does avoid the warning, but the conversion is now wrong.
tz = timezone(settings.TIME_ZONE)
the_date = datetime.datetime(2013, 3, 31, 3, 1, tzinfo = tz)
hit = hits(date= the_date); hit.save(); # Incorrectly saved as 02:01:00
The following works, with no runtime error:
I have installed pytz.
the_date = local_tz.localize(datetime.datetime(2013, 3, 31, 3, 1))
Getting to my question:
I get that tzinfo doesn't work because it doesn't account for daylight savings time. Fine, I won't use it. But then I was confused when the following seemed to work:
the_date = datetime.datetime.now(local_tz)
This correctly inserted as utc both in wintertime (where it subtracted 1 hour to get utc) and when I changed my computer systime to a date in the summer (where it subtracted 2 hours to get utc).
My question:
Does .now(local_tz) work or did I test it wrong? Why is this different than tzinfo = tz? Or am I using tzinfo wrong?

I recommend converting to UTC as soon as possible and only use UTC internally. Save for cases where you move across timezones (for example a marine vessel) and really need to save timezone information, for the constant timezone case, it's much simpler to just use local time for input/output and convert it to UTC at the user interface.
To convert from localtime to UTC, you need to use the pytz.timezone.normalize method which handles daylight savings time and other timezone transition. See this section of the pytz documentation. In your case, to convert the local datetime to UTC you need the following:
from pytz import timezone, utc
local_tz = timezone(settings.TIME_ZONE)
local_dt = datetime.datetime(2013, 3, 31, 3, 1, tzinfo = local_tz)
utc_dt = utc.normalize(local_dt.astimezone(utc))

Daylight savings time in Python

I am writing a program which deals a lot with timezones and crossing them. The two things I deal with most are creating a datetime object from "now" and then localizing a naive datetime object.
To create a datetime object from now in the pacific timezone, I am currently doing this (python 2.7.2+)
from datetime import datetime
import pytz
la = pytz.timezone("America/Los_Angeles")
now = datetime.now(la)
Is this correct with regards to DST? If not, I suppose I should be doing:
now2 = la.localize(datetime.now())
My question is why? Can anyone show me a case where the first is wrong and the seconds is right?
As for my seconds question, suppose I had a naive date and time from some user input for 9/1/2012 at 8:00am in Los Angeles, CA. Is the right way to make the datetime like this:
la.localize(datetime(2012, 9, 1, 8, 0))
If not, how should I be building these datetimes?

From the pytz documentation:
The preferred way of dealing with times is to always work in UTC, converting to localtime only when generating output to be read by humans.
So ideally you should be using utcnow instead of now.
Assuming for some reason that your hands are tied and you need to work with local times, you can still run into a problem with trying to localize the current time if you're doing it during the daylight saving transition window. The same datetime might occur twice, once during daylight time and again during standard time, and the localize method doesn't know how to settle the conflict unless you tell it explicitly with the is_dst parameter.
So to get the current UTC time:
utc = pytz.timezone('UTC')
now = utc.localize(datetime.datetime.utcnow())
And to convert it to your local time (but only when you must):
la = pytz.timezone('America/Los_Angeles')
local_time = now.astimezone(la)
Edit: as pointed out in the comments by #J.F. Sebastian, your first example using datetime.now(tz) will work in all cases. Your second example fails during the fall transition as I outlined above. I still advocate using UTC instead of local time for everything except display.

The first solution is correct with regards to DST, and the second solution is bad.
I'll give an example. Here in Europe, when running this code:
from datetime import datetime
import pytz # $ pip install pytz
la = pytz.timezone("America/Los_Angeles")
fmt = '%Y-%m-%d %H:%M:%S %Z%z'
now = datetime.now(la)
now2 = la.localize(datetime.now())
now3 = datetime.now()
print(now.strftime(fmt))
print(now2.strftime(fmt))
print(now3.strftime(fmt))
I get the following:
2012-08-30 12:34:06 PDT-0700
2012-08-30 21:34:06 PDT-0700
2012-08-30 21:34:06
datetime.now(la) creates a datetime with the current time in LA, plus the timezone information for LA.
la.localize(datetime.now()) adds timezone information to the naive datetime, but does no timezone conversion; it just assumes the time was already in this timezone.
datetime.now() creates a naive datetime (without timezone information) with the local time.
As long as you are in LA, you will not see the difference, but if your code ever runs somewhere else, it will probably not do what you wanted.
Apart from that, if you ever need to deal seriously with timezones, it is better to have all your times in UTC, saving yourself a lot of trouble with DST.

This works:
# naive datetime
d = datetime.datetime(2016, 11, 5, 16, 43, 45)
utc = pytz.UTC # UTC timezone
pst = pytz.timezone('America/Los_Angeles') # LA timezone
# Convert to UTC timezone aware datetime
d = utc.localize(d)
>>> datetime.datetime(2016, 11, 5, 16, 43, 45, tzinfo=<UTC>)
# show as in LA time zone (not converting here)
d.astimezone(pst)
>>> datetime.datetime(2016, 11, 5, 9, 43, 45,
tzinfo=<DstTzInfo 'America/Los_Angeles' PDT-1 day, 17:00:00 DST>)
# we get Pacific Daylight Time: PDT
# add 1 day to UTC date
d = d + datetime.timedelta(days=1)
>>> datetime.datetime(2016, 11, 6, 16, 43, 45, tzinfo=<UTC>)
d.astimezone(pst) # now cast to LA time zone
>>> datetime.datetime(2016, 11, 6, 8, 43, 45,
tzinfo=<DstTzInfo 'America/Los_Angeles' PST-1 day, 16:00:00 STD>)
# Daylight saving is applied -> we get Pacific Standard Time PST
This DOES NOT work:
# naive datetime
d = datetime.datetime(2016, 11, 5, 16, 43, 45)
utc = pytz.UTC # UTC timezone
pst = pytz.timezone('America/Los_Angeles') # LA timezone
# convert to UTC timezone aware datetime
d = utc.localize(d)
>>> datetime.datetime(2016, 11, 5, 16, 43, 45, tzinfo=<UTC>)
# convert to 'America/Los_Angeles' timezone: DON'T DO THIS
d = d.astimezone(pst)
>>> datetime.datetime(2016, 11, 5, 9, 43, 45,
tzinfo=<DstTzInfo 'America/Los_Angeles' PDT-1 day, 17:00:00 DST>)
# we are in Pacific Daylight Time PDT
# add 1 day to LA local date: DON'T DO THAT
d = d + datetime.timedelta(days=1)
>>> datetime.datetime(2016, 11, 6, 9, 43, 45,
tzinfo=<DstTzInfo 'America/Los_Angeles' PDT-1 day, 17:00:00 DST>)
# Daylight Saving is NOT respected, we are still in PDT time, not PST
Conclusion:
datetime.timedelta() DOES NOT account for daylight saving.
Do your time add/subtract in UTC timezone ALWAYS.
Cast to local time only for output / display.

The pytz website says:
Unfortunately using the tzinfo argument of the standard datetime
constructors ‘’does not work’’ with pytz for many timezones.
So you should not use datetime.now(la). I don't know the specifics, but some timezones operate on more exotic rules then we are used to, and python's datetime code can't handle them. By using pytz's code, they should be handled correctly since that's pytz's purpose. It may also have issues for the times that occur twice thanks to jumping times in daylight savings.
As for the second question, that's exactly what the documentation shows, so you should be good.

pytz: Why do these different methods give different UTC offsets?

When creating a datetime object in a specific time zone using pytz I get a different UTC offset depending on whether I use datetime.datetime() or datetime.datetime.now().
now() seems to give the correct UTC offset for the time zone, datetime() gives an offset that I don't recognise.
Why are they different? What is the significance of the offset that datetime() assigns?
Here's my code:
import datetime
import pytz
la_paz = pytz.timezone('America/La_Paz')
a = datetime.datetime.now(la_paz)
print a, a.utcoffset()
# 2011-03-22 05:30:13-04:00 -1 day, 20:00:00
# -4 hours is the correct UTC offset for La Paz
b = datetime.datetime(2011, 03, 22, 5, 30, tzinfo=la_paz)
print b, b.utcoffset()
# 2011-03-22 05:30:00-04:33 -1 day, 19:27:00
# What is the significance of -4:33?

It seems that datetime() will use the first recorded timezone for the region by default, and in many cases (like in La Paz) this is old and no longer valid.
The datetime must instead be created naive and then localised like so:
b = la_paz.localize(datetime.datetime(2011, 03, 22, 5, 30))
print b, b.utcoffset()
now() appears to do the localization automatically.

From the pytz documentation:
This library only supports two ways of building a localized time. The first is to use the localize() method provided by the pytz library. This is used to localize a naive datetime (datetime with no timezone information):
>>> loc_dt = eastern.localize(datetime(2002, 10, 27, 6, 0, 0))
>>> print loc_dt.strftime(fmt)
2002-10-27 06:00:00 EST-0500
The second way of building a localized time is by converting an existing localized time using the standard astimezone() method:
>>> ams_dt = loc_dt.astimezone(amsterdam)
>>> ams_dt.strftime(fmt)
'2002-10-27 12:00:00 CET+0100'
Or put another way:
b = datetime.datetime(2011, 03, 22, 5, 30, tzinfo=la_paz)
Is not supported by pytz