I am struggling to understand how to get time series in local time (with DST) from excel file into a pandas timeseries in UTC. I’ve tried various combinations of .localize(), .replace(), .astimezone(), .to_datetime() , all without luck and I feel like I am getting an inconsistent results in situations that should (at least in my mind) be returning the same thing. The example that gets to the heart of the dilemma follows:. I would be grateful if someone could explain why these two sections of code produce different answers. The first of these sections accomplishes what I would like. It seems like the second should do the same, but it doesn’t.
In[1]: UTCtz=timezone("UTC")
In[2]: localtz=timezone("US/Central")
In[3]: dt1=datetime.datetime(2016,3,13,1,0,0,0)
In[4]: dt2=datetime.datetime(2016,3,13,3,0,0,0)
In[5]: dt3=localtz.localize(dt1)
In[6]: dt3
Out[6]: datetime.datetime(2016, 3, 13, 1, 0, tzinfo=<DstTzInfo 'US/Central' CST-1 day, 18:00:00 STD>)
In[7]: dt4=localtz.localize(dt2)
In[8]: dt4
Out[8]: datetime.datetime(2016, 3, 13, 3, 0, tzinfo=<DstTzInfo 'US/Central' CDT-1 day, 19:00:00 DST>)
In[9]: dt3.astimezone(UTCtz)
Out[9]: datetime.datetime(2016, 3, 13, 7, 0, tzinfo=<UTC>)
In[10]: dt4.astimezone(UTCtz)
Out[10]: datetime.datetime(2016, 3, 13, 8, 0, tzinfo=<UTC>)
dt1 and dt2 straddle the DST switch and the conversion to UTC is what I was expecting.
Alternatively, here is what I get from the series that I’ve read from excel into a dataframe (column “timeindex”).
In[1]: df.iloc[2]["timeindex"]
Out[1]: Timestamp('2016-03-13 01:00:00')
In[2]: df.iloc[3]["timeindex"]
Out[2]: Timestamp('2016-03-13 03:00:00')
In[3]: localtz.localize(df.iloc[2]["timeindex"])
Out[3]: Timestamp('2016-03-13 01:00:00-0600', tz='US/Central')
In[4]: localtz.localize(df.iloc[3]["timeindex"])
Out[4]: Timestamp('2016-03-13 04:00:00-0500', tz='US/Central')
The same function calls as before but the utc gap between times increases by an hour !
What is going on here ?
Related
When I tried to convert a datetime object into timestamp and convert it back the result if off by 53 minutes.
timestamp = datetime.datetime(2022, 5, 3, 18, 0, 0, 0, tzinfo=pytz.timezone('US/Pacific')).timestamp()
# timetamp = 1651629180
datetime.datetime.fromtimestamp(timestamp, tz=pytz.timezone('US/Pacific'))
# formmated_timestamp = datetime.datetime(2022, 5, 3, 18, 53, tzinfo=<DstTzInfo 'US/Pacific' PDT-1 day, 17:00:00 DST>)
somehow the result is off by 53 minutes. This is true for all other values as well
The issue seems to be at the .timestamp() part
From the pytz docs:
This library only supports two ways of building a localized time. The first is to use the localize() method provided by the pytz library. This is used to localize a naive datetime (datetime with no timezone information):
>>> loc_dt = eastern.localize(datetime(2002, 10, 27, 6, 0, 0))
>>> print(loc_dt.strftime(fmt))
2002-10-27 06:00:00 EST-0500
The second way of building a localized time is by converting an existing localized time using the standard astimezone() method:
>>> ams_dt = loc_dt.astimezone(amsterdam)
>>> ams_dt.strftime(fmt)
'2002-10-27 12:00:00 CET+0100'
Unfortunately using the tzinfo argument of the standard datetime constructors ''does not work'' with pytz for many timezones.
>>> datetime(2002, 10, 27, 12, 0, 0, tzinfo=amsterdam).strftime(fmt)
'2002-10-27 12:00:00 LMT+0020'
I am currently trying to convert times from UTC and the problem that i am having is that the offsets seem to be backwards. As you can see when i convert the UTC to EST, it shows an offset of -4:56 yet when i print the time, it seems to add 4:56 as opposed to the way it should be. I would really like to be able to convert a UTC time to any other timezone and have it display the local time there without the offset so the UTC here would be converted to something along the lines of 2019-03-06 9:12 EST.
>>> example.created
datetime.datetime(2019, 3, 6, 14, 8, 49, 841881, tzinfo=<UTC>)
>>> original_utc = example.created
>>> original_utc
datetime.datetime(2019, 3, 6, 14, 8, 49, 841881, tzinfo=<UTC>)
>>> conv_est = original_utc.replace(tzinfo=pytz.timezone('US/Eastern'))
>>> conv_est
datetime.datetime(2019, 3, 6, 14, 8, 49, 841881, tzinfo=<DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>)
>>> print(conv_est)
2019-03-06 14:08:49.841881-04:56
>>> print(conv_est.astimezone())
2019-03-06 19:04:49.841881+00:00
I suspect that you misunderstood the method .astimezone().
Your original datetime is in UTC
>>> example.created
datetime.datetime(2019, 3, 6, 14, 8, 49, 841881, tzinfo=<UTC>)
Then you changed the timezone info for the variable conv_est, and indeed it works as designed:
>>> conv_est = original_utc.replace(tzinfo=pytz.timezone('US/Eastern'))
>>> conv_est
datetime.datetime(2019, 3, 6, 14, 8, 49, 841881, tzinfo=<DstTzInfo 'US/Eastern' LMT-1 day, 19:04:00 STD>)
If you print this variable, it shows the correct info
>>> print(conv_est)
2019-03-06 14:08:49.841881-04:56
But when you call .astimezone() without any argument, then the return value is a datetime object in UTC zone; that means the method is also working as designed, returning the same point in time but expressed as localtime in UTC (It will be 7PM/19hs in UTC when it is 2PM/14hs in US/Eastern).
>>> print(conv_est.astimezone())
2019-03-06 19:04:49.841881+00:00
You can test that yourself by calculating the difference (which will be 0):
>>> conv_est == conv_est.astimezone()
True
>>> conv_est - conv_est.astimezone()
datetime.timedelta(0)
How do you get a valid timedelta instance when differencing datetimes with different timezones in Python? I'm finding the timedelta is always 0 if the timezones are different.
>>> from dateutil.parser import parse
>>> dt0=parse('2017-02-06 18:14:32-05:00')
>>> dt0
datetime.datetime(2017, 2, 6, 18, 14, 32, tzinfo=tzoffset(None, -18000))
>>> dt1=parse('2017-02-06 23:14:32+00:00')
>>> dt1
datetime.datetime(2017, 2, 6, 23, 02, 12, tzinfo=tzutc())
>>> (dt1-dt0).total_seconds()
0.0
This doesn't make any sense to me. I would have thought that Python's datetime class would be smart enough to normalize both values to UTC internally, and then return a timedelta based on those values. Or throw an exception. Instead it returns 0, implying both datetimes are equal, which clearly they're not. What am I doing wrong here?
You are confused about what the timezone means; the two times you gave are identical, so of course their difference is zero. I can duplicate your results, except that I don't have the discrepancy between the second string and second datetime that you have:
>>> from dateutil.parser import parse
>>> dt0=parse('2017-02-06 18:14:32-05:00')
>>> dt0
datetime.datetime(2017, 2, 6, 18, 14, 32, tzinfo=tzoffset(None, -18000))
>>> dt1=parse('2017-02-06 23:14:32+00:00')
>>> dt1
datetime.datetime(2017, 2, 6, 23, 14, 32, tzinfo=tzutc())
>>> (dt1-dt0).total_seconds()
0.0
But watch what happens when I convert dt0 to UTC. The time gets adjusted by the 5 hour timezone difference, and it becomes identical to the second.
>>> dt0.astimezone(dt1.tzinfo)
datetime.datetime(2017, 2, 6, 23, 14, 32, tzinfo=tzutc())
This question already has answers here:
Converting between datetime, Timestamp and datetime64
(14 answers)
Closed 7 years ago.
I basically face the same problem posted here:Converting between datetime, Timestamp and datetime64
but I couldn't find satisfying answer from it, my question how to extract datetime from numpy.datetime64 type:
if I try:
np.datetime64('2012-06-18T02:00:05.453000000-0400').astype(datetime.datetime)
it gave me:
1339999205453000000L
my current solution is convert datetime64 into a string and then turn to datetime again. but it seems quite a silly method.
Borrowing from
Converting between datetime, Timestamp and datetime64
In [220]: x
Out[220]: numpy.datetime64('2012-06-17T23:00:05.453000000-0700')
In [221]: datetime.datetime.utcfromtimestamp(x.tolist()/1e9)
Out[221]: datetime.datetime(2012, 6, 18, 6, 0, 5, 452999)
Accounting for timezones I think that's right. Looks rather clunky though.
Using int() is more explicit (I think) than tolist()):
In [294]: datetime.datetime.utcfromtimestamp(int(x)/1e9)
Out[294]: datetime.datetime(2012, 6, 18, 6, 0, 5, 452999)
or to get datetime in local:
In [295]: datetime.datetime.fromtimestamp(x.astype('O')/1e9)
But in the test_datetime.py file
https://github.com/numpy/numpy/blob/master/numpy/core/tests/test_datetime.py
I find some other options - first convert the general datetime64 to one of the format that specifies units:
In [296]: x.astype('M8[D]').astype('O')
Out[296]: datetime.date(2012, 6, 18)
In [297]: x.astype('M8[ms]').astype('O')
Out[297]: datetime.datetime(2012, 6, 18, 6, 0, 5, 453000)
This works for arrays:
In [303]: np.array([[x,x],[x,x]],dtype='M8[ms]').astype('O')[0,1]
Out[303]: datetime.datetime(2012, 6, 18, 6, 0, 5, 453000)
Note that Timestamp IS a sub-class of datetime.datetime so the [4] will generally work
In [4]: pd.Timestamp(np.datetime64('2012-06-18T02:00:05.453000000-0400'))
Out[4]: Timestamp('2012-06-18 06:00:05.453000')
In [5]: pd.Timestamp(np.datetime64('2012-06-18T02:00:05.453000000-0400')).to_pydatetime()
Out[5]: datetime.datetime(2012, 6, 18, 6, 0, 5, 453000)
On 2013 Jun 1 I expect the "PST8PDT" timezone to behave like GMT+7, as it is daylight savings in that timezone. However, it behaves like GMT+8:
>>> import pytz, datetime
>>> Pacific = pytz.timezone("PST8PDT")
>>> datetime.datetime(2013, 6, 1, 12, tzinfo=Pacific).astimezone(pytz.utc)
datetime.datetime(2013, 6, 1, 20, 0, tzinfo=<UTC>)
In contrast, on 2013 Jan 1 it behaves (correctly) like GMT+8:
>>> datetime.datetime(2013, 1, 1, 12, tzinfo=Pacific).astimezone(pytz.utc)
datetime.datetime(2013, 1, 1, 20, 0, tzinfo=<UTC>)
What am I doing wrong? Thanks in advance!
You can't assign the timezone in the datetime constructor, because it doesn't give the timezone object a chance to adjust for daylight savings - the date isn't accessible to it. This causes even more problems for certain parts of the world, where the name and offset of the timezone have changed over the years.
From the pytz documentation:
Unfortunately using the tzinfo argument of the standard datetime constructors ‘’does not work’’ with pytz for many timezones.
Use the localize method with a naive datetime instead.
>>> Pacific.localize(datetime.datetime(2013, 6, 1, 12)).astimezone(pytz.utc)
datetime.datetime(2013, 6, 1, 19, 0, tzinfo=<UTC>)
>>> Pacific.localize(datetime.datetime(2013, 1, 1, 12)).astimezone(pytz.utc)
datetime.datetime(2013, 1, 1, 20, 0, tzinfo=<UTC>)