I have nested json file which has time zone which is in UTC format I am capturing that and putting it into a column and then trying to convert that to cst by creating a column for CST but it is not converting can anybody help am posting the code below
def extract_json_data(fpath):
print("Extracting " + fpath)
f = open(fpath, 'r')
json_data = json.loads(f.read())
data = json_data['data']
dt = datetime.datetime.strptime(json_data['time'], "%Y-%m-%dT%H:%M:%SZ")
dt_cst = dt.astimezone(timezone('US/Central'))
_ = [row.update({'time_UTC': dt.strftime("%Y-%m-%dT%H:%M:%SZ"),
'time_CST': dt_cst.strftime("%Y-%m-%dT%H:%M:%S CST")}) for row in data]
Use a format string to parse the timezone, so that the datetime object you work with is timezone-aware:
from datetime import datetime
# the string actually has timezone information: Z (UTC)
timestring = "2019-01-01T00:00:00Z"
# wrong:
dt = datetime.strptime(timestring, "%Y-%m-%dT%H:%M:%SZ")
# dt is naive:
# datetime.datetime(2019, 1, 1, 0, 0)
# right:
dt = datetime.strptime(timestring, "%Y-%m-%dT%H:%M:%S%z")
# dt now knows it's in UTC:
# datetime.datetime(2019, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
Now you can change the time of your datetime object to a different timezone:
import pytz
tz = pytz.timezone('US/Central')
dt_cst = dt.astimezone(tz)
# datetime.datetime(2018, 12, 31, 18, 0, tzinfo=<DstTzInfo 'US/Central' CST-1 day, 18:00:00 STD>)
A more convenient solution would be to skip pytz and use dateutil instead:
import dateutil
timestring = "2019-01-01T00:00:00Z"
dt = dateutil.parser.parse(timestring)
# dt
# datetime.datetime(2019, 1, 1, 0, 0, tzinfo=tzutc())
Here's a way to do that:
import datetime
from dateutil import tz
# create a time-zone object representing UTC.
from_zone = tz.gettz('UTC')
# Create another time zone object, representing the target time zone.
# note that according to the tz package documentation
# (https://dateutil.readthedocs.io/en/stable/tz.html#dateutil.tz.gettz),
# they have Windows-specific time-zone names support.
to_zone = tz.gettz('America/Chicago')
# This is just a sample dictionary, so I cam extract the 'time'
# field like you do in your code. It's really not needed here.
json_data = {'time': "2020-05-16T08:17:42Z"} # an example for a datetime
# Create a datetime object, representing the UTC time.
utc = datetime.datetime.strptime(json_data['time'], "%Y-%m-%dT%H:%M:%SZ")
# now replace the timezone field of the newly created datetime object,
# so it would be UTC.
utc = utc.replace(tzinfo=from_zone)
# Convert time zone from UTC to central
central = utc.astimezone(to_zone)
print(central)
you'll get:
2020-05-16 03:17:42-05:00
Related
I want to know if there's a daylight saving time change within the next hours. Thereby I recognized an unexpected dependency between datetime.timedelta, astimezone and daylight saving time. During the daylight saving time change the timedelta calculation seems no to work. Here's a simple example:
import datetime
import pytz
format = '%Y-%m-%d %H:%M:%S %Z%z'
local_tz = pytz.timezone('Europe/Berlin')
time_diff = datetime.timedelta(hours = 1)
print((datetime.datetime(2023, 2, 26, 1, 0, 0) + time_diff).astimezone(local_tz).strftime(format))
# => 2023-02-26 02:00:00 CET+0100
print((datetime.datetime(2023, 3, 26, 1, 0, 0) + time_diff).astimezone(local_tz).strftime(format))
# => 2023-03-26 01:00:00 CET+0100
I would expect that the last print would result "2023-03-26 02:00:00 CET+0100" or "2023-03-26 03:00:00 CEST+0200". Does anyone can explain this behaviour?
After different tries I found a solution by adding the time delta after adding the timezone to the timestamp.
print((datetime.datetime(2023, 3, 26, 1, 0, 0).astimezone(local_tz) + time_diff).strftime(format))
But I still don't understand the error in my first used code.
My versions:
- Python 3.10.2
- pytz 2022.7
See also this answer by Paul Ganssle - with native Python's timedelta combined with time zones, non-existing datetimes (like 2023-02-26 02:00:00 CET+0100) have to be expected.
Here's a slightly extended comparison for reference, + comments in the code. pytz is deprecated since the release of Python 3.9's zoneinfo.
from datetime import datetime, timedelta, timezone
from zoneinfo import ZoneInfo
import pandas as pd
import pytz
def absolute_add(dt: datetime, td: timedelta) -> datetime:
utc_in = dt.astimezone(timezone.utc) # Convert input to UTC
utc_out = utc_in + td # Do addition in UTC
civil_out = utc_out.astimezone(dt.tzinfo) # Back to original tzinfo
return civil_out
# -----
# in vanilla Python, we can create non-existent datetime...
# I) tz already set
t = datetime(2023, 3, 26, 1, tzinfo=ZoneInfo("Europe/Berlin"))
print(t + timedelta(hours=1))
# 2023-03-26 02:00:00+01:00
# this datetime should not exist in that time zone since there is a DST transtion,
# 1 am UTC+1 plus one hour gives 3 am UTC+2
# II) tz set after timedelta addition
print(datetime(2023, 3, 26, 1) + timedelta(hours=1))
# 2023-03-26 02:00:00
# this is ok since no tz specified
print((datetime(2023, 3, 26, 1) + timedelta(hours=1)).replace(tzinfo=ZoneInfo("Europe/Berlin")))
# 2023-03-26 02:00:00+01:00
# again, we have a non-existent datetime
print((datetime(2023, 3, 26, 1) + timedelta(hours=1)).astimezone(ZoneInfo("Europe/Berlin")))
# 2023-03-26 01:00:00+01:00
# also a bit confusing; 2 am would be non-existing, so the hour is "corrected"
# backwards before setting the time zone
# -----
# adding the timedelta in UTC as "absolute duration" works as expected:
print(absolute_add(t, timedelta(hours=1)))
# 2023-03-26 03:00:00+02:00
# with pytz timezones, you can normalize to correct the non-existent datetime:
tz = pytz.timezone("Europe/Berlin")
t = tz.localize(datetime(2023, 3, 26, 1))
print(tz.normalize(t + timedelta(hours=1)))
# 2023-03-26 03:00:00+02:00
# this is correctly implemented in pandas for instance:
t = pd.Timestamp(2023, 3, 26, 1).tz_localize("Europe/Berlin")
print(t + pd.Timedelta(hours=1))
# 2023-03-26 03:00:00+02:00
There is a web series starting on 2017-01-11 19:00 Warsaw time. I want to make a list of time zones for major cities to help people figure out when to tune in. How can I tell Python that the date variable is related to the time in Warsaw?
import datetime
from pytz import timezone
from pytz import common_timezones
# warsaw time
s = '2017-01-11 19:00:00.801000'
format = '%Y-%m-%d %H:%M:%S.%f'
date = datetime.datetime.strptime(s, format)
fmt = "%Y-%m-%d %H:%M:%S %Z%z"
warsaw_time = date
print(warsaw_time.strftime(fmt))
for zone in common_timezones:
print( zone + str(warsaw_time.astimezone(timezone(zone))) )
If I understand correctly, you are trying to set the date to Warsaw's local time (CET). Which you can do like this:
>>> warsaw = pytz.timezone("CET")
>>> dt = datetime.datetime(2017, 1, 11, 19, 0, 0, 0, warsaw)
>>> dt
datetime.datetime(2017, 1, 11, 19, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)
Ok, I need a way to get the timestamp for the current day but at a certain time.
So for example, I want the unix timestamp for today at 7:30PM - what would I do to get that value? In PHP it's possible with the strtotime() but I'm not sure how to do this in Python.
Edit: To clarify, I mean the current day not a statically written day. So if I ran this script tomorrow it would return the timestamp for 7:30PM tomorrow.
from datetime import datetime
now = datetime.utcnow() # Current time
then = datetime(1970,1,1) # 0 epoch time
ts = now - then
ts = ts.days * 24 * 3600 + ts.seconds
# alternatively, per Martijn Pieters
ts = int(ts.total_seconds())
you can use the time module :
from datetime import datetime
from time import mktime
# like said Ashoka
ts = datetime.strptime("2014-7-7 7:30","%Y-%m-%d %H:%M")
#you have now your datetime object
print mktime(ts.timetuple())
# print 1404711000.0
print int(mktime(ts.timetuple()))
# print 1404711000
be careful mktime don't care of time zone so if you want to have a UTC time zone and still use time , convert date before:
import pytz
fr = pytz.timezone('Europe/Paris')
#localize
ts = fr.localize(ts)
#timestamp in UTC
mktime(ts.astimezone(pytz.UTC).timetuple())
calendar.timegm method returns a timestamp out of passed time tuple:
import calendar
from datetime import datetime
d = datetime(year=2014, month=7, day=8, hour=7, minute=30)
calendar.timegm(d.utctimetuple())
# 1404804600
datetime.utcfromtimestamp(calendar.timegm(d.utctimetuple()))
# datetime.datetime(2014, 7, 8, 7, 30)
The important things are utctimetuple and utcfromtimestamp. You would certainly want a UTC timestamp, and not one in your local timezone.
import calendar
from datetime import datetime
from pytz import timezone, utc
tz = timezone('Europe/Warsaw')
aware = datetime(year=2014, month=7, day=8, hour=7, minute=30)
aware = tz.localize(aware)
# datetime.datetime(2014, 7, 8, 7, 30, tzinfo=<DstTzInfo 'Europe/Warsaw' CEST+2:00:00 DST>)
stamp = calendar.timegm(aware.utctimetuple())
# 1404797400
d = datetime.utcfromtimestamp(stamp)
# datetime.datetime(2014, 7, 8, 5, 30)
d = d.replace(tzinfo=utc)
d.astimezone(tz)
# datetime.datetime(2014, 7, 8, 7, 30, tzinfo=<DstTzInfo 'Europe/Warsaw' CEST+2:00:00 DST>)
In the code below, I am calculating now epoch and beginning of current day epoch.
import time
import pytz
from datetime import datetime
tz1 = pytz.timezone('CST6CDT')
utc = pytz.timezone('UTC')
now = pytz.UTC.localize(datetime.utcnow())
now_tz = now.astimezone(tz1)
print now_tz
print now_tz.strftime('%s')
begin_day = now_tz.replace(hour=0, minute=0, second=0)
print begin_day
print begin_day.strftime('%s')
print statements:
2012-08-28 13:52:21.595718-05:00
1346187141
2012-08-28 00:00:00.595718-05:00
1346137200
Converting epochs to timestamp with CDT timezone:
1346187141 - Aug 28 2012 15:52:21,
1346137200 - Aug 28 2012 02:00:00
I'd like the second epoch to be beginning of the day but it's 2 am. It looks like it is still using local timezone PST when converting to epoch.
What am I doing wrong ? or can this be done a different way?
Thanks!
To convert a datetime with timezone to epoch (POSIX timestamp):
from datetime import datetime
import pytz
tz = pytz.timezone('CST6CDT')
# a datetime with timezone
dt_with_tz = tz.localize(datetime(2012, 8, 28, 19, 33, 50), is_dst=None)
# get timestamp
ts = (dt_with_tz - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
# -> 1346200430.0
It is how datetime.timestamp method is implemented for timezone-aware datetime objects in Python 3.
To get "now epoch":
from datetime import datetime
now_epoch = (datetime.utcnow() - datetime(1970, 1, 1)).total_seconds()
Or (assuming time uses POSIX epoch):
import time
now_epoch = time.time()
Getting "beginning of current day epoch" is more complex because current day may be different in different timezones:
from datetime import datetime, time
import pytz
tz = pytz.timezone('CST6CDT')
# get current date in given timezone
today = datetime.now(tz).date()
# -> datetime.date(2013, 6, 22)
# get beginning of current day in given timezone as a datetime with timezone
midnight = tz.localize(datetime.combine(today, time(0, 0)), is_dst=None)
# -> datetime.datetime(2013, 6, 22, 0, 0, tzinfo=<DstTzInfo 'CST6CDT'...>)
# get timestamp
ts = (midnight - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
# -> 1371877200.0
See How do I get the UTC time of “midnight” for a given timezone?.
To get "beginning of current day epoch" assuming UTC date:
from datetime import datetime, date
# get current date in UTC
utc_date = datetime.utcnow().date()
# -> datetime.date(2013, 6, 23)
# get timestamp
ts = (utc_date - date(1970, 1, 1)).days * 86400
# -> 1371945600
See Converting datetime.date/datetime.datetime to UTC timestamp in Python.
NOTE: My answer is flat-out wrong. (I'd like to delete it, but am unable to do so until the accept flag is removed.)
Please see J.F.Sebastian's answer.
Here is code demonstrating a value of now_tz for which our two methods produce different results.
import calendar
import pytz
import datetime as dt
tz1 = pytz.timezone('US/Eastern')
utc = pytz.timezone('UTC')
now = utc.localize(dt.datetime(2002, 10, 28), is_dst=None)
now_tz = now.astimezone(tz1)
now_epoch = calendar.timegm(now_tz.utctimetuple())
begin_day = tz1.normalize(now_tz.replace(hour=0, minute=0, second=0))
midnight = tz1.localize(dt.datetime.combine(now_tz, dt.time(0, 0)), is_dst=None)
if begin_day != midnight:
print(begin_day)
# 2002-10-27 01:00:00-04:00 # my result -- is not midnight
print(midnight)
# 2002-10-27 00:00:00-04:00 # J.F.Sebastian's result is correct
(Original answer redacted)
the latest release of simple-date (version 0.2 on pypi) will manage the details for you:
>>> from simpledate import *
>>> now_utc = SimpleDate(tz='UTC')
>>> now_tz = now_utc.convert(tz='CST6CDT')
>>> begin_day = now_tz.replace(hour=0, minute=0, second=0, microsecond=0)
>>> now_utc.timestamp
1371950295.777453
>>> now_tz.timestamp
1371950295.777453
>>> begin_day.timestamp
1371877200.0
we can go backwards to check the timestamps (although it's clear above that switching timezone didn't change the epoch, while moving to start of day did):
>>> SimpleDate(1371877200.0, tz='CST6CDT')
SimpleDate('2013-06-22 00:00:00.000000 CDT', tz='CST6CDT')
>>> SimpleDate(1371877200.0, tz='UTC')
SimpleDate('2013-06-22 05:00:00.000000 UTC')
How to convert a string in the format "%d/%m/%Y" to timestamp?
"01/12/2011" -> 1322697600
>>> import time
>>> import datetime
>>> s = "01/12/2011"
>>> time.mktime(datetime.datetime.strptime(s, "%d/%m/%Y").timetuple())
1322697600.0
I use ciso8601, which is 62x faster than datetime's strptime.
t = "01/12/2011"
ts = ciso8601.parse_datetime(t)
# to get time in seconds:
time.mktime(ts.timetuple())
You can learn more here.
>>> int(datetime.datetime.strptime('01/12/2011', '%d/%m/%Y').strftime("%s"))
1322683200
To convert the string into a date object:
from datetime import date, datetime
date_string = "01/12/2011"
date_object = date(*map(int, reversed(date_string.split("/"))))
assert date_object == datetime.strptime(date_string, "%d/%m/%Y").date()
The way to convert the date object into POSIX timestamp depends on timezone. From Converting datetime.date to UTC timestamp in Python:
date object represents midnight in UTC
import calendar
timestamp1 = calendar.timegm(utc_date.timetuple())
timestamp2 = (utc_date.toordinal() - date(1970, 1, 1).toordinal()) * 24*60*60
assert timestamp1 == timestamp2
date object represents midnight in local time
import time
timestamp3 = time.mktime(local_date.timetuple())
assert timestamp3 != timestamp1 or (time.gmtime() == time.localtime())
The timestamps are different unless midnight in UTC and in local time is the same time instance.
Simply use datetime.datetime.strptime:
import datetime
stime = "01/12/2011"
print(datetime.datetime.strptime(stime, "%d/%m/%Y").timestamp())
Result:
1322697600
To use UTC instead of the local timezone use .replace:
datetime.datetime.strptime(stime, "%d/%m/%Y").replace(tzinfo=datetime.timezone.utc).timestamp()
The answer depends also on your input date timezone. If your date is a local date, then you can use mktime() like katrielalex said - only I don't see why he used datetime instead of this shorter version:
>>> time.mktime(time.strptime('01/12/2011', "%d/%m/%Y"))
1322694000.0
But observe that my result is different than his, as I am probably in a different TZ (and the result is timezone-free UNIX timestamp)
Now if the input date is already in UTC, than I believe the right solution is:
>>> calendar.timegm(time.strptime('01/12/2011', '%d/%m/%Y'))
1322697600
I would give a answer for beginners (like me):
You have the date string "01/12/2011". Then it can be written by the format "%d/%m/%Y". If you want to format to another format like "July 9, 2015", here a good cheatsheet.
Import the datetime library.
Use the datetime.datetime class to handle date and time combinations.
Use the strptime method to convert a string datetime to a object datetime.
Finally, use the timestamp method to get the Unix epoch time as a float. So,
import datetime
print( int( datetime.datetime.strptime( "01/12/2011","%d/%m/%Y" ).timestamp() ) )
# prints 1322712000
A lot of these answers don't bother to consider that the date is naive to begin with
To be correct, you need to make the naive date a timezone aware datetime first
import datetime
import pytz
# naive datetime
d = datetime.datetime.strptime('01/12/2011', '%d/%m/%Y')
>>> datetime.datetime(2011, 12, 1, 0, 0)
# add proper timezone
pst = pytz.timezone('America/Los_Angeles')
d = pst.localize(d)
>>> datetime.datetime(2011, 12, 1, 0, 0,
tzinfo=<DstTzInfo 'America/Los_Angeles' PST-1 day, 16:00:00 STD>)
# convert to UTC timezone
utc = pytz.UTC
d = d.astimezone(utc)
>>> datetime.datetime(2011, 12, 1, 8, 0, tzinfo=<UTC>)
# epoch is the beginning of time in the UTC timestamp world
epoch = datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.UTC)
>>> datetime.datetime(1970, 1, 1, 0, 0, tzinfo=<UTC>)
# get the total second difference
ts = (d - epoch).total_seconds()
>>> 1322726400.0
Also:
Be careful, using pytz for tzinfo in a datetime.datetime DOESN'T WORK for many timezones. See datetime with pytz timezone. Different offset depending on how tzinfo is set
# Don't do this:
d = datetime.datetime(2011, 12, 1,0,0,0, tzinfo=pytz.timezone('America/Los_Angeles'))
>>> datetime.datetime(2011, 1, 12, 0, 0,
tzinfo=<DstTzInfo 'America/Los_Angeles' LMT-1 day, 16:07:00 STD>)
# tzinfo in not PST but LMT here, with a 7min offset !!!
# when converting to UTC:
d = d.astimezone(pytz.UTC)
>>> datetime.datetime(2011, 1, 12, 7, 53, tzinfo=<UTC>)
# you end up with an offset
https://en.wikipedia.org/wiki/Local_mean_time
First you must the strptime class to convert the string to a struct_time format.
Then just use mktime from there to get your float.
I would suggest dateutil:
import dateutil.parser
dateutil.parser.parse("01/12/2011", dayfirst=True).timestamp()
Seems to be quite efficient:
import datetime
day, month, year = '01/12/2011'.split('/')
datetime.datetime(int(year), int(month), int(day)).timestamp()
1.61 µs ± 120 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
you can convert to isoformat
my_date = '2020/08/08'
my_date = my_date.replace('/','-') # just to adapte to your question
date_timestamp = datetime.datetime.fromisoformat(my_date).timestamp()
You can refer this following link for using strptime function from datetime.datetime, to convert date from any format along with time zone.
https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior
just use datetime.timestamp(your datetime instanse), datetime instance contains the timezone infomation, so the timestamp will be a standard utc timestamp. if you transform the datetime to timetuple, it will lose it's timezone, so the result will be error.
if you want to provide an interface, you should write like this:
int(datetime.timestamp(time_instance)) * 1000
A simple function to get UNIX Epoch time.
NOTE: This function assumes the input date time is in UTC format (Refer to comments here).
def utctimestamp(ts: str, DATETIME_FORMAT: str = "%d/%m/%Y"):
import datetime, calendar
ts = datetime.datetime.utcnow() if ts is None else datetime.datetime.strptime(ts, DATETIME_FORMAT)
return calendar.timegm(ts.utctimetuple())
Usage:
>>> utctimestamp("01/12/2011")
1322697600
>>> utctimestamp("2011-12-01", "%Y-%m-%d")
1322697600
You can go both directions, unix epoch <==> datetime :
import datetime
import time
the_date = datetime.datetime.fromtimestamp( 1639763585 )
unix_time = time.mktime(the_date.timetuple())
assert ( the_date == datetime.datetime.fromtimestamp(unix_time) ) & \
( time.mktime(the_date.timetuple()) == unix_time )