I am using datetime.strptime() to convert a string containing time and date from a sensor into a datetime object.
The code sometimes fails. Minimal example:
datetime.strptime('1/9/2021 24:01:53', '%d/%m/%Y %H:%M:%S')
Output error:
ValueError: time data '1/9/2021 24:01:53' does not match format '%d/%m/%Y %H:%M:%S'
I am guessing this has to do with the fact that the time is more than 23:59:59 - which seems to me a non-realistic time (I would think that 1/9/2021 24:01:53 could potentially be 2/9/2021 00:01:53 - a time format which I have never seen).
Is this a non-standard way of representing time or possibly a hardware/software issue with the sensor acquisition system? If it is a different way of representing time, how can I convert it to a standard datetime object?
Kind regards,
D.F.
If the hour exceeds 23 in a variable representing time, a good option is to create a timedelta from it, which you can then add to a datetime object. For given example that might look like
from datetime import datetime, timedelta
def custom_todatetime(s):
"""
split date/time string formatted as 'DD/MM/YYYY hh:mm:ss' into date and time parts.
parse date part to datetime and add time part as timedelta.
"""
parts = s.split(' ')
seconds = sum(int(x) * 60 ** i for i, x in enumerate(reversed(parts[1].split(':'))))
return datetime.strptime(parts[0], "%d/%m/%Y") + timedelta(seconds=seconds)
s = '1/9/2021 24:01:53'
print(custom_todatetime(s))
# 2021-09-02 00:01:53
Note: conversion of hh:mm:ss to seconds taken from here - give a +1 there if helpful.
I can convert from an UTC timestamp e.g. 1417392000 to a local datetime object including daylight saving time, but when I try and convert the local datetime object to a "local timestamp" then I get the same UTC timestamp as before.
Am I thinking in circles and they are the same? I was supposed to save "local timestamp" from incoming UTC timestamp.
This is my code
print("UTC timestamp %d" % hour[0])
day = self.get_day(hour)
month = self.get_month(hour)
year = self.get_year(hour)
tz = pytz.timezone('Europe/Stockholm')
utc_dt = datetime.utcfromtimestamp(int(hour[0])).replace(tzinfo=pytz.utc)
print("UTC datetime %s" % utc_dt)
dt = tz.normalize(utc_dt.astimezone(tz))
print("STO datetime %s" % dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
print("STO ts %d" % int(time.mktime(dt.timetuple())))
print("STO timestamp %d" % utc_dt.astimezone(tz).timestamp())
day = int(dt.strftime('%d'))
month = int(dt.strftime('%m'))
year = int(dt.strftime('%Y'))
Output
UTC timestamp 1417395600
UTC datetime 2014-12-01 01:00:00+00:00
STO datetime 2014-12-01 02:00:00 CET+0100
STO ts 1417395600
STO timestamp 1417395600
All "timestamps" (i.e. integer representations of the time) are the same. Is it possible to make a "local timestamp" ? The data type is supposed to be a timestamp that is a number and in local time.
As per Wikipedia
Unix time (also known as POSIX time or Epoch time) is a system for
describing instants in time, defined as the number of seconds that
have elapsed since 00:00:00 Coordinated Universal Time (UTC),
Thursday, 1 January 1970
So regardless of what timezone you're on, the epoch will always be calculated in the UTC timezone.
For display purposes, you can convert the timestamp to a local time, but as otherwise the internal representation of the epoch will always be in the UTC timezone
Formal timestamps
The Unix timestamp typically refers to the number of seconds since the epoch in UTC. This value is invariant to timezone. It allows global event ordering but loses timezone information.
Preserving timezone
To preserve timezone information, a standardized format is RFC3339, Date and Time on the Internet: Timestamps. This is just a standardized formatting that encodes date+time+timezone. Some examples:
1985-04-12T23:20:50.52Z
1996-12-19T16:39:57-08:00
1990-12-31T23:59:60Z
1990-12-31T15:59:60-08:00
Normalizing for timezone without preservation of timezone
However, it may depend on your requirements. I once wanted to record some events relative to local-time-of-day and did not mind losing timezone information. I normalized the timestamp with respect to 1970-01-01T00:00:00 in the local timezone. I am a little sheepish about this now as I think it may too easily cause confusion.
import time
# Number of seconds since 1970-01-01T00:00+LTZ (current timezone).
# unix timestamp - timezone offset in seconds
timestamp_localized = time.time() - time.mktime(time.gmtime(0))
However this syntax can be simplified, perhaps at the loss of clarity, by noticing that Python has some localtime and UTC specific functions.
import time
import calendar
# Number of seconds since 1970-01-01T00:00+LTZ (current timezone).
# Interpret local date and time as if it were UTC
timestamp_localized = calendar.timegm(time.localtime())
The difference between these two is that calendar conversion truncates to the second while the difference calculation includes a fractional second.
I am trying to write a function that will convert a string date/time from local time to UTC in Python.
According to this question, you can use time.tzname to get some forms of the local timezone, but I have not found a way to use this in any of the datetime conversion methods. For example, this article shows there are a couple of things you can do with pytz and datetime to convert times, but all of them have timezones that are hardcoded in and are of different formats than what time.tznamereturns.
Currently I have the following code to translate a string-formatted time into milliseconds (Unix epoch):
local_time = time.strptime(datetime_str, "%m/%d/%Y %H:%M:%S") # expects UTC, but I want this to be local
dt = datetime.datetime(*local_time[:6])
ms = int((dt - datetime.datetime.utcfromtimestamp(0)).total_seconds() * 1000)
However, this is expecting the time to be input as UTC. Is there a way to convert the string formatted time as if it were in the local timezone? Thanks.
Essentially, I want to be able to do what this answer does, but instead of hard-coding in "America/Los_Angeles", I want to be able to dynamically specify the local timezone.
If I understand your question correctly you want this :
from time import strftime,gmtime,mktime,strptime
# you can pass any time you want
strftime("%Y-%m-%d %H:%M:%S", gmtime(mktime(strptime("Thu, 30 Jun 2016 03:12:40", "%a, %d %b %Y %H:%M:%S"))))
# and here for real time
strftime("%Y-%m-%d %H:%M:%S", gmtime(mktime(strptime(strftime("%a, %d %b %Y %H:%M:%S"), "%a, %d %b %Y %H:%M:%S"))))
make a time structure from a timetuple then use the structure to create a utc time
from datetime import datetime
def local_to_utc(local_st):
time_struct = time.mktime(local_st)
utc_st = datetime.utcfromtimestamp(time_struct)
return utc_st
d=datetime(2016,6,30,3,12,40,0)
timeTuple = d.timetuple()
print(local_to_utc(timeTuple))
output:
2016-06-30 09:12:40
I am looking to analyze traffic flow with relation to weather data. The traffic data has a UNIX timestamp (aka epoch), but I am running into trouble with converting the timestamp (in the weather data) to epoch. The problem is that I am in Norway and the UTC timestamp in the weather data isn't in the same timezone as me (GMT+1).
My initial approach
I first tried converting it into epoch and treating the data as if it was in the GMT+1 timezone. Then I compensated by subtracting the difference in number of seconds between UTC and GMT+1.
Problems with the approach
I realize first of all that this approach is very primitive and not very elegant (in fact probably it is at best an ugly hack). However, the biggest problem here is that the difference between UTC and GMT+1 is not constant (due to daylight savings).
Question
Is there any reliable way of turning UTC time to a UNIX time stamp in python (taking into account that my machine is in GMT+1)? The timestamp is in the following format:
Y-m-d HH:MM:SS
Edit:
Tried rmunns' solution:
def convert_UTC_to_epoch(timestamp):
tz_UTC = pytz.timezone('UTC')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime.strptime(timestamp, time_format)
aware_timestamp = tz_UTC.localize(naive_timestamp)
epoch = aware_timestamp.strftime("%s")
return (int) (epoch)
This does not work properly as evidenced below:
#Current time at time of the edit is 15:55:00 UTC on June 9th 2014.
>>> diff = time.time() - convert_UTC_to_epoch("2014-06-09 15:55:00")
>>> diff
3663.25887799263
>>> #This is about an hour off.
The solution was to use the calendar module (inspired from here)
>>>#Quick and dirty demo
>>>print calendar.timegm(datetime.datetime.utcnow().utctimetuple()) - time.time()
>>>-0.6182510852813721
And here is the conversion function:
import calendar, datetime, time
#Timestamp is a datetime object in UTC time
def UTC_time_to_epoch(timestamp):
epoch = calendar.timegm(timestamp.utctimetuple())
return epoch
An alternative, datetime has it's own .strptime() method.
http://en.wikipedia.org/wiki/Unix_time
The Unix epoch is the time 00:00:00 UTC on 1 January 1970 (or 1970-01-01T00:00:00Z ISO 8601).
import datetime
unix_epoch = datetime.datetime(1970, 1, 1)
log_dt = datetime.datetime.strptime("14-05-07 12:14:16", "%y-%m-%d %H:%M:%S")
seconds_from_epoch = (log_dt - unix_epoch).total_seconds()
>>> 1399490056.0
The pytz module will probably help you. It allows you to write code like:
import pytz
import datetime
tz_oslo = pytz.timezone('Europe/Oslo')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime(2014, 6, 4, 12, 34, 56)
# Or:
naive_timestamp = datetime.datetime.strptime("2014-06-04 12:34:56", time_format)
aware_timestamp = tz_oslo.localize(naive_timestamp)
print(aware_timestamp.strftime(time_format + " %Z%z"))
This should print "2014-06-04 14:34:56 CEST+0200".
Do note the following from the pytz manual:
The preferred way of dealing with times is to always work in UTC, converting to localtime only when generating output to be read by humans.
So keep that in mind as you write your code: do the conversion to local time once and once only, and you'll have a much easier time doing, say, comparisons between two timestamps correctly.
Update: Here are a couple of videos you may find useful:
What you need to know about datetimes, a PyCon 2012 presentation by Taavi Burns (30 minutes)
Drive-in Double Header: Datetimes and Log Analysis, a two-part presentation. (Caution: annoying buzz in the video, but I couldn't find a copy with better sound). The first part is the "What you need to know about datetimes" presentation I linked just above, and the second part has some practical tips for parsing log files and doing useful things with them. (50 minutes)
Update 2: The convert_UTC_to_epoch() function you mention in your updated question (which I've reproduced below) is returning local time, not UTC:
def convert_UTC_to_epoch(timestamp):
tz_UTC = pytz.timezone('UTC')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime.strptime(timestamp, time_format)
aware_timestamp = tz_UTC.localize(naive_timestamp)
epoch = aware_timestamp.strftime("%s")
return (int) (epoch)
The problem is that you're using strftime("%s"), which is undocumented and is returning the wrong result. Python doesn't support the %s parameter, but it appears to work because it gets passed to your system's strftime() function, which does support the %s parameter -- but it returns local time! You're taking a UTC timestamp and parsing it as local time, which is why it's an hour off. (The mystery is why it isn't two hours off -- isn't Norway in daylight savings time right now? Shouldn't you be at UTC+2?)
As you can see from the interactive Python session below, I'm in the UTC+7 timezone and your convert_UTC_to_epoch() function is seven hours off for me.
# Current time is 02:42 UTC on June 10th 2014, 09:42 local time
>>> time.timezone
-25200
>>> time.time() - convert_UTC_to_epoch("2014-06-10 02:42:00")
25204.16531395912
>>> time.time() + time.timezone - convert_UTC_to_epoch("2014-06-10 02:42:00")
6.813306093215942
The strftime("%s") call is interpreting 02:42 on June 10th as being in local time, which would be 19:42 UTC on June 9th. Subtracting 19:42 UTC on June 9th from 02:42 UTC June 10th (which is what time.time() returns) gives a difference of seven hours. See Convert python datetime to epoch with strftime for more details on why you should never use strftime("%s").
(By the way, if you saw what I had previously written under the heading "Update 2", where I claimed that time.time() was returning local time, ignore that -- I got it wrong. I was fooled at first by the strftime("%s") bug just like you were.)
You can use the time and datetime modules:
import time, datetime
date = "14-05-07 12:14:16" #Change to whatever date you want
date = time.strptime(date, "%y-%m-%d %H:%M:%S")
epoch = datetime.datetime.fromtimestamp(time.mktime(date)).strftime('%s')
This runs as:
>>> import time, datetime
>>> date = "14-05-07 12:14:16"
>>> date = time.strptime(date, "%y-%m-%d %H:%M:%S")
>>> epoch = datetime.datetime.fromtimestamp(time.mktime(date)).strftime('%s')
>>> epoch
'1399490056'
>>>
I've never had to convert time to and from UTC. Recently had a request to have my app be timezone aware, and I've been running myself in circles. Lots of information on converting local time to UTC, which I found fairly elementary (maybe I'm doing that wrong as well), but I can not find any information on easily converting the UTC time to the end-users timezone.
In a nutshell, and android app sends me (appengine app) data and within that data is a timestamp. To store that timestamp to utc time I am using:
datetime.utcfromtimestamp(timestamp)
That seems to be working. When my app stores the data, it is being store as 5 hours ahead (I am EST -5)
The data is being stored on appengine's BigTable, and when retrieved it comes out as a string like so:
"2011-01-21 02:37:21"
How do I convert this string to a DateTime in the users correct time zone?
Also, what is the recommended storage for a users timezone information? (How do you typically store tz info ie: "-5:00" or "EST" etc etc ?) I'm sure the answer to my first question might contain a parameter the answers the second.
If you don't want to provide your own tzinfo objects, check out the python-dateutil library. It provides tzinfo implementations on top of a zoneinfo (Olson) database such that you can refer to time zone rules by a somewhat canonical name.
from datetime import datetime
from dateutil import tz
# METHOD 1: Hardcode zones:
from_zone = tz.gettz('UTC')
to_zone = tz.gettz('America/New_York')
# METHOD 2: Auto-detect zones:
from_zone = tz.tzutc()
to_zone = tz.tzlocal()
# utc = datetime.utcnow()
utc = datetime.strptime('2011-01-21 02:37:21', '%Y-%m-%d %H:%M:%S')
# Tell the datetime object that it's in UTC time zone since
# datetime objects are 'naive' by default
utc = utc.replace(tzinfo=from_zone)
# Convert time zone
central = utc.astimezone(to_zone)
Edit Expanded example to show strptime usage
Edit 2 Fixed API usage to show better entry point method
Edit 3 Included auto-detect methods for timezones (Yarin)
Here's a resilient method that doesn't depend on any external libraries:
from datetime import datetime
import time
def datetime_from_utc_to_local(utc_datetime):
now_timestamp = time.time()
offset = datetime.fromtimestamp(now_timestamp) - datetime.utcfromtimestamp(now_timestamp)
return utc_datetime + offset
This avoids the timing issues in DelboyJay's example. And the lesser timing issues in Erik van Oosten's amendment.
As an interesting footnote, the timezone offset computed above can differ from the following seemingly equivalent expression, probably due to daylight savings rule changes:
offset = datetime.fromtimestamp(0) - datetime.utcfromtimestamp(0) # NO!
Update: This snippet has the weakness of using the UTC offset of the present time, which may differ from the UTC offset of the input datetime. See comments on this answer for another solution.
To get around the different times, grab the epoch time from the time passed in. Here's what I do:
def utc2local(utc):
epoch = time.mktime(utc.timetuple())
offset = datetime.fromtimestamp(epoch) - datetime.utcfromtimestamp(epoch)
return utc + offset
See the datetime documentation on tzinfo objects. You have to implement the timezones you want to support yourself. The are examples at the bottom of the documentation.
Here's a simple example:
from datetime import datetime,tzinfo,timedelta
class Zone(tzinfo):
def __init__(self,offset,isdst,name):
self.offset = offset
self.isdst = isdst
self.name = name
def utcoffset(self, dt):
return timedelta(hours=self.offset) + self.dst(dt)
def dst(self, dt):
return timedelta(hours=1) if self.isdst else timedelta(0)
def tzname(self,dt):
return self.name
GMT = Zone(0,False,'GMT')
EST = Zone(-5,False,'EST')
print datetime.utcnow().strftime('%m/%d/%Y %H:%M:%S %Z')
print datetime.now(GMT).strftime('%m/%d/%Y %H:%M:%S %Z')
print datetime.now(EST).strftime('%m/%d/%Y %H:%M:%S %Z')
t = datetime.strptime('2011-01-21 02:37:21','%Y-%m-%d %H:%M:%S')
t = t.replace(tzinfo=GMT)
print t
print t.astimezone(EST)
Output
01/22/2011 21:52:09
01/22/2011 21:52:09 GMT
01/22/2011 16:52:09 EST
2011-01-21 02:37:21+00:00
2011-01-20 21:37:21-05:00a
If you want to get the correct result even for the time that corresponds to an ambiguous local time (e.g., during a DST transition) and/or the local utc offset is different at different times in your local time zone then use pytz timezones:
#!/usr/bin/env python
from datetime import datetime
import pytz # $ pip install pytz
import tzlocal # $ pip install tzlocal
local_timezone = tzlocal.get_localzone() # get pytz tzinfo
utc_time = datetime.strptime("2011-01-21 02:37:21", "%Y-%m-%d %H:%M:%S")
local_time = utc_time.replace(tzinfo=pytz.utc).astimezone(local_timezone)
This answer should be helpful if you don't want to use any other modules besides datetime.
datetime.utcfromtimestamp(timestamp) returns a naive datetime object (not an aware one). Aware ones are timezone aware, and naive are not. You want an aware one if you want to convert between timezones (e.g. between UTC and local time).
If you aren't the one instantiating the date to start with, but you can still create a naive datetime object in UTC time, you might want to try this Python 3.x code to convert it:
import datetime
d=datetime.datetime.strptime("2011-01-21 02:37:21", "%Y-%m-%d %H:%M:%S") #Get your naive datetime object
d=d.replace(tzinfo=datetime.timezone.utc) #Convert it to an aware datetime object in UTC time.
d=d.astimezone() #Convert it to your local timezone (still aware)
print(d.strftime("%d %b %Y (%I:%M:%S:%f %p) %Z")) #Print it with a directive of choice
Be careful not to mistakenly assume that if your timezone is currently MDT that daylight savings doesn't work with the above code since it prints MST. You'll note that if you change the month to August, it'll print MDT.
Another easy way to get an aware datetime object (also in Python 3.x) is to create it with a timezone specified to start with. Here's an example, using UTC:
import datetime, sys
aware_utc_dt_obj=datetime.datetime.now(datetime.timezone.utc) #create an aware datetime object
dt_obj_local=aware_utc_dt_obj.astimezone() #convert it to local time
#The following section is just code for a directive I made that I liked.
if sys.platform=="win32":
directive="%#d %b %Y (%#I:%M:%S:%f %p) %Z"
else:
directive="%-d %b %Y (%-I:%M:%S:%f %p) %Z"
print(dt_obj_local.strftime(directive))
If you use Python 2.x, you'll probably have to subclass datetime.tzinfo and use that to help you create an aware datetime object, since datetime.timezone doesn't exist in Python 2.x.
If using Django, you can use the timezone.localtime method:
from django.utils import timezone
date
# datetime.datetime(2014, 8, 1, 20, 15, 0, 513000, tzinfo=<UTC>)
timezone.localtime(date)
# datetime.datetime(2014, 8, 1, 16, 15, 0, 513000, tzinfo=<DstTzInfo 'America/New_York' EDT-1 day, 20:00:00 DST>)
The following worked for me in a Cloud environment for US west:
import datetime
import pytz
#set the timezone
tzInfo = pytz.timezone('America/Los_Angeles')
dt = datetime.datetime.now(tz=tzInfo)
print(dt)
Consolidating the answer from franksands into a convenient method.
import calendar
import datetime
def to_local_datetime(utc_dt):
"""
convert from utc datetime to a locally aware datetime according to the host timezone
:param utc_dt: utc datetime
:return: local timezone datetime
"""
return datetime.datetime.fromtimestamp(calendar.timegm(utc_dt.timetuple()))
You can use arrow
from datetime import datetime
import arrow
now = datetime.utcnow()
print(arrow.get(now).to('local').format())
# '2018-04-04 15:59:24+02:00'
you can feed arrow.get() with anything. timestamp, iso string etc
You can use calendar.timegm to convert your time to seconds since Unix epoch and time.localtime to convert back:
import calendar
import time
time_tuple = time.strptime("2011-01-21 02:37:21", "%Y-%m-%d %H:%M:%S")
t = calendar.timegm(time_tuple)
print time.ctime(t)
Gives Fri Jan 21 05:37:21 2011 (because I'm in UTC+03:00 timezone).
import datetime
def utc_str_to_local_str(utc_str: str, utc_format: str, local_format: str):
"""
:param utc_str: UTC time string
:param utc_format: format of UTC time string
:param local_format: format of local time string
:return: local time string
"""
temp1 = datetime.datetime.strptime(utc_str, utc_format)
temp2 = temp1.replace(tzinfo=datetime.timezone.utc)
local_time = temp2.astimezone()
return local_time.strftime(local_format)
utc_tz_example_str = '2018-10-17T00:00:00.111Z'
utc_fmt = '%Y-%m-%dT%H:%M:%S.%fZ'
local_fmt = '%Y-%m-%dT%H:%M:%S+08:00'
# call my function here
local_tz_str = utc_str_to_local_str(utc_tz_example_str, utc_fmt, local_fmt)
print(local_tz_str) # 2018-10-17T08:00:00+08:00
When I input utc_tz_example_str = 2018-10-17T00:00:00.111Z, (UTC +00:00)
then I will get local_tz_str = 2018-10-17T08:00:00+08:00 (My target timezone +08:00)
parameter utc_format is a format determined by your specific utc_tz_example_str.
parameter local_fmt is the final desired format.
In my case, my desired format is %Y-%m-%dT%H:%M:%S+08:00 ( +08:00 timezone). You should construct the format you want.
This worked for me:
from django.utils import timezone
from datetime import timedelta,datetime
ist_time = timezone.now() + timedelta(hours=5,minutes=30)
#second method
ist_time = datetime.now() + timedelta(hours=5,minutes=30)
I traditionally defer this to the frontend -- send times from the backend as timestamps or some other datetime format in UTC, then let the client figure out the timezone offset and render this data in the proper timezone.
For a webapp, this is pretty easy to do in javascript -- you can figure out the browser's timezone offset pretty easily using builtin methods and then render the data from the backend properly.
From the answer here, you can use the time module to convert from utc to the local time set in your computer:
utc_time = time.strptime("2018-12-13T10:32:00.000", "%Y-%m-%dT%H:%M:%S.%f")
utc_seconds = calendar.timegm(utc_time)
local_time = time.localtime(utc_seconds)
Here is a quick and dirty version that uses the local systems settings to work out the time difference. NOTE: This will not work if you need to convert to a timezone that your current system is not running in. I have tested this with UK settings under BST timezone
from datetime import datetime
def ConvertP4DateTimeToLocal(timestampValue):
assert isinstance(timestampValue, int)
# get the UTC time from the timestamp integer value.
d = datetime.utcfromtimestamp( timestampValue )
# calculate time difference from utcnow and the local system time reported by OS
offset = datetime.now() - datetime.utcnow()
# Add offset to UTC time and return it
return d + offset
Short and simple:
from datetime import datetime
t = "2011-01-21 02:37:21"
datetime.fromisoformat(t) + (datetime.now() - datetime.utcnow())