1634515205001 converted to date & timer is : Mon Oct 18 2021 00:00:05
I attempt to convert 1634515205001 to date and time using :
import datetime
dt1 = datetime.datetime.strptime(
str(datetime.datetime.fromtimestamp(1634515205001 / 1000)),
"%Y-%m-%d %H:%M:%S.%f").strftime(
'%Y-%m-%dT%H:%M:%SZ')
print('dt1' , dt1)
which prints :
dt1 2021-10-18T01:00:05Z
Why is a division by 1000 (1634515205001 / 1000) required ? Using :
dt1 = datetime.datetime.strptime(
str(datetime.datetime.fromtimestamp(1634515205001)),
"%Y-%m-%d %H:%M:%S.%f").strftime(
'%Y-%m-%dT%H:%M:%SZ')
print('dt1', dt1)
renders :
str(datetime.datetime.fromtimestamp(1634515205001)),
OSError: [Errno 22] Invalid argument
Does fromtimestamp not accept millisecond as parameter ?
The datetime.fromtimestamp() function converts time in seconds (not millseconds) so epoch times in milliseconds need to be converted to seconds (divide by 1000).
This call is trying to compute 1634515205001 seconds since 1-Jan-1970 which is year 53765, and raises an exception.
dt = datetime.fromtimestamp(1634515205001)
Can get the number of expected digits for fromtimestamp() for the current time by calling time.time() to get the time in seconds for the current time.
print(time.time())
Output:
1649365827.417279 <= 10 digits before the decimal place
Time zones
The function fromtimestamp() is timezone aware so if the timezone argument is not provided then date time is converted to the local time zone. If the timezone is not specified and the timestamp time zone is different than the local time zone then the converted date time will be computed incorrectly. Below the local time zone is UTC-4.
dt = datetime.fromtimestamp(1634515205.001, tz=timezone.utc)
print(dt) # => 2021-10-18 00:00:05.001000+00:00
dt = datetime.fromtimestamp(1634515205.001)
print(dt) # => 2021-10-17 20:00:05.001000
Warning: Because naive datetime objects are treated by many datetime methods as
local times, it is preferred to use aware datetimes to represent times
in UTC. As such, the recommended way to create an object representing
a specific timestamp in UTC is by calling
datetime.fromtimestamp(timestamp, tz=timezone.utc).
To answer your question, yes, Python takes seconds instead of milliseconds when computing the dates.
From the documentation, datetime.datetime.fromtimestamp function definition says:
Return the local date corresponding to the POSIX timestamp, such as is returned by time.time().
The POSIX timestamp is the number of seconds that have elapsed since the 1st of January from 1970 (UTC).
So yes, fromtimestamp does not accept milliseconds, since POSIX timestamps are defined as seconds
If you are working with milliseconds, you have also the alternative to use something like:
datetime = datetime(1970, 1, 1) + timedelta(milliseconds=1634515205001)
Related
I can convert from an UTC timestamp e.g. 1417392000 to a local datetime object including daylight saving time, but when I try and convert the local datetime object to a "local timestamp" then I get the same UTC timestamp as before.
Am I thinking in circles and they are the same? I was supposed to save "local timestamp" from incoming UTC timestamp.
This is my code
print("UTC timestamp %d" % hour[0])
day = self.get_day(hour)
month = self.get_month(hour)
year = self.get_year(hour)
tz = pytz.timezone('Europe/Stockholm')
utc_dt = datetime.utcfromtimestamp(int(hour[0])).replace(tzinfo=pytz.utc)
print("UTC datetime %s" % utc_dt)
dt = tz.normalize(utc_dt.astimezone(tz))
print("STO datetime %s" % dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
print("STO ts %d" % int(time.mktime(dt.timetuple())))
print("STO timestamp %d" % utc_dt.astimezone(tz).timestamp())
day = int(dt.strftime('%d'))
month = int(dt.strftime('%m'))
year = int(dt.strftime('%Y'))
Output
UTC timestamp 1417395600
UTC datetime 2014-12-01 01:00:00+00:00
STO datetime 2014-12-01 02:00:00 CET+0100
STO ts 1417395600
STO timestamp 1417395600
All "timestamps" (i.e. integer representations of the time) are the same. Is it possible to make a "local timestamp" ? The data type is supposed to be a timestamp that is a number and in local time.
As per Wikipedia
Unix time (also known as POSIX time or Epoch time) is a system for
describing instants in time, defined as the number of seconds that
have elapsed since 00:00:00 Coordinated Universal Time (UTC),
Thursday, 1 January 1970
So regardless of what timezone you're on, the epoch will always be calculated in the UTC timezone.
For display purposes, you can convert the timestamp to a local time, but as otherwise the internal representation of the epoch will always be in the UTC timezone
Formal timestamps
The Unix timestamp typically refers to the number of seconds since the epoch in UTC. This value is invariant to timezone. It allows global event ordering but loses timezone information.
Preserving timezone
To preserve timezone information, a standardized format is RFC3339, Date and Time on the Internet: Timestamps. This is just a standardized formatting that encodes date+time+timezone. Some examples:
1985-04-12T23:20:50.52Z
1996-12-19T16:39:57-08:00
1990-12-31T23:59:60Z
1990-12-31T15:59:60-08:00
Normalizing for timezone without preservation of timezone
However, it may depend on your requirements. I once wanted to record some events relative to local-time-of-day and did not mind losing timezone information. I normalized the timestamp with respect to 1970-01-01T00:00:00 in the local timezone. I am a little sheepish about this now as I think it may too easily cause confusion.
import time
# Number of seconds since 1970-01-01T00:00+LTZ (current timezone).
# unix timestamp - timezone offset in seconds
timestamp_localized = time.time() - time.mktime(time.gmtime(0))
However this syntax can be simplified, perhaps at the loss of clarity, by noticing that Python has some localtime and UTC specific functions.
import time
import calendar
# Number of seconds since 1970-01-01T00:00+LTZ (current timezone).
# Interpret local date and time as if it were UTC
timestamp_localized = calendar.timegm(time.localtime())
The difference between these two is that calendar conversion truncates to the second while the difference calculation includes a fractional second.
I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
I am trying to convert them into epoch timestamps for UTC time.
I wrote the following function:
def ymdhms_timezone_dst_to_epoch(input_str, tz="US/Eastern"):
print(input_str)
dt = datetime.datetime.fromtimestamp(time.mktime(time.strptime(input_str,'%Y-%m-%d %H:%M:%S')))
local_dt = pytz.timezone(tz).localize(dt)
print(local_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
utc_dt = local_dt.astimezone(pytz.utc)
print(utc_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
e = int(utc_dt.strftime("%s"))
print(e)
return e
Given string `2015-04-20 21:12:07` this prints:
2015-04-20 21:12:07
2015-04-20 21:12:07 EDT-0400 #<- so far so good?
2015-04-21 01:12:07 UTC+0000 #<- so far so good?
1429596727
which looks ok up to the epoch timestamp. But http://www.epochconverter.com/epoch/timezones.php?epoch=1429596727 says it should mao to
Greenwich Mean Time Apr 21 2015 06:12:07 UTC.
What is wrong?
I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
A portable way is to use pytz:
#!/usr/bin/env python
from datetime import datetime
import pytz # $ pip install pytz
naive_dt = datetime.strptime('2015-04-20 21:12:07', '%Y-%m-%d %H:%M:%S')
tz = pytz.timezone('US/Eastern')
eastern_dt = tz.normalize(tz.localize(naive_dt))
print(eastern_dt)
# -> 2015-04-20 21:12:07-04:00
I am trying to convert them into epoch timestamps for UTC time.
timestamp = (eastern_dt - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
# -> 1429578727.0
See Converting datetime.date to UTC timestamp in Python.
There are multiple issues in your code:
time.mktime() may return a wrong result for ambiguous input time (50% chance) e.g., during "fall back" DST transition in the Fall
time.mktime() and datetime.fromtimestamp() may fail for past/future dates if they have no access to a historical timezone database on a system (notably, Windows)
localize(dt) may return a wrong result for ambiguous or non-existent time i.e., during DST transitions. If you know that the time corresponds to the summer time then use is_dst=True. tz.normalize() is necessary here, to adjust possible non-existing times in the input
utc_dt.strftime("%s") is not portable and it does not respect tzinfo object. It interprets input as a local time i.e., it returns a wrong result unless your local timezone is UTC.
Can I just always set is_dst=True?
You can, if you don't mind getting imprecise results for ambiguous or non-existent times e.g., there is DST transition in the Fall in America/New_York time zone:
>>> from datetime import datetime
>>> import pytz # $ pip install pytz
>>> tz = pytz.timezone('America/New_York')
>>> ambiguous_time = datetime(2015, 11, 1, 1, 30)
>>> time_fmt = '%Y-%m-%d %H:%M:%S%z (%Z)'
>>> tz.localize(ambiguous_time).strftime(time_fmt)
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=False).strftime(time_fmt) # same
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=True).strftime(time_fmt) # different
'2015-11-01 01:30:00-0400 (EDT)'
>>> tz.localize(ambiguous_time, is_dst=None).strftime(time_fmt)
Traceback (most recent call last):
...
pytz.exceptions.AmbiguousTimeError: 2015-11-01 01:30:00
The clocks are turned back at 2a.m. on the first Sunday in November:
is_dst disambiguation flag may have three values:
False -- default, assume the winter time
True -- assume the summer time
None -- raise an exception for ambiguous/non-existent times.
is_dst value is ignored for existing unique local times.
Here's a plot from PEP 0495 -- Local Time Disambiguation that illustrates the DST transition:
The local time repeats itself twice in the fold (summer time -- before the fold, winter time -- after).
To be able to disambiguate the local time automatically, you need some additional info e.g., if you read a series of local times then it may help if you know that they are sorted: Parsing of Ordered Timestamps in Local Time (to UTC) While Observing Daylight Saving Time.
First of all '%s' is not supported on all platforms , its actually working for you because your platform C library’s strftime() function (that is called by Python) supports it. This function is what is causing the issue most probably, I am guessing its not timezone aware , hence when taking difference from epoch time it is using your local timezone, which is most probably EST(?)
Instead of relying on '%s' , which only works in few platforms (linux, I believe) , you should manually subtract the datetime you got from epoch (1970/1/1 00:00:00) to get the actual seconds since epoch . Example -
e = (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
Demo -
>>> (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
1429578727.0
This correctly corresponds to the date-time you get.
I don't exactly know why but you have to remove the timezone info from your utc_dt before using %s to print it.
e = int(utc_dt.replace(tzinfo=None).strftime("%s"))
print(e)
return e
I am trying to write a countdown clock script. I want to use a set date in the future and have it count down in a nice readable format. Hours, Min, Sec. I am going to print to a 16x2 lCD display. The problem I'm having is trying to take the output of the difference between dates into a nice format. I have attached what I have so far. I receive the error:
AttributeError: 'datetime.timedelta' object has no attribute 'strftime'
This is my code:
from datetime import datetime
from time import strftime
deploy = datetime(2015, 3, 21, 0, 0)
mydate = datetime.now() - deploy
print (mydate.strftime("%b %d %H:%M:%S"))
I know how to print to my LCD and create a loop, just need help with this part.
There are two issues:
the time difference may be incorrect if you use local time represented as a naive datetime object if the corresponding local times have different utc offsets e.g., around a DST transition
the difference is timedelta object that has no strftime() method
To fix it, convert deploy from local timezone to UTC:
#!/usr/bin/env python
import time
from datetime import datetime, timedelta
deploy = datetime(2015, 3, 21, 0, 0) # assume local time
timestamp = time.mktime(deploy.timetuple()) # may fail, see the link below
deploy_utc = datetime.utcfromtimestamp(timestamp)
elapsed = deploy_utc - datetime.utcnow() # `deploy` is in the future
where elapsed is the elapsed time not counting leap seconds (such as 2015-07-01 00:59:60 BST+0100).
More details on when time.mktime() may fail see at Find if 24 hrs have passed between datetimes - Python.
To convert timedelta to string, you could use str() function:
print(elapsed) # print full timedelta
# remove microseconds
trunc_micros = timedelta(days=elapsed.days, seconds=elapsed.seconds)
print(trunc_micros) # -> 20 days, 13:44:14 <- 17 chars
# remove comma
print(str(trunc_micros).replace(',', ''))
# -> 20 days 13:44:14 <- 16 chars
If you want a different format then convert to hours, minutes, second using divmod() function:
seconds = elapsed.days*86400 + elapsed.seconds # drop microseconds
minutes, seconds = divmod(seconds, 60)
hours, minutes = divmod(minutes, 60)
print("{hours:02d}:{minutes:02d}:{seconds:02d}".format(**vars()))
# -> 493:44:14
I would like to convert string date format to timestamp with microseconds
I try the following but not giving expected result:
"""input string date -> 2014-08-01 04:41:52,117
expected result -> 1410748201.117"""
import time
import datetime
myDate = "2014-08-01 04:41:52,117"
timestamp = time.mktime(datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timetuple())
print timestamp
> 1410748201.0
Where did the milliseconds go?
There is no slot for the microseconds component in a time tuple:
>>> import time
>>> import datetime
>>> myDate = "2014-08-01 04:41:52,117"
>>> datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timetuple()
time.struct_time(tm_year=2014, tm_mon=8, tm_mday=1, tm_hour=4, tm_min=41, tm_sec=52, tm_wday=4, tm_yday=213, tm_isdst=-1)
You'll have to add those manually:
>>> dt = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
>>> time.mktime(dt.timetuple()) + (dt.microsecond / 1000000.0)
1406864512.117
The other method you could follow is to produce a timedelta() object relative to the epoch, then get the timestamp with the timedelta.total_seconds() method:
epoch = datetime.datetime.fromtimestamp(0)
(dt - epoch).total_seconds()
The use of a local time epoch is quite deliberate since you have a naive (not timezone-aware) datetime value. This method can be inaccurate based on the history of your local timezone however, see J.F. Sebastian's comment. You'd have to convert the naive datetime value to a timezone-aware datetime value first using your local timezone before subtracting a timezone-aware epoch.
As such, it is easier to stick to the timetuple() + microseconds approach.
Demo:
>>> dt = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
>>> epoch = datetime.datetime.fromtimestamp(0)
>>> (dt - epoch).total_seconds()
1406864512.117
In Python 3.4 and later you can use
timestamp = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timestamp()
This doesn't require importing the time module. It also uses less steps so it should be faster. For older versions of python the other provided answers are probably your best option.
However, the resulting timestamp will interpret myDate in local time, rather than UTC, which may cause issues if myDate was given in UTC
Where did the milliseconds go?
It is the easy part. .timetuple() call drops them. You could add them back using .microsecond attribute. The datetime.timestamp() method from the standard library works that way for naive datetime objects:
def timestamp(self):
"Return POSIX timestamp as float"
if self._tzinfo is None:
return _time.mktime((self.year, self.month, self.day,
self.hour, self.minute, self.second,
-1, -1, -1)) + self.microsecond / 1e6
else:
return (self - _EPOCH).total_seconds()
It is enough if possible ~1 hour errors could be ignored in your case. I assume that you want microseconds and therefore you can't ignore ~1 hour time errors silently.
To convert the local time given as a string to the POSIX timestamp correctly is a complex task in general. You could convert the local time to UTC and then get the timestamp from UTC time.
There are two main issues:
local time may be non-existent or ambiguous e.g. during DST transitions the same time may occur twice
UTC offset for the local timezone may be different in the past and therefore a naive: local time minus epoch in local time formula may fail
Both can be solved using the tz database (pytz module in Python):
from datetime import datetime
import pytz # $ pip install pytz
from tzlocal import get_localzone # $ pip install tzlocal
tz = get_localzone() # get pytz timezone corresponding to the local timezone
naive_d = datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
# a) raise exception for non-existent or ambiguous times
d = tz.localize(naive_d, is_dst=None)
## b) assume standard time, adjust non-existent times
#d = tz.normalize(tz.localize(naive_d, is_dst=False))
## c) assume DST is in effect, adjust non-existent times
#d = tz.normalize(tz.localize(naive_d, is_dst=True))
timestamp = d - datetime(1970, 1, 1, tzinfo=pytz.utc)
The result is timestamp -- a timedelta object, you can convert it to seconds, milliseconds, etc.
Also different systems may behave differently around/during leap seconds. Most application can ignore that they exist.
In general, it might be simpler to store POSIX timestamps in addition to the local time instead of trying to guess it from the local time.
I am looking to analyze traffic flow with relation to weather data. The traffic data has a UNIX timestamp (aka epoch), but I am running into trouble with converting the timestamp (in the weather data) to epoch. The problem is that I am in Norway and the UTC timestamp in the weather data isn't in the same timezone as me (GMT+1).
My initial approach
I first tried converting it into epoch and treating the data as if it was in the GMT+1 timezone. Then I compensated by subtracting the difference in number of seconds between UTC and GMT+1.
Problems with the approach
I realize first of all that this approach is very primitive and not very elegant (in fact probably it is at best an ugly hack). However, the biggest problem here is that the difference between UTC and GMT+1 is not constant (due to daylight savings).
Question
Is there any reliable way of turning UTC time to a UNIX time stamp in python (taking into account that my machine is in GMT+1)? The timestamp is in the following format:
Y-m-d HH:MM:SS
Edit:
Tried rmunns' solution:
def convert_UTC_to_epoch(timestamp):
tz_UTC = pytz.timezone('UTC')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime.strptime(timestamp, time_format)
aware_timestamp = tz_UTC.localize(naive_timestamp)
epoch = aware_timestamp.strftime("%s")
return (int) (epoch)
This does not work properly as evidenced below:
#Current time at time of the edit is 15:55:00 UTC on June 9th 2014.
>>> diff = time.time() - convert_UTC_to_epoch("2014-06-09 15:55:00")
>>> diff
3663.25887799263
>>> #This is about an hour off.
The solution was to use the calendar module (inspired from here)
>>>#Quick and dirty demo
>>>print calendar.timegm(datetime.datetime.utcnow().utctimetuple()) - time.time()
>>>-0.6182510852813721
And here is the conversion function:
import calendar, datetime, time
#Timestamp is a datetime object in UTC time
def UTC_time_to_epoch(timestamp):
epoch = calendar.timegm(timestamp.utctimetuple())
return epoch
An alternative, datetime has it's own .strptime() method.
http://en.wikipedia.org/wiki/Unix_time
The Unix epoch is the time 00:00:00 UTC on 1 January 1970 (or 1970-01-01T00:00:00Z ISO 8601).
import datetime
unix_epoch = datetime.datetime(1970, 1, 1)
log_dt = datetime.datetime.strptime("14-05-07 12:14:16", "%y-%m-%d %H:%M:%S")
seconds_from_epoch = (log_dt - unix_epoch).total_seconds()
>>> 1399490056.0
The pytz module will probably help you. It allows you to write code like:
import pytz
import datetime
tz_oslo = pytz.timezone('Europe/Oslo')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime(2014, 6, 4, 12, 34, 56)
# Or:
naive_timestamp = datetime.datetime.strptime("2014-06-04 12:34:56", time_format)
aware_timestamp = tz_oslo.localize(naive_timestamp)
print(aware_timestamp.strftime(time_format + " %Z%z"))
This should print "2014-06-04 14:34:56 CEST+0200".
Do note the following from the pytz manual:
The preferred way of dealing with times is to always work in UTC, converting to localtime only when generating output to be read by humans.
So keep that in mind as you write your code: do the conversion to local time once and once only, and you'll have a much easier time doing, say, comparisons between two timestamps correctly.
Update: Here are a couple of videos you may find useful:
What you need to know about datetimes, a PyCon 2012 presentation by Taavi Burns (30 minutes)
Drive-in Double Header: Datetimes and Log Analysis, a two-part presentation. (Caution: annoying buzz in the video, but I couldn't find a copy with better sound). The first part is the "What you need to know about datetimes" presentation I linked just above, and the second part has some practical tips for parsing log files and doing useful things with them. (50 minutes)
Update 2: The convert_UTC_to_epoch() function you mention in your updated question (which I've reproduced below) is returning local time, not UTC:
def convert_UTC_to_epoch(timestamp):
tz_UTC = pytz.timezone('UTC')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime.strptime(timestamp, time_format)
aware_timestamp = tz_UTC.localize(naive_timestamp)
epoch = aware_timestamp.strftime("%s")
return (int) (epoch)
The problem is that you're using strftime("%s"), which is undocumented and is returning the wrong result. Python doesn't support the %s parameter, but it appears to work because it gets passed to your system's strftime() function, which does support the %s parameter -- but it returns local time! You're taking a UTC timestamp and parsing it as local time, which is why it's an hour off. (The mystery is why it isn't two hours off -- isn't Norway in daylight savings time right now? Shouldn't you be at UTC+2?)
As you can see from the interactive Python session below, I'm in the UTC+7 timezone and your convert_UTC_to_epoch() function is seven hours off for me.
# Current time is 02:42 UTC on June 10th 2014, 09:42 local time
>>> time.timezone
-25200
>>> time.time() - convert_UTC_to_epoch("2014-06-10 02:42:00")
25204.16531395912
>>> time.time() + time.timezone - convert_UTC_to_epoch("2014-06-10 02:42:00")
6.813306093215942
The strftime("%s") call is interpreting 02:42 on June 10th as being in local time, which would be 19:42 UTC on June 9th. Subtracting 19:42 UTC on June 9th from 02:42 UTC June 10th (which is what time.time() returns) gives a difference of seven hours. See Convert python datetime to epoch with strftime for more details on why you should never use strftime("%s").
(By the way, if you saw what I had previously written under the heading "Update 2", where I claimed that time.time() was returning local time, ignore that -- I got it wrong. I was fooled at first by the strftime("%s") bug just like you were.)
You can use the time and datetime modules:
import time, datetime
date = "14-05-07 12:14:16" #Change to whatever date you want
date = time.strptime(date, "%y-%m-%d %H:%M:%S")
epoch = datetime.datetime.fromtimestamp(time.mktime(date)).strftime('%s')
This runs as:
>>> import time, datetime
>>> date = "14-05-07 12:14:16"
>>> date = time.strptime(date, "%y-%m-%d %H:%M:%S")
>>> epoch = datetime.datetime.fromtimestamp(time.mktime(date)).strftime('%s')
>>> epoch
'1399490056'
>>>