Is there a "local" timestamp? - python

I can convert from an UTC timestamp e.g. 1417392000 to a local datetime object including daylight saving time, but when I try and convert the local datetime object to a "local timestamp" then I get the same UTC timestamp as before.
Am I thinking in circles and they are the same? I was supposed to save "local timestamp" from incoming UTC timestamp.
This is my code
print("UTC timestamp %d" % hour[0])
day = self.get_day(hour)
month = self.get_month(hour)
year = self.get_year(hour)
tz = pytz.timezone('Europe/Stockholm')
utc_dt = datetime.utcfromtimestamp(int(hour[0])).replace(tzinfo=pytz.utc)
print("UTC datetime %s" % utc_dt)
dt = tz.normalize(utc_dt.astimezone(tz))
print("STO datetime %s" % dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
print("STO ts %d" % int(time.mktime(dt.timetuple())))
print("STO timestamp %d" % utc_dt.astimezone(tz).timestamp())
day = int(dt.strftime('%d'))
month = int(dt.strftime('%m'))
year = int(dt.strftime('%Y'))
Output
UTC timestamp 1417395600
UTC datetime 2014-12-01 01:00:00+00:00
STO datetime 2014-12-01 02:00:00 CET+0100
STO ts 1417395600
STO timestamp 1417395600
All "timestamps" (i.e. integer representations of the time) are the same. Is it possible to make a "local timestamp" ? The data type is supposed to be a timestamp that is a number and in local time.

As per Wikipedia
Unix time (also known as POSIX time or Epoch time) is a system for
describing instants in time, defined as the number of seconds that
have elapsed since 00:00:00 Coordinated Universal Time (UTC),
Thursday, 1 January 1970
So regardless of what timezone you're on, the epoch will always be calculated in the UTC timezone.
For display purposes, you can convert the timestamp to a local time, but as otherwise the internal representation of the epoch will always be in the UTC timezone

Formal timestamps
The Unix timestamp typically refers to the number of seconds since the epoch in UTC. This value is invariant to timezone. It allows global event ordering but loses timezone information.
Preserving timezone
To preserve timezone information, a standardized format is RFC3339, Date and Time on the Internet: Timestamps. This is just a standardized formatting that encodes date+time+timezone. Some examples:
1985-04-12T23:20:50.52Z
1996-12-19T16:39:57-08:00
1990-12-31T23:59:60Z
1990-12-31T15:59:60-08:00
Normalizing for timezone without preservation of timezone
However, it may depend on your requirements. I once wanted to record some events relative to local-time-of-day and did not mind losing timezone information. I normalized the timestamp with respect to 1970-01-01T00:00:00 in the local timezone. I am a little sheepish about this now as I think it may too easily cause confusion.
import time
# Number of seconds since 1970-01-01T00:00+LTZ (current timezone).
# unix timestamp - timezone offset in seconds
timestamp_localized = time.time() - time.mktime(time.gmtime(0))
However this syntax can be simplified, perhaps at the loss of clarity, by noticing that Python has some localtime and UTC specific functions.
import time
import calendar
# Number of seconds since 1970-01-01T00:00+LTZ (current timezone).
# Interpret local date and time as if it were UTC
timestamp_localized = calendar.timegm(time.localtime())
The difference between these two is that calendar conversion truncates to the second while the difference calculation includes a fractional second.

Related

Converting millisecond to timestamp

1634515205001 converted to date & timer is : Mon Oct 18 2021 00:00:05
I attempt to convert 1634515205001 to date and time using :
import datetime
dt1 = datetime.datetime.strptime(
str(datetime.datetime.fromtimestamp(1634515205001 / 1000)),
"%Y-%m-%d %H:%M:%S.%f").strftime(
'%Y-%m-%dT%H:%M:%SZ')
print('dt1' , dt1)
which prints :
dt1 2021-10-18T01:00:05Z
Why is a division by 1000 (1634515205001 / 1000) required ? Using :
dt1 = datetime.datetime.strptime(
str(datetime.datetime.fromtimestamp(1634515205001)),
"%Y-%m-%d %H:%M:%S.%f").strftime(
'%Y-%m-%dT%H:%M:%SZ')
print('dt1', dt1)
renders :
str(datetime.datetime.fromtimestamp(1634515205001)),
OSError: [Errno 22] Invalid argument
Does fromtimestamp not accept millisecond as parameter ?
The datetime.fromtimestamp() function converts time in seconds (not millseconds) so epoch times in milliseconds need to be converted to seconds (divide by 1000).
This call is trying to compute 1634515205001 seconds since 1-Jan-1970 which is year 53765, and raises an exception.
dt = datetime.fromtimestamp(1634515205001)
Can get the number of expected digits for fromtimestamp() for the current time by calling time.time() to get the time in seconds for the current time.
print(time.time())
Output:
1649365827.417279 <= 10 digits before the decimal place
Time zones
The function fromtimestamp() is timezone aware so if the timezone argument is not provided then date time is converted to the local time zone. If the timezone is not specified and the timestamp time zone is different than the local time zone then the converted date time will be computed incorrectly. Below the local time zone is UTC-4.
dt = datetime.fromtimestamp(1634515205.001, tz=timezone.utc)
print(dt) # => 2021-10-18 00:00:05.001000+00:00
dt = datetime.fromtimestamp(1634515205.001)
print(dt) # => 2021-10-17 20:00:05.001000
Warning: Because naive datetime objects are treated by many datetime methods as
local times, it is preferred to use aware datetimes to represent times
in UTC. As such, the recommended way to create an object representing
a specific timestamp in UTC is by calling
datetime.fromtimestamp(timestamp, tz=timezone.utc).
To answer your question, yes, Python takes seconds instead of milliseconds when computing the dates.
From the documentation, datetime.datetime.fromtimestamp function definition says:
Return the local date corresponding to the POSIX timestamp, such as is returned by time.time().
The POSIX timestamp is the number of seconds that have elapsed since the 1st of January from 1970 (UTC).
So yes, fromtimestamp does not accept milliseconds, since POSIX timestamps are defined as seconds
If you are working with milliseconds, you have also the alternative to use something like:
datetime = datetime(1970, 1, 1) + timedelta(milliseconds=1634515205001)

Python time library: how do I preserve dst with strptime and strftime

I need to store a timestamp in a readable format, and then later on I need to convert it to epoch for comparison purposes.
I tried doing this:
import time
format = '%Y %m %d %H:%M:%S +0000'
timestamp1 = time.strftime(format,time.gmtime()) # '2016 03 25 04:06:22 +0000'
t1 = time.strptime(timestamp1, format) # time.struct_time(..., tm_isdst=-1)
time.sleep(1)
epoch_now = time.mktime(time.gmtime())
epoch_t1 = time.mktime(t1)
print "Delta: %s" % (epoch_now - epoch_t1)
Running this, instead of getting Delta of 1 sec, I get 3601 (1 hr 1 sec), CONSISTENTLY.
Investigating further, it seems that when I just do time.gmtime(), the struct has tm_isdst=0, whereas the converted struct t1 from timestamp1 string has tm_isdst=-1.
How can I ensure the isdst is preserved to 0. I think that's probably the issue here.
Or is there a better way to record time in human readable format (UTC), and yet be able to convert back to epoch properly for time diff calculation?
UPDATES:
After doing more research last night, I switched to using datetime because it preserves more information in the datetime object, and this is confirmed by albertoql answer below.
Here's what I have now:
from datetime import datetime
format = '%Y-%m-%d %H:%M:%S.%f +0000' # +0000 is optional; only for user to see it's UTC
d1 = datetime.utcnow()
timestamp1 = d1.strftime(format)
d1a = datetime.strptime(timestamp1, format)
time.sleep(1)
d2 = datetime.utcnow()
print "Delta: %s" % (d2 - d1a).seconds
I chose not to add tz to keep it simple/shorter; I can still strptime that way.
Below, first an explanation about the problem, then two possible solutions, one using time, another using datetime.
Problem explanation
The problem is on the observation that the OP made in the question: tm_isdst=-1. tm_isdst is a flag that determines whether daylight savings time is in effect or not (see for more details https://docs.python.org/2/library/time.html#time.struct_time).
Specifically, given the format of the string for the time from the OP (that complies with RFC 2822 Internet email standard), [time.strptime]4 does not store the information about the timezone, namely +0000. Thus, when the struct_time is created again according to the information in the string, tm_isdst=-1, namely unknown. The guess on how to fill in that information when making the calculation is based on the local system. For example, as if the system refers to North America, where daylight savings time is in effect, tm_isdst is set.
Solution with time
If you want to use only time package, then, the easiest way to parse directly the information is to specify that the time is in UTC, and thus adding %Z to the format. Note that time does not provide a way to store the information about the timezone in struct_time. As a result, it does not print the actual time zone associated with the time saved in the variable. The time zone is retrieved from the system. Therefore, it is not possible to directly use the same format for time.strftime. The part of the code for writing and reading the string would look like:
format = '%Y %m %d %H:%M:%S UTC'
format2 = '%Y %m %d %H:%M:%S %Z'
timestamp1 = time.strftime(format, time.gmtime())
t1 = time.strptime(timestamp1, format2)
Solution with datetime
Another solution involves the use datetime and dateutil packages, which directly support timezone, and the code could be (assuming that preserving the timezone information is a requirement):
from datetime import datetime
from dateutil import tz, parser
import time
time_format = '%Y %m %d %H:%M:%S %z'
utc_zone = tz.gettz('UTC')
utc_time1 = datetime.utcnow()
utc_time1 = utc_time1.replace(tzinfo=utc_zone)
utc_time1_string = utc_time1.strftime(time_format)
utc_time1 = parser.parse(utc_time1_string)
time.sleep(1)
utc_time2 = datetime.utcnow()
utc_time2 = utc_time2.replace(tzinfo=utc_zone)
print "Delta: %s" % (utc_time2 - utc_time1).total_seconds()
There are some aspects to pay attention to:
After the call of utcnow, the timezone is not set, as it is a naive UTC datetime. If the information about UTC is not needed, it is possible to delete both lines where the timezone is set for the two times, and the result would be the same, as there is no guess about DST.
It is not possible to use datetime.strptime because of %z, which is not correctly parsed. If the string contains the information about the timezone, then parser should be used.
It is possible to directly perform the difference from two instances of datetime and transform the resulting delta into seconds.
If it is necessary to get the time in seconds since the epoch, an explicit computation should be made, as there is no direct function that does that automatically in datetime (at the time of the answer). Below the code, for example for utc_time2:
epoch_time = datetime(1970,1,1)
epoch2 = (utc_time2 - epoch_time).total_seconds()
datetime.resolution, namely the smallest possible difference between two non-equal datetime objects. This results in a difference that is up to the resolution.

python converting string in localtime to UTC epoch timestamp

I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
I am trying to convert them into epoch timestamps for UTC time.
I wrote the following function:
def ymdhms_timezone_dst_to_epoch(input_str, tz="US/Eastern"):
print(input_str)
dt = datetime.datetime.fromtimestamp(time.mktime(time.strptime(input_str,'%Y-%m-%d %H:%M:%S')))
local_dt = pytz.timezone(tz).localize(dt)
print(local_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
utc_dt = local_dt.astimezone(pytz.utc)
print(utc_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
e = int(utc_dt.strftime("%s"))
print(e)
return e
Given string `2015-04-20 21:12:07` this prints:
2015-04-20 21:12:07
2015-04-20 21:12:07 EDT-0400 #<- so far so good?
2015-04-21 01:12:07 UTC+0000 #<- so far so good?
1429596727
which looks ok up to the epoch timestamp. But http://www.epochconverter.com/epoch/timezones.php?epoch=1429596727 says it should mao to
Greenwich Mean Time Apr 21 2015 06:12:07 UTC.
What is wrong?
I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
A portable way is to use pytz:
#!/usr/bin/env python
from datetime import datetime
import pytz # $ pip install pytz
naive_dt = datetime.strptime('2015-04-20 21:12:07', '%Y-%m-%d %H:%M:%S')
tz = pytz.timezone('US/Eastern')
eastern_dt = tz.normalize(tz.localize(naive_dt))
print(eastern_dt)
# -> 2015-04-20 21:12:07-04:00
I am trying to convert them into epoch timestamps for UTC time.
timestamp = (eastern_dt - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
# -> 1429578727.0
See Converting datetime.date to UTC timestamp in Python.
There are multiple issues in your code:
time.mktime() may return a wrong result for ambiguous input time (50% chance) e.g., during "fall back" DST transition in the Fall
time.mktime() and datetime.fromtimestamp() may fail for past/future dates if they have no access to a historical timezone database on a system (notably, Windows)
localize(dt) may return a wrong result for ambiguous or non-existent time i.e., during DST transitions. If you know that the time corresponds to the summer time then use is_dst=True. tz.normalize() is necessary here, to adjust possible non-existing times in the input
utc_dt.strftime("%s") is not portable and it does not respect tzinfo object. It interprets input as a local time i.e., it returns a wrong result unless your local timezone is UTC.
Can I just always set is_dst=True?
You can, if you don't mind getting imprecise results for ambiguous or non-existent times e.g., there is DST transition in the Fall in America/New_York time zone:
>>> from datetime import datetime
>>> import pytz # $ pip install pytz
>>> tz = pytz.timezone('America/New_York')
>>> ambiguous_time = datetime(2015, 11, 1, 1, 30)
>>> time_fmt = '%Y-%m-%d %H:%M:%S%z (%Z)'
>>> tz.localize(ambiguous_time).strftime(time_fmt)
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=False).strftime(time_fmt) # same
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=True).strftime(time_fmt) # different
'2015-11-01 01:30:00-0400 (EDT)'
>>> tz.localize(ambiguous_time, is_dst=None).strftime(time_fmt)
Traceback (most recent call last):
...
pytz.exceptions.AmbiguousTimeError: 2015-11-01 01:30:00
The clocks are turned back at 2a.m. on the first Sunday in November:
is_dst disambiguation flag may have three values:
False -- default, assume the winter time
True -- assume the summer time
None -- raise an exception for ambiguous/non-existent times.
is_dst value is ignored for existing unique local times.
Here's a plot from PEP 0495 -- Local Time Disambiguation that illustrates the DST transition:
The local time repeats itself twice in the fold (summer time -- before the fold, winter time -- after).
To be able to disambiguate the local time automatically, you need some additional info e.g., if you read a series of local times then it may help if you know that they are sorted: Parsing of Ordered Timestamps in Local Time (to UTC) While Observing Daylight Saving Time.
First of all '%s' is not supported on all platforms , its actually working for you because your platform C library’s strftime() function (that is called by Python) supports it. This function is what is causing the issue most probably, I am guessing its not timezone aware , hence when taking difference from epoch time it is using your local timezone, which is most probably EST(?)
Instead of relying on '%s' , which only works in few platforms (linux, I believe) , you should manually subtract the datetime you got from epoch (1970/1/1 00:00:00) to get the actual seconds since epoch . Example -
e = (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
Demo -
>>> (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
1429578727.0
This correctly corresponds to the date-time you get.
I don't exactly know why but you have to remove the timezone info from your utc_dt before using %s to print it.
e = int(utc_dt.replace(tzinfo=None).strftime("%s"))
print(e)
return e

Convert a UTC time to epoch

I am looking to analyze traffic flow with relation to weather data. The traffic data has a UNIX timestamp (aka epoch), but I am running into trouble with converting the timestamp (in the weather data) to epoch. The problem is that I am in Norway and the UTC timestamp in the weather data isn't in the same timezone as me (GMT+1).
My initial approach
I first tried converting it into epoch and treating the data as if it was in the GMT+1 timezone. Then I compensated by subtracting the difference in number of seconds between UTC and GMT+1.
Problems with the approach
I realize first of all that this approach is very primitive and not very elegant (in fact probably it is at best an ugly hack). However, the biggest problem here is that the difference between UTC and GMT+1 is not constant (due to daylight savings).
Question
Is there any reliable way of turning UTC time to a UNIX time stamp in python (taking into account that my machine is in GMT+1)? The timestamp is in the following format:
Y-m-d HH:MM:SS
Edit:
Tried rmunns' solution:
def convert_UTC_to_epoch(timestamp):
tz_UTC = pytz.timezone('UTC')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime.strptime(timestamp, time_format)
aware_timestamp = tz_UTC.localize(naive_timestamp)
epoch = aware_timestamp.strftime("%s")
return (int) (epoch)
This does not work properly as evidenced below:
#Current time at time of the edit is 15:55:00 UTC on June 9th 2014.
>>> diff = time.time() - convert_UTC_to_epoch("2014-06-09 15:55:00")
>>> diff
3663.25887799263
>>> #This is about an hour off.
The solution was to use the calendar module (inspired from here)
>>>#Quick and dirty demo
>>>print calendar.timegm(datetime.datetime.utcnow().utctimetuple()) - time.time()
>>>-0.6182510852813721
And here is the conversion function:
import calendar, datetime, time
#Timestamp is a datetime object in UTC time
def UTC_time_to_epoch(timestamp):
epoch = calendar.timegm(timestamp.utctimetuple())
return epoch
An alternative, datetime has it's own .strptime() method.
http://en.wikipedia.org/wiki/Unix_time
The Unix epoch is the time 00:00:00 UTC on 1 January 1970 (or 1970-01-01T00:00:00Z ISO 8601).
import datetime
unix_epoch = datetime.datetime(1970, 1, 1)
log_dt = datetime.datetime.strptime("14-05-07 12:14:16", "%y-%m-%d %H:%M:%S")
seconds_from_epoch = (log_dt - unix_epoch).total_seconds()
>>> 1399490056.0
The pytz module will probably help you. It allows you to write code like:
import pytz
import datetime
tz_oslo = pytz.timezone('Europe/Oslo')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime(2014, 6, 4, 12, 34, 56)
# Or:
naive_timestamp = datetime.datetime.strptime("2014-06-04 12:34:56", time_format)
aware_timestamp = tz_oslo.localize(naive_timestamp)
print(aware_timestamp.strftime(time_format + " %Z%z"))
This should print "2014-06-04 14:34:56 CEST+0200".
Do note the following from the pytz manual:
The preferred way of dealing with times is to always work in UTC, converting to localtime only when generating output to be read by humans.
So keep that in mind as you write your code: do the conversion to local time once and once only, and you'll have a much easier time doing, say, comparisons between two timestamps correctly.
Update: Here are a couple of videos you may find useful:
What you need to know about datetimes, a PyCon 2012 presentation by Taavi Burns (30 minutes)
Drive-in Double Header: Datetimes and Log Analysis, a two-part presentation. (Caution: annoying buzz in the video, but I couldn't find a copy with better sound). The first part is the "What you need to know about datetimes" presentation I linked just above, and the second part has some practical tips for parsing log files and doing useful things with them. (50 minutes)
Update 2: The convert_UTC_to_epoch() function you mention in your updated question (which I've reproduced below) is returning local time, not UTC:
def convert_UTC_to_epoch(timestamp):
tz_UTC = pytz.timezone('UTC')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime.strptime(timestamp, time_format)
aware_timestamp = tz_UTC.localize(naive_timestamp)
epoch = aware_timestamp.strftime("%s")
return (int) (epoch)
The problem is that you're using strftime("%s"), which is undocumented and is returning the wrong result. Python doesn't support the %s parameter, but it appears to work because it gets passed to your system's strftime() function, which does support the %s parameter -- but it returns local time! You're taking a UTC timestamp and parsing it as local time, which is why it's an hour off. (The mystery is why it isn't two hours off -- isn't Norway in daylight savings time right now? Shouldn't you be at UTC+2?)
As you can see from the interactive Python session below, I'm in the UTC+7 timezone and your convert_UTC_to_epoch() function is seven hours off for me.
# Current time is 02:42 UTC on June 10th 2014, 09:42 local time
>>> time.timezone
-25200
>>> time.time() - convert_UTC_to_epoch("2014-06-10 02:42:00")
25204.16531395912
>>> time.time() + time.timezone - convert_UTC_to_epoch("2014-06-10 02:42:00")
6.813306093215942
The strftime("%s") call is interpreting 02:42 on June 10th as being in local time, which would be 19:42 UTC on June 9th. Subtracting 19:42 UTC on June 9th from 02:42 UTC June 10th (which is what time.time() returns) gives a difference of seven hours. See Convert python datetime to epoch with strftime for more details on why you should never use strftime("%s").
(By the way, if you saw what I had previously written under the heading "Update 2", where I claimed that time.time() was returning local time, ignore that -- I got it wrong. I was fooled at first by the strftime("%s") bug just like you were.)
You can use the time and datetime modules:
import time, datetime
date = "14-05-07 12:14:16" #Change to whatever date you want
date = time.strptime(date, "%y-%m-%d %H:%M:%S")
epoch = datetime.datetime.fromtimestamp(time.mktime(date)).strftime('%s')
This runs as:
>>> import time, datetime
>>> date = "14-05-07 12:14:16"
>>> date = time.strptime(date, "%y-%m-%d %H:%M:%S")
>>> epoch = datetime.datetime.fromtimestamp(time.mktime(date)).strftime('%s')
>>> epoch
'1399490056'
>>>

find Last mid-night time stamp in python

I want to find the last mid-night time stamp (only input is current timestamp). What is the best way?
I am writing python script for a global mobile application. The user request with the current timestamp, and in server side I want to find the last mid-night timestamp of the user with out affect time zone parameters.
I searched for it, I got a solution
import time
etime = int(time.time())
midnight = (etime - (etime % 86400)) + time.altzon
Its worked for me. But I am confused with time.altzon function, Is it create any problem for the users in different timezones.
To get the midnight timestamp of the client(mobile) you need to know the client's timezone.
from datetime import datetime
import pytz # pip install pytz
fmt = '%Y-%m-%d %H:%M:%S %Z%z'
tz = pytz.timezone("America/New_York") # supply client's timezone here
# Get correct date for the midnight using given timezone.
# due to we are interested only in midnight we can:
# 1. ignore ambiguity when local time repeats itself during DST change e.g.,
# 2012-04-01 02:30:00 EST+1100 and
# 2012-04-01 02:30:00 EST+1000
# otherwise we should have started with UTC time
# 2. rely on .now(tz) to choose timezone correctly (dst/no dst)
now = datetime.now(tz)
print(now.strftime(fmt))
# Get midnight in the correct timezone (taking into account DST)
midnight = tz.localize(now.replace(hour=0, minute=0, second=0, microsecond=0, tzinfo=None),
is_dst=None)
print(midnight.strftime(fmt))
# Convert to UTC (no need to call `tz.normalize()` due to UTC has no DST transitions)
dt = midnight.astimezone(pytz.utc)
print(dt.strftime(fmt))
# Get POSIX timestamp
print((dt - datetime(1970,1,1, tzinfo=pytz.utc)).total_seconds())
Output
2012-08-09 08:46:29 EDT-0400
2012-08-09 00:00:00 EDT-0400
2012-08-09 04:00:00 UTC+0000
1344484800.0
Note: on my machine #phihag's answer produces 1344470400.0 that is different from the above (my machine is not in New York).

Categories