We Keep Coding

'Europe/Madrid' timezone doesn't match 'Etc/GMT+1'

I'm trying to convert a UTC timestamp to one in the Spanish timezone.
>>> import datetime as dt
>>> import pytz
>>> today = dt.datetime.utcfromtimestamp(1573516800)
datetime.datetime(2019, 11, 12, 0, 0)
>>> today.replace(tzinfo=pytz.timezone('Europe/Madrid')).timestamp()
1573517700.0
>>> today.replace(tzinfo=pytz.timezone('Etc/GMT+1')).timestamp()
1573520400.0
I'm surprised that I get different results for Europe/Madrid and Etc/GMT+1. Why is this? Should Europe/Madrid be used differently, or it is possibly a bug?

A few things:
Europe/Madrid is UTC+1 during standard time, and UTC+2 during summer time (aka daylight saving time).
Etc/GMT+1 is UTC-1 for the entire year. Note the sign is opposite what you might expect. See the explanation in the tzdata sources, and on Wikipedia.
Since Madrid is on UTC+1 on the date you gave, you would get the same result for that date if you used Etc/GMT-1. However, I don't recommend that, as you would then later get the wrong result for a date during summer time.
The Etc/GMT±X zones are intended to be used primarily for non-localizable scenarios such as tracking time onboard ships at sea - not for populated locations on land.
As Mason's answer showed, you should be using the localize function rather than replace to assign a time zone. This is covered in the pytz documentation.

UTC Timestamp: The number of seconds since January 1st, 1970 at UTC.
Python datetime: A nice way of seeing this time that is user friendly
The UTC timestamp is not effected by timezones, but the datetime is.
This code takes the given timestamp and converts it to a UTC datetime and a Europe/Madrid timezone.
import datetime as dt
import pytz
# define the old and new timezones
old_timezone = pytz.timezone("UTC")
new_timezone = pytz.timezone("Europe/Madrid")
# get an 'offset-aware' datetime
today = dt.datetime.utcfromtimestamp(1573516800)
my_datetime = old_timezone.localize(today)
# returns datetime in the new timezone
my_datetime_in_new_timezone = my_datetime.astimezone(new_timezone)
print("Old:", str(my_datetime), "\nNew:", str(my_datetime_in_new_timezone), "\nDifference:",
str(my_datetime - my_datetime_in_new_timezone))
Output:
Old: 2019-11-12 00:00:00+00:00
New: 2019-11-12 01:00:00+01:00
Difference: 0:00:00
Code adapted from:
Python: How do you convert datetime/timestamp from one timezone to another timezone?

Python - Standard Time to Daylight Savings Time [duplicate]

How do I check if daylight saving time is in effect?

You can use time.localtime and look at the tm_isdst flag in the return value.
>>> import time
>>> time.localtime()
(2010, 5, 21, 21, 48, 51, 4, 141, 0)
>>> _.tm_isdst
0
Using time.localtime(), you can ask the same question for any arbitrary time to see whether DST would be (or was) in effect for your current time zone.

The accepted answer is fine if you are running code on your laptop, but most
python applications are running on a server using UTC as local time, so they
will NEVER be in daylight savings time according to the accepted answer.
The second problem is that different regions implement daylight savings on
different days and times. So even if you have an unambiguous time, such as
datetime.utcnow(), it could be daylight savings time in one timezone but not
in another.
The best we can do then, is tell whether a given time occurs during DST for a
specific timezone, and the best method I can find for doing it has already
been implemtend by pytz localize function and we can use it to get a
pretty good answer that works both on our laptop and on a server.
import pytz
from datetime import datetime
def is_dst(dt=None, timezone="UTC"):
if dt is None:
dt = datetime.utcnow()
timezone = pytz.timezone(timezone)
timezone_aware_date = timezone.localize(dt, is_dst=None)
return timezone_aware_date.tzinfo._dst.seconds != 0
Some examples
>>> is_dst() # it is never DST in UTC
False
>>> is_dst(datetime(2019, 1, 1), timezone="US/Pacific")
False
>>> is_dst(datetime(2019, 4, 1), timezone="US/Pacific")
True
>>> is_dst(datetime(2019, 3, 10, 2), timezone="US/Pacific")
NonExistentTimeError
>>> is_dst(datetime(2019, 11, 3, 1), timezone="US/Pacific")
AmbiguousTimeError
In our is_dst function, we specified is_dst=None as a parameter to
timezone.localize, which will cause nonsense times to throw errors. You
could use is_dst=False to ignore these errors and return False for those
times.

Assuming you want to perform this on a datetime
Use pytz to make it timezone aware and then check its dst property:
import datetime
import pytz
def is_dst(dt,timeZone):
aware_dt = timeZone.localize(dt)
return aware_dt.dst() != datetime.timedelta(0,0)
timeZone = pytz.timezone("Europe/London")
dt = datetime.datetime(2019,8,2)
is_dst(dt,timeZone)
True
dt = datetime.datetime(2019,2,2)
is_dst(dt,timeZone)
False

I would have posted this as a comment to the answer by #mehtunguh above, but my current reputation level does not allow me to comment.
I think there may be an issue with the is_dst function as written when the dt argument is omitted.
When the dt argument is omitted, dt is set to datetime.utcnow() which returns a naive datetime representing the current UTC time. When that is passed to pytz.localize, the resulting localized time is not the current time in the specified time zone, but rather the local time that has the same hour, minute, second, as the current UTC time.
So, for example, as I write this it is 10:50 AM EST in the US/Eastern time zone, and datetime.utcnow() returns a datetime value with hour=15 and minute=50. As written, when invoked as is_dst(timezone='US/Eastern'), is_dst is not checking whether the current local time of 10:50 AM EST is during daylight saving time, it is checking whether 3:50 PM EST is during daylight saving time.
I think is_dst should perhaps be coded as follows:
import datetime
import pytz
def is_dst(dt=None, timezone='UTC'):
timezone = pytz.timezone(timezone)
if dt is None:
dt = datetime.datetime.now(datetime.timezone.utc)
if dt.tzinfo is None:
tz_aware_dt = timezone.localize(dt, is_dst=None)
else:
tz_aware_dt = dt.astimezone(timezone)
return tz_aware_dt.tzinfo._dst.seconds != 0
This version allows passing either a naive datetime value or a timezone-aware datetime value as the dt argument. When the dt argument is omitted, it uses a timezone-aware version of the current UTC time so that when that gets localized to the specified timezone it represents the current time in that timezone.

None of the above helped me so I found my own workaround.
I relate to the logic implemented in https://gist.github.com/dpapathanasiou/09bd2885813038d7d3eb while there's still a problem, it doesn't work in real life apparently :(
Currently I'm in Israel and here we move the clock in the end of the month,
while in Australia they have already moved the clock.
All the codes return True for both Asia/Jerusalem and Australia/Sydney.
Eventually I used an external 3rd party API - https://worldtimeapi.org/ - by which I analyse whether the utc_offset is 11 hours (rather that 10:05).
from requests import get as Get
is_dst = True
try:
tz_check = Get('https://worldtimeapi.org/api/timezone/Australia/Sydney')
is_dst = tz_check.json().get('utc_offset') == '+11:00'
except Exception as e:
print('!!! Error getting timezone', e)
I agree this is a private case, but I hope this can help someone :)

Expanding #Greg Hewgill's answer above, plus coping with local timezone (with help of pip install tzlocal), you get:
import time
from datetime import datetime, timedelta
from tzlocal import get_localzone
def to_local(dt):
"""From any timezone to local datetime - also cope with DST"""
localtime = time.localtime()
if localtime.tm_isdst:
utctime = time.gmtime()
hours_delta = timedelta(hours=(localtime.tm_hour - utctime.tm_hour))
dt = dt - hours_delta
return dt.replace(tzinfo=get_localzone())

I'm from the UK and this is how I handled my server returning the wrong time for half the year:
import pytz
from typing import Optional
from datetime import datetime
class BritishTime(datetime):
timezone = pytz.timezone('Europe/London')
#classmethod
def dst(cls, dt: Optional[datetime] = None):
dt = dt if dt is not None else cls.now()
return cls.timezone.dst(dt)
Now if I create a datetime object with BritishTime, it has the dst method which I can use to both check and update the time, something like this:
def get_correct_time(timestamp):
updated = BritishTime.fromtimestamp(timestamp)
return updated + updated.dst()
Works pretty well.

Below, I'll show yet another way, but this could never be overemphasized:
DST rules are magic (determined by local law) and can change from year to year.
# IMHO a really nasty magic.
(From the Python doc for the time module.)
In Unix, yet another way is to simply invoke the date command, if you are familiar with the target timezone:
import subprocess
# First Sun of 2022-03
subprocess.run(['date', '--date=2022-03-06 00:00:00'], env={"TZ": "US/Eastern"})
# Next Sun, 00:00 (
subprocess.run(['date', '--date=2022-03-13 00:00:00'], env={"TZ": "US/Eastern"})
# Ditto, 12:00
subprocess.run(['date', '--date=2022-03-13 12:00:00'], env={"TZ": "US/Eastern"})
The output:
Thu Mar 10 12:00:00 EST 2022
Sun Mar 13 00:00:00 EST 2022
Sun Mar 13 12:00:00 EDT 2022
EST is for the not-DST season, and EDT for the DST.
Anyway the transition period between the DST/non-DST is really tricky.

Is there a "local" timestamp?

I can convert from an UTC timestamp e.g. 1417392000 to a local datetime object including daylight saving time, but when I try and convert the local datetime object to a "local timestamp" then I get the same UTC timestamp as before.
Am I thinking in circles and they are the same? I was supposed to save "local timestamp" from incoming UTC timestamp.
This is my code
print("UTC timestamp %d" % hour[0])
day = self.get_day(hour)
month = self.get_month(hour)
year = self.get_year(hour)
tz = pytz.timezone('Europe/Stockholm')
utc_dt = datetime.utcfromtimestamp(int(hour[0])).replace(tzinfo=pytz.utc)
print("UTC datetime %s" % utc_dt)
dt = tz.normalize(utc_dt.astimezone(tz))
print("STO datetime %s" % dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
print("STO ts %d" % int(time.mktime(dt.timetuple())))
print("STO timestamp %d" % utc_dt.astimezone(tz).timestamp())
day = int(dt.strftime('%d'))
month = int(dt.strftime('%m'))
year = int(dt.strftime('%Y'))
Output
UTC timestamp 1417395600
UTC datetime 2014-12-01 01:00:00+00:00
STO datetime 2014-12-01 02:00:00 CET+0100
STO ts 1417395600
STO timestamp 1417395600
All "timestamps" (i.e. integer representations of the time) are the same. Is it possible to make a "local timestamp" ? The data type is supposed to be a timestamp that is a number and in local time.

As per Wikipedia
Unix time (also known as POSIX time or Epoch time) is a system for
describing instants in time, defined as the number of seconds that
have elapsed since 00:00:00 Coordinated Universal Time (UTC),
Thursday, 1 January 1970
So regardless of what timezone you're on, the epoch will always be calculated in the UTC timezone.
For display purposes, you can convert the timestamp to a local time, but as otherwise the internal representation of the epoch will always be in the UTC timezone

Formal timestamps
The Unix timestamp typically refers to the number of seconds since the epoch in UTC. This value is invariant to timezone. It allows global event ordering but loses timezone information.
Preserving timezone
To preserve timezone information, a standardized format is RFC3339, Date and Time on the Internet: Timestamps. This is just a standardized formatting that encodes date+time+timezone. Some examples:
1985-04-12T23:20:50.52Z
1996-12-19T16:39:57-08:00
1990-12-31T23:59:60Z
1990-12-31T15:59:60-08:00
Normalizing for timezone without preservation of timezone
However, it may depend on your requirements. I once wanted to record some events relative to local-time-of-day and did not mind losing timezone information. I normalized the timestamp with respect to 1970-01-01T00:00:00 in the local timezone. I am a little sheepish about this now as I think it may too easily cause confusion.
import time
# Number of seconds since 1970-01-01T00:00+LTZ (current timezone).
# unix timestamp - timezone offset in seconds
timestamp_localized = time.time() - time.mktime(time.gmtime(0))
However this syntax can be simplified, perhaps at the loss of clarity, by noticing that Python has some localtime and UTC specific functions.
import time
import calendar
# Number of seconds since 1970-01-01T00:00+LTZ (current timezone).
# Interpret local date and time as if it were UTC
timestamp_localized = calendar.timegm(time.localtime())
The difference between these two is that calendar conversion truncates to the second while the difference calculation includes a fractional second.

Python: Converting string to timestamp with microseconds

I would like to convert string date format to timestamp with microseconds
I try the following but not giving expected result:
"""input string date -> 2014-08-01 04:41:52,117
expected result -> 1410748201.117"""
import time
import datetime
myDate = "2014-08-01 04:41:52,117"
timestamp = time.mktime(datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timetuple())
print timestamp
> 1410748201.0
Where did the milliseconds go?

There is no slot for the microseconds component in a time tuple:
>>> import time
>>> import datetime
>>> myDate = "2014-08-01 04:41:52,117"
>>> datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timetuple()
time.struct_time(tm_year=2014, tm_mon=8, tm_mday=1, tm_hour=4, tm_min=41, tm_sec=52, tm_wday=4, tm_yday=213, tm_isdst=-1)
You'll have to add those manually:
>>> dt = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
>>> time.mktime(dt.timetuple()) + (dt.microsecond / 1000000.0)
1406864512.117
The other method you could follow is to produce a timedelta() object relative to the epoch, then get the timestamp with the timedelta.total_seconds() method:
epoch = datetime.datetime.fromtimestamp(0)
(dt - epoch).total_seconds()
The use of a local time epoch is quite deliberate since you have a naive (not timezone-aware) datetime value. This method can be inaccurate based on the history of your local timezone however, see J.F. Sebastian's comment. You'd have to convert the naive datetime value to a timezone-aware datetime value first using your local timezone before subtracting a timezone-aware epoch.
As such, it is easier to stick to the timetuple() + microseconds approach.
Demo:
>>> dt = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
>>> epoch = datetime.datetime.fromtimestamp(0)
>>> (dt - epoch).total_seconds()
1406864512.117

In Python 3.4 and later you can use
timestamp = datetime.datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f").timestamp()
This doesn't require importing the time module. It also uses less steps so it should be faster. For older versions of python the other provided answers are probably your best option.
However, the resulting timestamp will interpret myDate in local time, rather than UTC, which may cause issues if myDate was given in UTC

Where did the milliseconds go?
It is the easy part. .timetuple() call drops them. You could add them back using .microsecond attribute. The datetime.timestamp() method from the standard library works that way for naive datetime objects:
def timestamp(self):
"Return POSIX timestamp as float"
if self._tzinfo is None:
return _time.mktime((self.year, self.month, self.day,
self.hour, self.minute, self.second,
-1, -1, -1)) + self.microsecond / 1e6
else:
return (self - _EPOCH).total_seconds()
It is enough if possible ~1 hour errors could be ignored in your case. I assume that you want microseconds and therefore you can't ignore ~1 hour time errors silently.
To convert the local time given as a string to the POSIX timestamp correctly is a complex task in general. You could convert the local time to UTC and then get the timestamp from UTC time.
There are two main issues:
local time may be non-existent or ambiguous e.g. during DST transitions the same time may occur twice
UTC offset for the local timezone may be different in the past and therefore a naive: local time minus epoch in local time formula may fail
Both can be solved using the tz database (pytz module in Python):
from datetime import datetime
import pytz # $ pip install pytz
from tzlocal import get_localzone # $ pip install tzlocal
tz = get_localzone() # get pytz timezone corresponding to the local timezone
naive_d = datetime.strptime(myDate, "%Y-%m-%d %H:%M:%S,%f")
# a) raise exception for non-existent or ambiguous times
d = tz.localize(naive_d, is_dst=None)
## b) assume standard time, adjust non-existent times
#d = tz.normalize(tz.localize(naive_d, is_dst=False))
## c) assume DST is in effect, adjust non-existent times
#d = tz.normalize(tz.localize(naive_d, is_dst=True))
timestamp = d - datetime(1970, 1, 1, tzinfo=pytz.utc)
The result is timestamp -- a timedelta object, you can convert it to seconds, milliseconds, etc.
Also different systems may behave differently around/during leap seconds. Most application can ignore that they exist.
In general, it might be simpler to store POSIX timestamps in addition to the local time instead of trying to guess it from the local time.

Convert a UTC time to epoch

I am looking to analyze traffic flow with relation to weather data. The traffic data has a UNIX timestamp (aka epoch), but I am running into trouble with converting the timestamp (in the weather data) to epoch. The problem is that I am in Norway and the UTC timestamp in the weather data isn't in the same timezone as me (GMT+1).
My initial approach
I first tried converting it into epoch and treating the data as if it was in the GMT+1 timezone. Then I compensated by subtracting the difference in number of seconds between UTC and GMT+1.
Problems with the approach
I realize first of all that this approach is very primitive and not very elegant (in fact probably it is at best an ugly hack). However, the biggest problem here is that the difference between UTC and GMT+1 is not constant (due to daylight savings).
Question
Is there any reliable way of turning UTC time to a UNIX time stamp in python (taking into account that my machine is in GMT+1)? The timestamp is in the following format:
Y-m-d HH:MM:SS
Edit:
Tried rmunns' solution:
def convert_UTC_to_epoch(timestamp):
tz_UTC = pytz.timezone('UTC')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime.strptime(timestamp, time_format)
aware_timestamp = tz_UTC.localize(naive_timestamp)
epoch = aware_timestamp.strftime("%s")
return (int) (epoch)
This does not work properly as evidenced below:
#Current time at time of the edit is 15:55:00 UTC on June 9th 2014.
>>> diff = time.time() - convert_UTC_to_epoch("2014-06-09 15:55:00")
>>> diff
3663.25887799263
>>> #This is about an hour off.

The solution was to use the calendar module (inspired from here)
>>>#Quick and dirty demo
>>>print calendar.timegm(datetime.datetime.utcnow().utctimetuple()) - time.time()
>>>-0.6182510852813721
And here is the conversion function:
import calendar, datetime, time
#Timestamp is a datetime object in UTC time
def UTC_time_to_epoch(timestamp):
epoch = calendar.timegm(timestamp.utctimetuple())
return epoch

An alternative, datetime has it's own .strptime() method.
http://en.wikipedia.org/wiki/Unix_time
The Unix epoch is the time 00:00:00 UTC on 1 January 1970 (or 1970-01-01T00:00:00Z ISO 8601).
import datetime
unix_epoch = datetime.datetime(1970, 1, 1)
log_dt = datetime.datetime.strptime("14-05-07 12:14:16", "%y-%m-%d %H:%M:%S")
seconds_from_epoch = (log_dt - unix_epoch).total_seconds()
>>> 1399490056.0

The pytz module will probably help you. It allows you to write code like:
import pytz
import datetime
tz_oslo = pytz.timezone('Europe/Oslo')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime(2014, 6, 4, 12, 34, 56)
# Or:
naive_timestamp = datetime.datetime.strptime("2014-06-04 12:34:56", time_format)
aware_timestamp = tz_oslo.localize(naive_timestamp)
print(aware_timestamp.strftime(time_format + " %Z%z"))
This should print "2014-06-04 14:34:56 CEST+0200".
Do note the following from the pytz manual:
The preferred way of dealing with times is to always work in UTC, converting to localtime only when generating output to be read by humans.
So keep that in mind as you write your code: do the conversion to local time once and once only, and you'll have a much easier time doing, say, comparisons between two timestamps correctly.
Update: Here are a couple of videos you may find useful:
What you need to know about datetimes, a PyCon 2012 presentation by Taavi Burns (30 minutes)
Drive-in Double Header: Datetimes and Log Analysis, a two-part presentation. (Caution: annoying buzz in the video, but I couldn't find a copy with better sound). The first part is the "What you need to know about datetimes" presentation I linked just above, and the second part has some practical tips for parsing log files and doing useful things with them. (50 minutes)
Update 2: The convert_UTC_to_epoch() function you mention in your updated question (which I've reproduced below) is returning local time, not UTC:
def convert_UTC_to_epoch(timestamp):
tz_UTC = pytz.timezone('UTC')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime.strptime(timestamp, time_format)
aware_timestamp = tz_UTC.localize(naive_timestamp)
epoch = aware_timestamp.strftime("%s")
return (int) (epoch)
The problem is that you're using strftime("%s"), which is undocumented and is returning the wrong result. Python doesn't support the %s parameter, but it appears to work because it gets passed to your system's strftime() function, which does support the %s parameter -- but it returns local time! You're taking a UTC timestamp and parsing it as local time, which is why it's an hour off. (The mystery is why it isn't two hours off -- isn't Norway in daylight savings time right now? Shouldn't you be at UTC+2?)
As you can see from the interactive Python session below, I'm in the UTC+7 timezone and your convert_UTC_to_epoch() function is seven hours off for me.
# Current time is 02:42 UTC on June 10th 2014, 09:42 local time
>>> time.timezone
-25200
>>> time.time() - convert_UTC_to_epoch("2014-06-10 02:42:00")
25204.16531395912
>>> time.time() + time.timezone - convert_UTC_to_epoch("2014-06-10 02:42:00")
6.813306093215942
The strftime("%s") call is interpreting 02:42 on June 10th as being in local time, which would be 19:42 UTC on June 9th. Subtracting 19:42 UTC on June 9th from 02:42 UTC June 10th (which is what time.time() returns) gives a difference of seven hours. See Convert python datetime to epoch with strftime for more details on why you should never use strftime("%s").
(By the way, if you saw what I had previously written under the heading "Update 2", where I claimed that time.time() was returning local time, ignore that -- I got it wrong. I was fooled at first by the strftime("%s") bug just like you were.)

You can use the time and datetime modules:
import time, datetime
date = "14-05-07 12:14:16" #Change to whatever date you want
date = time.strptime(date, "%y-%m-%d %H:%M:%S")
epoch = datetime.datetime.fromtimestamp(time.mktime(date)).strftime('%s')
This runs as:
>>> import time, datetime
>>> date = "14-05-07 12:14:16"
>>> date = time.strptime(date, "%y-%m-%d %H:%M:%S")
>>> epoch = datetime.datetime.fromtimestamp(time.mktime(date)).strftime('%s')
>>> epoch
'1399490056'
>>>