We Keep Coding

When subtracting a datetime that was converted from string, total_seconds() is wrong

The total_seconds() is incorrect when I do this:
from datetime import timedelta, datetime
from pytz import timezone
timezone = timezone('Australia/Sydney')
startDate = datetime.now(timezone)
dateStr = '2020-05-18 20:12:30' # In our brain we know this time is in Sydney Time
endDate = datetime.strptime(dateStr, '%Y-%m-%d %H:%M:%S').replace(tzinfo=timezone)
diff = endDate - startDate
print(diff.total_seconds()) # incorrect answer
When both datetime objects are datetime objects originally, and you substract them, they are right
from datetime import timedelta, datetime
from pytz import timezone
timezone = timezone('Australia/Sydney')
startDate = datetime.now(timezone)
endDate = datetime.now(timezone) + timedelta(hours=2, seconds=32)
diff = endDate - startDate
print(diff.total_seconds()) # correct answer
How can I fix my issue ?

So it seems like all things in the horrible world of date-times, timezones and offsets this is one of these weird and wonderful things. the issue seems to stem from the fact that pytz.timezone will return a timezone object with several timezones.
{
(datetime.timedelta(seconds=36300), datetime.timedelta(0), 'LMT'): <DstTzInfo 'Australia/Sydney' LMT+10:05:00 STD>,
(datetime.timedelta(seconds=36000), datetime.timedelta(0), 'AEST'): <DstTzInfo 'Australia/Sydney' AEST+10:00:00 STD>,
(datetime.timedelta(seconds=39600), datetime.timedelta(seconds=3600), 'AEDT'): <DstTzInfo 'Australia/Sydney' AEDT+11:00:00 DST>
}
It seems when you are passing the timezone to the now method it's picking the timezone from your choice of 3 based on probably some local TZINFO in your setup. However, when passing the timezone to replace, it's just picking the LMT which is different by 300. A quick mention about LMT:
Local Mean Time Today: While Local Mean Time does not directly
determine civil time these days, it is still used to make sure our
clocks follow the Sun as closely as possible. UT1, a version of
Universal Time is the Local Mean Time at the prime meridian in
Greenwich, London. It is one of the components used to calculate
Coordinated Universal Time (UTC), the time scale used to determine
local times worldwide.
LMT is also used by astronomers around the world to time their
observations.
Essentially your issue spans from datetime.now() acting on the local timezone and datetime.replace() acting on the LMT timezone. So as I mentioned in my post create your dates consistently either create them both via replace (although you will still be off by 5 mins in terms of actual time, the difference will be correct.)
UPDATE
If you want both datetime objects to be in local Sydney time then you can create your stardate as you did before using datetime.now(). But you should create your end date from your timezone objects asking it to localize it for you like.
from datetime import datetime
from pytz import timezone
timezone = timezone('Australia/Sydney')
startDate = datetime.now(timezone)
dateStr = '2020-05-18 18:52:30' # In our brain we know this time is in Sydney Time
endDate = timezone.localize(datetime.strptime(dateStr, '%Y-%m-%d %H:%M:%S'))
print(startDate, endDate, sep="\n")
diff = endDate - startDate
print(diff.total_seconds())
OUTPUT
2020-05-18 18:51:24.722614+10:00
2020-05-18 18:52:30+10:00
65.277386

'Europe/Madrid' timezone doesn't match 'Etc/GMT+1'

I'm trying to convert a UTC timestamp to one in the Spanish timezone.
>>> import datetime as dt
>>> import pytz
>>> today = dt.datetime.utcfromtimestamp(1573516800)
datetime.datetime(2019, 11, 12, 0, 0)
>>> today.replace(tzinfo=pytz.timezone('Europe/Madrid')).timestamp()
1573517700.0
>>> today.replace(tzinfo=pytz.timezone('Etc/GMT+1')).timestamp()
1573520400.0
I'm surprised that I get different results for Europe/Madrid and Etc/GMT+1. Why is this? Should Europe/Madrid be used differently, or it is possibly a bug?

A few things:
Europe/Madrid is UTC+1 during standard time, and UTC+2 during summer time (aka daylight saving time).
Etc/GMT+1 is UTC-1 for the entire year. Note the sign is opposite what you might expect. See the explanation in the tzdata sources, and on Wikipedia.
Since Madrid is on UTC+1 on the date you gave, you would get the same result for that date if you used Etc/GMT-1. However, I don't recommend that, as you would then later get the wrong result for a date during summer time.
The Etc/GMT±X zones are intended to be used primarily for non-localizable scenarios such as tracking time onboard ships at sea - not for populated locations on land.
As Mason's answer showed, you should be using the localize function rather than replace to assign a time zone. This is covered in the pytz documentation.

UTC Timestamp: The number of seconds since January 1st, 1970 at UTC.
Python datetime: A nice way of seeing this time that is user friendly
The UTC timestamp is not effected by timezones, but the datetime is.
This code takes the given timestamp and converts it to a UTC datetime and a Europe/Madrid timezone.
import datetime as dt
import pytz
# define the old and new timezones
old_timezone = pytz.timezone("UTC")
new_timezone = pytz.timezone("Europe/Madrid")
# get an 'offset-aware' datetime
today = dt.datetime.utcfromtimestamp(1573516800)
my_datetime = old_timezone.localize(today)
# returns datetime in the new timezone
my_datetime_in_new_timezone = my_datetime.astimezone(new_timezone)
print("Old:", str(my_datetime), "\nNew:", str(my_datetime_in_new_timezone), "\nDifference:",
str(my_datetime - my_datetime_in_new_timezone))
Output:
Old: 2019-11-12 00:00:00+00:00
New: 2019-11-12 01:00:00+01:00
Difference: 0:00:00
Code adapted from:
Python: How do you convert datetime/timestamp from one timezone to another timezone?

Python - Standard Time to Daylight Savings Time [duplicate]

How do I check if daylight saving time is in effect?

You can use time.localtime and look at the tm_isdst flag in the return value.
>>> import time
>>> time.localtime()
(2010, 5, 21, 21, 48, 51, 4, 141, 0)
>>> _.tm_isdst
0
Using time.localtime(), you can ask the same question for any arbitrary time to see whether DST would be (or was) in effect for your current time zone.

The accepted answer is fine if you are running code on your laptop, but most
python applications are running on a server using UTC as local time, so they
will NEVER be in daylight savings time according to the accepted answer.
The second problem is that different regions implement daylight savings on
different days and times. So even if you have an unambiguous time, such as
datetime.utcnow(), it could be daylight savings time in one timezone but not
in another.
The best we can do then, is tell whether a given time occurs during DST for a
specific timezone, and the best method I can find for doing it has already
been implemtend by pytz localize function and we can use it to get a
pretty good answer that works both on our laptop and on a server.
import pytz
from datetime import datetime
def is_dst(dt=None, timezone="UTC"):
if dt is None:
dt = datetime.utcnow()
timezone = pytz.timezone(timezone)
timezone_aware_date = timezone.localize(dt, is_dst=None)
return timezone_aware_date.tzinfo._dst.seconds != 0
Some examples
>>> is_dst() # it is never DST in UTC
False
>>> is_dst(datetime(2019, 1, 1), timezone="US/Pacific")
False
>>> is_dst(datetime(2019, 4, 1), timezone="US/Pacific")
True
>>> is_dst(datetime(2019, 3, 10, 2), timezone="US/Pacific")
NonExistentTimeError
>>> is_dst(datetime(2019, 11, 3, 1), timezone="US/Pacific")
AmbiguousTimeError
In our is_dst function, we specified is_dst=None as a parameter to
timezone.localize, which will cause nonsense times to throw errors. You
could use is_dst=False to ignore these errors and return False for those
times.

Assuming you want to perform this on a datetime
Use pytz to make it timezone aware and then check its dst property:
import datetime
import pytz
def is_dst(dt,timeZone):
aware_dt = timeZone.localize(dt)
return aware_dt.dst() != datetime.timedelta(0,0)
timeZone = pytz.timezone("Europe/London")
dt = datetime.datetime(2019,8,2)
is_dst(dt,timeZone)
True
dt = datetime.datetime(2019,2,2)
is_dst(dt,timeZone)
False

I would have posted this as a comment to the answer by #mehtunguh above, but my current reputation level does not allow me to comment.
I think there may be an issue with the is_dst function as written when the dt argument is omitted.
When the dt argument is omitted, dt is set to datetime.utcnow() which returns a naive datetime representing the current UTC time. When that is passed to pytz.localize, the resulting localized time is not the current time in the specified time zone, but rather the local time that has the same hour, minute, second, as the current UTC time.
So, for example, as I write this it is 10:50 AM EST in the US/Eastern time zone, and datetime.utcnow() returns a datetime value with hour=15 and minute=50. As written, when invoked as is_dst(timezone='US/Eastern'), is_dst is not checking whether the current local time of 10:50 AM EST is during daylight saving time, it is checking whether 3:50 PM EST is during daylight saving time.
I think is_dst should perhaps be coded as follows:
import datetime
import pytz
def is_dst(dt=None, timezone='UTC'):
timezone = pytz.timezone(timezone)
if dt is None:
dt = datetime.datetime.now(datetime.timezone.utc)
if dt.tzinfo is None:
tz_aware_dt = timezone.localize(dt, is_dst=None)
else:
tz_aware_dt = dt.astimezone(timezone)
return tz_aware_dt.tzinfo._dst.seconds != 0
This version allows passing either a naive datetime value or a timezone-aware datetime value as the dt argument. When the dt argument is omitted, it uses a timezone-aware version of the current UTC time so that when that gets localized to the specified timezone it represents the current time in that timezone.

None of the above helped me so I found my own workaround.
I relate to the logic implemented in https://gist.github.com/dpapathanasiou/09bd2885813038d7d3eb while there's still a problem, it doesn't work in real life apparently :(
Currently I'm in Israel and here we move the clock in the end of the month,
while in Australia they have already moved the clock.
All the codes return True for both Asia/Jerusalem and Australia/Sydney.
Eventually I used an external 3rd party API - https://worldtimeapi.org/ - by which I analyse whether the utc_offset is 11 hours (rather that 10:05).
from requests import get as Get
is_dst = True
try:
tz_check = Get('https://worldtimeapi.org/api/timezone/Australia/Sydney')
is_dst = tz_check.json().get('utc_offset') == '+11:00'
except Exception as e:
print('!!! Error getting timezone', e)
I agree this is a private case, but I hope this can help someone :)

Expanding #Greg Hewgill's answer above, plus coping with local timezone (with help of pip install tzlocal), you get:
import time
from datetime import datetime, timedelta
from tzlocal import get_localzone
def to_local(dt):
"""From any timezone to local datetime - also cope with DST"""
localtime = time.localtime()
if localtime.tm_isdst:
utctime = time.gmtime()
hours_delta = timedelta(hours=(localtime.tm_hour - utctime.tm_hour))
dt = dt - hours_delta
return dt.replace(tzinfo=get_localzone())

I'm from the UK and this is how I handled my server returning the wrong time for half the year:
import pytz
from typing import Optional
from datetime import datetime
class BritishTime(datetime):
timezone = pytz.timezone('Europe/London')
#classmethod
def dst(cls, dt: Optional[datetime] = None):
dt = dt if dt is not None else cls.now()
return cls.timezone.dst(dt)
Now if I create a datetime object with BritishTime, it has the dst method which I can use to both check and update the time, something like this:
def get_correct_time(timestamp):
updated = BritishTime.fromtimestamp(timestamp)
return updated + updated.dst()
Works pretty well.

Below, I'll show yet another way, but this could never be overemphasized:
DST rules are magic (determined by local law) and can change from year to year.
# IMHO a really nasty magic.
(From the Python doc for the time module.)
In Unix, yet another way is to simply invoke the date command, if you are familiar with the target timezone:
import subprocess
# First Sun of 2022-03
subprocess.run(['date', '--date=2022-03-06 00:00:00'], env={"TZ": "US/Eastern"})
# Next Sun, 00:00 (
subprocess.run(['date', '--date=2022-03-13 00:00:00'], env={"TZ": "US/Eastern"})
# Ditto, 12:00
subprocess.run(['date', '--date=2022-03-13 12:00:00'], env={"TZ": "US/Eastern"})
The output:
Thu Mar 10 12:00:00 EST 2022
Sun Mar 13 00:00:00 EST 2022
Sun Mar 13 12:00:00 EDT 2022
EST is for the not-DST season, and EDT for the DST.
Anyway the transition period between the DST/non-DST is really tricky.

Python - Convert ISO 8601 to BST time

So basically I have learned a bit with ISO 8601 where the format is
"2018-07-06T07:00:00.000"
and basically what I have achieved is that I starting of to change the ISO to a more formal timestamp which is:
etatime = str(datetime.datetime.strptime("2018-07-06T07:00:00.000", "%Y-%m-%dT%H:%M:%S.%f"))
which will give an output of:
2018-07-06 07:00:00
However I noticed the time is 1 hour behind the BST (British time) which should be added one hour.
My question is, is there possible to go from (2018-07-06T07:00:00.000) to (2018-07-06 08:00:00 BST)?

Assumptions: the input represents a UTC timestamp, and you want to localise that to London time. You probably do not want to localise it to BST time, since BST is the DST variation of GMT, and an actual location like London will switch between BST and GMT depending on the time of year. You'll want to install the pytz module.
from datetime import datetime, timezone
import pytz
date = '2018-07-06T07:00:00.000'
utc_date = datetime.strptime(date, '%Y-%m-%dT%H:%M:%S.%f').replace(tzinfo=timezone.utc)
london_date = utc_date.astimezone(pytz.timezone('Europe/London'))
datetime.datetime(2018, 7, 6, 8, 0, tzinfo=<DstTzInfo 'Europe/London' BST+1:00:00 DST>)
strptime gives you a naïve datetime object (without timezone information), .replace gives you an aware datetime object (with timezone information), which then enables you to simply convert that to a different timezone.

One suggestion is that you can use the timedelta function from datetime module:
from datetime import datetime, timedelta
etatime = datetime.strptime("2018-07-06T07:00:00.000", "%Y-%m-%dT%H:%M:%S.%f")
# Before adding one hour
print(etatime)
etatime = etatime + timedelta(hours=1)
# After adding one hour
print(etatime)
Output:
2018-07-06 07:00:00
2018-07-06 08:00:00

python converting string in localtime to UTC epoch timestamp

I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
I am trying to convert them into epoch timestamps for UTC time.
I wrote the following function:
def ymdhms_timezone_dst_to_epoch(input_str, tz="US/Eastern"):
print(input_str)
dt = datetime.datetime.fromtimestamp(time.mktime(time.strptime(input_str,'%Y-%m-%d %H:%M:%S')))
local_dt = pytz.timezone(tz).localize(dt)
print(local_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
utc_dt = local_dt.astimezone(pytz.utc)
print(utc_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
e = int(utc_dt.strftime("%s"))
print(e)
return e
Given string `2015-04-20 21:12:07` this prints:
2015-04-20 21:12:07
2015-04-20 21:12:07 EDT-0400 #<- so far so good?
2015-04-21 01:12:07 UTC+0000 #<- so far so good?
1429596727
which looks ok up to the epoch timestamp. But http://www.epochconverter.com/epoch/timezones.php?epoch=1429596727 says it should mao to
Greenwich Mean Time Apr 21 2015 06:12:07 UTC.
What is wrong?

I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
A portable way is to use pytz:
#!/usr/bin/env python
from datetime import datetime
import pytz # $ pip install pytz
naive_dt = datetime.strptime('2015-04-20 21:12:07', '%Y-%m-%d %H:%M:%S')
tz = pytz.timezone('US/Eastern')
eastern_dt = tz.normalize(tz.localize(naive_dt))
print(eastern_dt)
# -> 2015-04-20 21:12:07-04:00
I am trying to convert them into epoch timestamps for UTC time.
timestamp = (eastern_dt - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
# -> 1429578727.0
See Converting datetime.date to UTC timestamp in Python.
There are multiple issues in your code:
time.mktime() may return a wrong result for ambiguous input time (50% chance) e.g., during "fall back" DST transition in the Fall
time.mktime() and datetime.fromtimestamp() may fail for past/future dates if they have no access to a historical timezone database on a system (notably, Windows)
localize(dt) may return a wrong result for ambiguous or non-existent time i.e., during DST transitions. If you know that the time corresponds to the summer time then use is_dst=True. tz.normalize() is necessary here, to adjust possible non-existing times in the input
utc_dt.strftime("%s") is not portable and it does not respect tzinfo object. It interprets input as a local time i.e., it returns a wrong result unless your local timezone is UTC.
Can I just always set is_dst=True?
You can, if you don't mind getting imprecise results for ambiguous or non-existent times e.g., there is DST transition in the Fall in America/New_York time zone:
>>> from datetime import datetime
>>> import pytz # $ pip install pytz
>>> tz = pytz.timezone('America/New_York')
>>> ambiguous_time = datetime(2015, 11, 1, 1, 30)
>>> time_fmt = '%Y-%m-%d %H:%M:%S%z (%Z)'
>>> tz.localize(ambiguous_time).strftime(time_fmt)
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=False).strftime(time_fmt) # same
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=True).strftime(time_fmt) # different
'2015-11-01 01:30:00-0400 (EDT)'
>>> tz.localize(ambiguous_time, is_dst=None).strftime(time_fmt)
Traceback (most recent call last):
...
pytz.exceptions.AmbiguousTimeError: 2015-11-01 01:30:00
The clocks are turned back at 2a.m. on the first Sunday in November:
is_dst disambiguation flag may have three values:
False -- default, assume the winter time
True -- assume the summer time
None -- raise an exception for ambiguous/non-existent times.
is_dst value is ignored for existing unique local times.
Here's a plot from PEP 0495 -- Local Time Disambiguation that illustrates the DST transition:
The local time repeats itself twice in the fold (summer time -- before the fold, winter time -- after).
To be able to disambiguate the local time automatically, you need some additional info e.g., if you read a series of local times then it may help if you know that they are sorted: Parsing of Ordered Timestamps in Local Time (to UTC) While Observing Daylight Saving Time.

First of all '%s' is not supported on all platforms , its actually working for you because your platform C library’s strftime() function (that is called by Python) supports it. This function is what is causing the issue most probably, I am guessing its not timezone aware , hence when taking difference from epoch time it is using your local timezone, which is most probably EST(?)
Instead of relying on '%s' , which only works in few platforms (linux, I believe) , you should manually subtract the datetime you got from epoch (1970/1/1 00:00:00) to get the actual seconds since epoch . Example -
e = (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
Demo -
>>> (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
1429578727.0
This correctly corresponds to the date-time you get.

I don't exactly know why but you have to remove the timezone info from your utc_dt before using %s to print it.
e = int(utc_dt.replace(tzinfo=None).strftime("%s"))
print(e)
return e