We Keep Coding

Python Date to EPOCH conversion

import time
print(int(time.mktime(time.strptime('2017-08-12T17:07:46', '%Y-%m-%dT%H:%M:%S'))))
I get 1502582866 and I expect 1502557666? Any help is welcome

Yes, #Paul is correct: according to the documentation, mktime will take a struct_time in local time and convert it to seconds since the epoch, which has no timezones nor daylight savings.
So, your initial time of:
`2017-08-12T17:07:46`
becomes:
1502582866 ==> `GMT: Sunday, August 13, 2017 12:07:46 AM`
which is correct.
If you are wondering how to correctly convert back, you would use localtime:
import time
epoch_time = 1502582866
time_string = time.strftime('%Y-%m-%dT%H:%M:%S', time.localtime(epoch_time))
print time_string
>> 2017-08-12T17:07:46
Also see a sample in action here: https://eval.in/845075

Python Arrow Not Formatting Date Properly

I am using the Python Arrow package for formatting a date that I've got.
My input is:
datetime=2017-10-01T00:10:00Z and timezone=US/Pacific
My expected output is:
Sun 10/01 # 6:10pm
I've tried a host of different date time conversions but I always end up with Sun 10/01 # 12:10AM which is not time zone dependent.
If I try and say:
x = arrow.Arrow.strptime('2017-10-01T00:10:00Z',
'%Y-%m-%dT%H:%M:%SZ',
tzinfo=tz.gettz('US/Pacific'))
x is equal to:
<Arrow [2017-10-01T00:10:00-07:00]>
I then say:
x.strftime('%a %m/%d # %I:%M%p')
and it outputs
Sun 10/01 # 12:10AM
The Arrow object knows about the timezone as evidenced by the -7:00 but does not format the date accordingly.
Any ideas?

I think that there are a couple of misunderstandings in this question.
Convert to a timezone
I can see no way that the timestamp,
2017-10-01T00:10:00Z and timezone=US/Pacific
Can become,
Sun 10/01 # 6:10pm
There are several problems here.
The Z at the end of the timestamp is a timezone and means Zulu aka GMT, so the timestamp already has a timezone.
If we ignore problem #1, then for that timestamp 10 minutes after midnight (minus the Z) to become 6:10 pm the same day would require a timezone that was +18. This timezone does not exist.
US/Pacific is -7/-8 depending on the time of the year. If we accept the Z as the timezone and want to convert to US/Pacific, then the time should be 9/31 at 5:10pm
What does -7:00 mean?
So I am going to guess that what you intend is that the timestamp is in fact Zulu, and you want to display that timestamp as US/Pacific. If this is true than you need to do:
from dateutil import tz
import arrow
x = arrow.Arrow.strptime(
'2017-10-01T00:10:00Z',
'%Y-%m-%dT%H:%M:%SZ').to(tz.gettz('US/Pacific'))
print(x)
print(x.strftime('%a %m/%d # %I:%M%p'))
This results in:
2017-09-30T17:10:00-07:00
Sat 09/30 # 05:10PM
You will note, as you noted earlier, that these produce the same time. The difference is that the first also has a -7:00. This does not indicate, as you intimated in your question, that time needs to have 7 hours removed to be shown as in the timezone, it instead indicates that the timestamp has already had 7 hours removed from Zulu.

python converting string in localtime to UTC epoch timestamp

I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
I am trying to convert them into epoch timestamps for UTC time.
I wrote the following function:
def ymdhms_timezone_dst_to_epoch(input_str, tz="US/Eastern"):
print(input_str)
dt = datetime.datetime.fromtimestamp(time.mktime(time.strptime(input_str,'%Y-%m-%d %H:%M:%S')))
local_dt = pytz.timezone(tz).localize(dt)
print(local_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
utc_dt = local_dt.astimezone(pytz.utc)
print(utc_dt.strftime('%Y-%m-%d %H:%M:%S %Z%z'))
e = int(utc_dt.strftime("%s"))
print(e)
return e
Given string `2015-04-20 21:12:07` this prints:
2015-04-20 21:12:07
2015-04-20 21:12:07 EDT-0400 #<- so far so good?
2015-04-21 01:12:07 UTC+0000 #<- so far so good?
1429596727
which looks ok up to the epoch timestamp. But http://www.epochconverter.com/epoch/timezones.php?epoch=1429596727 says it should mao to
Greenwich Mean Time Apr 21 2015 06:12:07 UTC.
What is wrong?

I have strings in YMD hms format that had the timezone stripped. But I know they are in Eastern time with daylight savings time.
A portable way is to use pytz:
#!/usr/bin/env python
from datetime import datetime
import pytz # $ pip install pytz
naive_dt = datetime.strptime('2015-04-20 21:12:07', '%Y-%m-%d %H:%M:%S')
tz = pytz.timezone('US/Eastern')
eastern_dt = tz.normalize(tz.localize(naive_dt))
print(eastern_dt)
# -> 2015-04-20 21:12:07-04:00
I am trying to convert them into epoch timestamps for UTC time.
timestamp = (eastern_dt - datetime(1970, 1, 1, tzinfo=pytz.utc)).total_seconds()
# -> 1429578727.0
See Converting datetime.date to UTC timestamp in Python.
There are multiple issues in your code:
time.mktime() may return a wrong result for ambiguous input time (50% chance) e.g., during "fall back" DST transition in the Fall
time.mktime() and datetime.fromtimestamp() may fail for past/future dates if they have no access to a historical timezone database on a system (notably, Windows)
localize(dt) may return a wrong result for ambiguous or non-existent time i.e., during DST transitions. If you know that the time corresponds to the summer time then use is_dst=True. tz.normalize() is necessary here, to adjust possible non-existing times in the input
utc_dt.strftime("%s") is not portable and it does not respect tzinfo object. It interprets input as a local time i.e., it returns a wrong result unless your local timezone is UTC.
Can I just always set is_dst=True?
You can, if you don't mind getting imprecise results for ambiguous or non-existent times e.g., there is DST transition in the Fall in America/New_York time zone:
>>> from datetime import datetime
>>> import pytz # $ pip install pytz
>>> tz = pytz.timezone('America/New_York')
>>> ambiguous_time = datetime(2015, 11, 1, 1, 30)
>>> time_fmt = '%Y-%m-%d %H:%M:%S%z (%Z)'
>>> tz.localize(ambiguous_time).strftime(time_fmt)
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=False).strftime(time_fmt) # same
'2015-11-01 01:30:00-0500 (EST)'
>>> tz.localize(ambiguous_time, is_dst=True).strftime(time_fmt) # different
'2015-11-01 01:30:00-0400 (EDT)'
>>> tz.localize(ambiguous_time, is_dst=None).strftime(time_fmt)
Traceback (most recent call last):
...
pytz.exceptions.AmbiguousTimeError: 2015-11-01 01:30:00
The clocks are turned back at 2a.m. on the first Sunday in November:
is_dst disambiguation flag may have three values:
False -- default, assume the winter time
True -- assume the summer time
None -- raise an exception for ambiguous/non-existent times.
is_dst value is ignored for existing unique local times.
Here's a plot from PEP 0495 -- Local Time Disambiguation that illustrates the DST transition:
The local time repeats itself twice in the fold (summer time -- before the fold, winter time -- after).
To be able to disambiguate the local time automatically, you need some additional info e.g., if you read a series of local times then it may help if you know that they are sorted: Parsing of Ordered Timestamps in Local Time (to UTC) While Observing Daylight Saving Time.

First of all '%s' is not supported on all platforms , its actually working for you because your platform C library’s strftime() function (that is called by Python) supports it. This function is what is causing the issue most probably, I am guessing its not timezone aware , hence when taking difference from epoch time it is using your local timezone, which is most probably EST(?)
Instead of relying on '%s' , which only works in few platforms (linux, I believe) , you should manually subtract the datetime you got from epoch (1970/1/1 00:00:00) to get the actual seconds since epoch . Example -
e = (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
Demo -
>>> (utc_dt - datetime.datetime(1970,1,1,0,0,0,tzinfo=pytz.utc)).total_seconds()
1429578727.0
This correctly corresponds to the date-time you get.

I don't exactly know why but you have to remove the timezone info from your utc_dt before using %s to print it.
e = int(utc_dt.replace(tzinfo=None).strftime("%s"))
print(e)
return e

Convert a UTC time to epoch

I am looking to analyze traffic flow with relation to weather data. The traffic data has a UNIX timestamp (aka epoch), but I am running into trouble with converting the timestamp (in the weather data) to epoch. The problem is that I am in Norway and the UTC timestamp in the weather data isn't in the same timezone as me (GMT+1).
My initial approach
I first tried converting it into epoch and treating the data as if it was in the GMT+1 timezone. Then I compensated by subtracting the difference in number of seconds between UTC and GMT+1.
Problems with the approach
I realize first of all that this approach is very primitive and not very elegant (in fact probably it is at best an ugly hack). However, the biggest problem here is that the difference between UTC and GMT+1 is not constant (due to daylight savings).
Question
Is there any reliable way of turning UTC time to a UNIX time stamp in python (taking into account that my machine is in GMT+1)? The timestamp is in the following format:
Y-m-d HH:MM:SS
Edit:
Tried rmunns' solution:
def convert_UTC_to_epoch(timestamp):
tz_UTC = pytz.timezone('UTC')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime.strptime(timestamp, time_format)
aware_timestamp = tz_UTC.localize(naive_timestamp)
epoch = aware_timestamp.strftime("%s")
return (int) (epoch)
This does not work properly as evidenced below:
#Current time at time of the edit is 15:55:00 UTC on June 9th 2014.
>>> diff = time.time() - convert_UTC_to_epoch("2014-06-09 15:55:00")
>>> diff
3663.25887799263
>>> #This is about an hour off.

The solution was to use the calendar module (inspired from here)
>>>#Quick and dirty demo
>>>print calendar.timegm(datetime.datetime.utcnow().utctimetuple()) - time.time()
>>>-0.6182510852813721
And here is the conversion function:
import calendar, datetime, time
#Timestamp is a datetime object in UTC time
def UTC_time_to_epoch(timestamp):
epoch = calendar.timegm(timestamp.utctimetuple())
return epoch

An alternative, datetime has it's own .strptime() method.
http://en.wikipedia.org/wiki/Unix_time
The Unix epoch is the time 00:00:00 UTC on 1 January 1970 (or 1970-01-01T00:00:00Z ISO 8601).
import datetime
unix_epoch = datetime.datetime(1970, 1, 1)
log_dt = datetime.datetime.strptime("14-05-07 12:14:16", "%y-%m-%d %H:%M:%S")
seconds_from_epoch = (log_dt - unix_epoch).total_seconds()
>>> 1399490056.0

The pytz module will probably help you. It allows you to write code like:
import pytz
import datetime
tz_oslo = pytz.timezone('Europe/Oslo')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime(2014, 6, 4, 12, 34, 56)
# Or:
naive_timestamp = datetime.datetime.strptime("2014-06-04 12:34:56", time_format)
aware_timestamp = tz_oslo.localize(naive_timestamp)
print(aware_timestamp.strftime(time_format + " %Z%z"))
This should print "2014-06-04 14:34:56 CEST+0200".
Do note the following from the pytz manual:
The preferred way of dealing with times is to always work in UTC, converting to localtime only when generating output to be read by humans.
So keep that in mind as you write your code: do the conversion to local time once and once only, and you'll have a much easier time doing, say, comparisons between two timestamps correctly.
Update: Here are a couple of videos you may find useful:
What you need to know about datetimes, a PyCon 2012 presentation by Taavi Burns (30 minutes)
Drive-in Double Header: Datetimes and Log Analysis, a two-part presentation. (Caution: annoying buzz in the video, but I couldn't find a copy with better sound). The first part is the "What you need to know about datetimes" presentation I linked just above, and the second part has some practical tips for parsing log files and doing useful things with them. (50 minutes)
Update 2: The convert_UTC_to_epoch() function you mention in your updated question (which I've reproduced below) is returning local time, not UTC:
def convert_UTC_to_epoch(timestamp):
tz_UTC = pytz.timezone('UTC')
time_format = "%Y-%m-%d %H:%M:%S"
naive_timestamp = datetime.datetime.strptime(timestamp, time_format)
aware_timestamp = tz_UTC.localize(naive_timestamp)
epoch = aware_timestamp.strftime("%s")
return (int) (epoch)
The problem is that you're using strftime("%s"), which is undocumented and is returning the wrong result. Python doesn't support the %s parameter, but it appears to work because it gets passed to your system's strftime() function, which does support the %s parameter -- but it returns local time! You're taking a UTC timestamp and parsing it as local time, which is why it's an hour off. (The mystery is why it isn't two hours off -- isn't Norway in daylight savings time right now? Shouldn't you be at UTC+2?)
As you can see from the interactive Python session below, I'm in the UTC+7 timezone and your convert_UTC_to_epoch() function is seven hours off for me.
# Current time is 02:42 UTC on June 10th 2014, 09:42 local time
>>> time.timezone
-25200
>>> time.time() - convert_UTC_to_epoch("2014-06-10 02:42:00")
25204.16531395912
>>> time.time() + time.timezone - convert_UTC_to_epoch("2014-06-10 02:42:00")
6.813306093215942
The strftime("%s") call is interpreting 02:42 on June 10th as being in local time, which would be 19:42 UTC on June 9th. Subtracting 19:42 UTC on June 9th from 02:42 UTC June 10th (which is what time.time() returns) gives a difference of seven hours. See Convert python datetime to epoch with strftime for more details on why you should never use strftime("%s").
(By the way, if you saw what I had previously written under the heading "Update 2", where I claimed that time.time() was returning local time, ignore that -- I got it wrong. I was fooled at first by the strftime("%s") bug just like you were.)

You can use the time and datetime modules:
import time, datetime
date = "14-05-07 12:14:16" #Change to whatever date you want
date = time.strptime(date, "%y-%m-%d %H:%M:%S")
epoch = datetime.datetime.fromtimestamp(time.mktime(date)).strftime('%s')
This runs as:
>>> import time, datetime
>>> date = "14-05-07 12:14:16"
>>> date = time.strptime(date, "%y-%m-%d %H:%M:%S")
>>> epoch = datetime.datetime.fromtimestamp(time.mktime(date)).strftime('%s')
>>> epoch
'1399490056'
>>>

how to calculate timedelta python

What I am trying to do is to subtract 7 hours from a date. I searched stack overflow and found the answer on how to do it here. I then went to go read the documentation on timedelta because I was unable to understand what that line in the accepted answer does, rewritten here for ease:
from datetime import datetime
dt = datetime.strptime( date, '%Y-%m-%d %H:%M' )
dt_plus_25 = dt + datetime.timedelta( 0, 2*60*60 + 30*60 )
Unfortunately, even after reading the documentation I still do not understand how that line works.
What is the timedelta line doing? How does it work?
Additionally, before I found this stackoverflow post, I was working with time.struct_time tuples. I had a variable tm:
tm = time.strptime(...)
I was simply accessing the hour through tm.tm_hour and subtracting seven from it but this, for obvious reasons, does not work. This is why I am now trying to use datetime. tm now has the value
tm = datetime.strptime(...)
I'm assuming using datetime is the best way to subtract seven hours?
Note: subtracting seven hours because I want to go from UTC to US/Pacific timezone. Is there a built-in way to do this?

What is the timedelta line doing? How does it work?
It creates a timedelta object.
There are two meanings of "time".
"Point in Time" (i.e, date or datetime)
"Duration" or interval or "time delta"
A time delta is an interval, a duration, a span of time. You provided 3 values.
0 days.
2*60*60 + 30*60 seconds.

timedelta() generates an object representing an amount of time—the Greek letter delta is used in math to represent "difference". So to compute an addition or a subtraction of an amount of time, you take the starting time and add the change, or delta, that you want.
The specific call you've quoted is for generating the timedelta for 2.5 hours. The first parameter is days, and the second is seconds, so you have (0 days, 2.5 hours), and 2.5 hours in seconds is (2 hours * 60 minutes/hour * 60 seconds/minute) + (30 minutes * 60 seconds / minute).
For your case, you have a negative time delta of 0 days, 7 hours, so you'd write:
timedelta(0, -7 * 60 * 60)
... or timedelta(0, -7 * 3600) or whatever makes it clear to you what you're doing.

Note: subtracting seven hours because I want to go from UTC to US/Pacific timezone. Is there a built-in way to do this?
Yes there is: datetime has built-in timezone conversion capabilities. If you get your datetime object using something like this:
tm = datetime.strptime(date_string, '%Y-%m-%d %H:%M')
it will not have any particular timezone "attached" to it at first, but you can give it a timezone using
tm_utc = tm.replace(tzinfo=pytz.UTC)
Then you can convert it to US/Pacific with
tm_pacific = tm_utc.astimezone(pytz.all_timezones('US/Pacific'))
I'd suggest doing this instead of subtracting seven hours manually because it makes it clear that you're keeping the actual time the same, just converting it to a different timezone, whereas if you manually subtracted seven hours, it looks more like you're actually trying to get a time seven hours in the past. Besides, the timezone conversion properly handles oddities like daylight savings time.
To do this you will need to install the pytz package, which is not included in the Python standard library.