We Keep Coding

datetime.timestamp() loses an time (an hour)

I encountered this issue but found the solution after a bit of research. I have posted my answer below to show my findings. If anyone has alternative suggestions please post them.
I needed to convert a datetime.datime object to a Unix timestamp. I tried using the datetime.timestamp. I found the result was 1 hour behind what I expected. I was able to replicate this issue with the following.
from datetime import datetime, timestamp
dt = datetime.utcfromtimestamp(1438956602.0)
dt now equals datetime.datetime(2015, 8, 7, 14, 10, 2)
Then:
dt_stamp = datetime.timestamp(dt)
Which gives dt_stamp = 1438953002.0 (which is different from our original timestamp). If we convert it back to datetime
datetime.utcfromtimestamp(dt_stamp)
We get:
datetime.datetime(2015, 8, 7, 13, 10, 2)
Which is an hour earlier than our original time.
For context I am using Python 3 and based in Britain where we're currently using British summer time (1 hour ahead of UTC).
My solution can be found below. If you think I have missed anything from my explanation or there's a better solution, please post your own answer.

I met the same problem recently, my case is that part EDF recording from one hostipal in UK have one hour bias, which is considered due to British summer time.
Following is the solution to my case.
from datetime import datetime as dt
Please use
dt = dt.utcfromtimestamp(#YOUR_TIME_STAMP)
INSTEAD of
dt = dt.fromtimestamp(#YOUR_TIME_STAMP)

The cause for this difference is actually shown in the
datetime.timestamp documentation.
Naive datetime instances are assumed to represent local time and this method relies on the platform C mktime() function to perform the conversion. Since datetime supports wider range of values than mktime() on many platforms, this method may raise OverflowError for times far in the past or far in the future.
Because I am in UTC+1 (during British summer time) this is the timezone datetime.timestamp uses to calculate the timestamp. This is where the mistake comes in. The documentation recommends a few ways to deal with this. I went with the following.
from datetime import datetime, timestamp
dt = datetime.utcfromtimestamp(1438956602.0)
dt_stamp = datetime.timestamp(dt.replace(tzinfo=timezone.utc))
By adding .replace(tzinfo=timezone.utc) to the end of dt it specifies that this is done in the UTC timezone. datetime.timestamp then knows to use the UTC time rather than whatever timezone my machine is running.
People in America or other parts of the world will encounter this issue if not using the UTC timezone. If this is the case you can set tzinfo to whatever your timezone is. Also note that datetime.utcfromtimestamp is also clearly designed for people using the UTC timezone.

I think you need a so called aware datetime object. Aware means it nows the time difference you have:
datetime.fromtimestamp(timestamp, timezone(timedelta(hours=1)))
Try it with that line of code, where timestamp is your Unix timestamp.

Use current time of the day as an integer in python

I am looking to create an if then statement that involves the current time of the day. For example I want something like if it is past 2pm then do this function.
I have tried using the time module but I can't seem to find a way to get just the time of day without the extra stuff like the date. Any help?

Here is a start, and I think it'll be enough for you to get to the answer and use it how you need.
import time
print(time.strftime("%Y-%m-%d %H:%M"))
print(time.strftime("I can format the date and time in many ways. Time is: %H:%M"))
Output (when I ran it):
2017-06-21 10:40
I can format the date and time in many ways. Time is: 10:40

Converting human readable date and time into Unix time float

I have to convert date and time returned by ls of rsync into Unix epoch time float as returned by time.time().
For me here at this moment it looks like:
2017/05/24 hh:mm:ss.
But as far as I know it can vary from machine to machine as rsync uses ssh and native ls, I expect through it.
Is there any easy way to universally convert most common human readable date and time back to the Unix time float?
To be clear. I want to be able to convert any textual representation of D and T into the float.
If datetime can do this I cannot find how at the moment.

You need to use time.strptime first and then calendar.timegm
there are different options depending if you want to convert to local time or UTC time. Have a look to the documentation for that.
To get the float part, you need to input hours, minutes, seconds and milliseconds. In your example, you give only the year, month and day, so the rest is supposed to be zero i.e. no milliseconds thus no float part.
Here a minimal example:
import calendar, time
t = time.strptime('2017/05/24', '%Y/%m/%d')
epoch = calendar.timegm(time.struct_time(t))
print(epoch)
1495584000

Why do I get NonExistentTimeError in python for time stamps between 12 to 1am on 2017-03-12

I get this error when trying to append two Pandas DFs together in a for loop:
Aggdata=Aggdata.append(Newdata)
This is the full error:
File "pandas\tslib.pyx", line 4096, in pandas.tslib.tz_localize_to_utc (pandas
\tslib.c:69713)
pytz.exceptions.NonExistentTimeError: 2017-03-12 02:01:24
However, in my files, I do not have such a time stamp, but I do have ones like 03/12/17 00:45:26 or 03/12/17 00:01:24. Where it is 2 hours before daylight savings. And if I manually delete the offending row, I get that same error for the next row with times between 12 and 1am on the 12th of March.
My original date/time column has no TZ info, but I calculate another column in EST, before the concatenation and localize it to EST, with time with TZ information:
`data['EST_DateTimeStamp']=pd.DatetimeIndex(pd.to_datetime(da‌​ta['myDate'])).tz_lo‌​calize('US/Eastern')‌​.tz_convert('US/East‌​ern')`
Doing some research here, I understand that 2 to 3am on the 12th should be having such error, but why midnight to 1am. So am I localizing it incorrectly? and then why is the error on the append line, and not the localization line?
I was able to reproduce this behavior in a very simple MCVE, saved here:
https://codeshare.io/GLjrLe
It absolutely boggles my mind that the error is raised on the third append, and only if the next 3 appends follow. In others words, if I comment out the last 3 copies of appends, it works fine.. can't imagine what is happening.
Thank you for reading.

In case someone else may still find this helpful:
Talking about it with #hashcode55, the solution was to upgrade Pandas on my server, as this was likely a bug in my previous version of that module.

The problem seems to occur at daylight savings switch - there are local times that do not exist, once per year. In the opposite direction there will be duplicate times.
This could be from say your input dates being converted from UTC to "local time" by adding a fixed offset. When you try to localize these you will hit a non existent times over that hour (or 30 minutes if you are in Adelaide).

Is Pandas.tslib.normalize_date() timezone aware?

I found the following behaviour at normalizing Timestamps at daylight saving time change boundaries at pandas 0.16.2:
import pandas as pd
original_midnight = pd.Timestamp('20121104', tz='US/Eastern')
original_midday = pd.Timestamp('20121104T120000', tz='US/Eastern')
str(pd.tslib.normalize_date(original_midday))
`Out[10]:'2012-11-04 00:00:00-05:00'`
str(original_midnight)
`Out[12]:'2012-11-04 00:00:00-04:00'`
I believe the normalized Timestamp should have the same timezone than the original_midnight.
Is it a bug, or do I miss something?

The implementation simply truncates the time. It does not appear to manipulate the offset at all, so I will say no, it is not timezone aware.
Also, consider that this type of operation (in many languages) tends to gloss over the fact that not every local day has a midnight. For example, if the time zone is 'America/Sao_Paulo' (Brazil), and the date is on the spring-forward transition (such as 2015-10-18), the hour from 00:00 to 00:59 is skipped, meaning the start of the day is actually 01:00. If the function were to be updated to be timezone aware, it would have to adjust the time as well as the offset.