We Keep Coding

Pandas to_datetime error 'unconverted data remains'

I'm trying to convert date column in my Pandas DataFrame to datetime format. If I don't specify date format, it works fine, but then further along in the code I get issues because of different time formats.
The original dates looks like this 10/10/2019 6:00 in european date format.
I tried specifying format like so:
df['PeriodStartDate'] = pd.to_datetime(df['PeriodStartDate'],
format="%d/%m/%Y")
which results in an error: unconverted data remains 6:00
I then tried to update format directive to format="%d/%m/%Y %-I/%H" which comes up with another error: '-' is a bad directive in format '%d/%m/%Y %-I/%H' even though I thought that to_datetime uses the same directives and strftime and in the latter %-I is allowed.
In frustration I then decided to chop off the end of the string since I don't really need hours and minutes:
df['PeriodStartDate'] = df['PeriodStartDate'].str[:10]
df['PeriodStartDate'] = pd.to_datetime(df['PeriodStartDate'],
format="%d/%m/%Y")
But this once again results in an error: ValueError: unconverted data remains: which of course comes from the fact that some dates have 9 digits like 3/10/2019 6:00
Not quite sure where to go from here.

format %H:%M would work(don't forget the : in between)
pd.to_datetime('10/10/2019 6:00', format="%m/%d/%Y %H:%M")
Out[1049]: Timestamp('2019-10-10 06:00:00')
pd.to_datetime('3/10/2019 18:00', format="%d/%m/%Y %H:%M")
Out[1064]: Timestamp('2019-10-03 18:00:00')

Oh, I feel so dumb. I figured out what the issue was. For some reason I thought that hours were in a 12-hour format, but they were in fact in a 24-hour format, so changing directive to "%d/%m/%Y %H:%M" solved it.

Python convert string to datetime in variable formats

I am generating date and time information which is string from an API. The generated string is in your system's date/time format by default (Win 10 in my case). For example if you are using MM/DD/YYYY HH:MM:SS tt in your computer, the generated string would be something like "05/07/2019 06:00:00 AM".
For comparison purpose, I would then convert the string to datetime format by using datetime.datetime.strptime(i,"%m/%d/%Y %I:%M:%S %p"). This works prefectly fine, however if someone else whose system date/time format is different from' MM/DD/YYYY HH:MM:SS tt' runs my script, he would get a mismatch error as the string can no longer be converted by %m/%d/%Y %I:%M:%S %p.
So would it be possible to make the desired datetime format become a variable argument in the strptime function? Or even simpler, just make the format to be the same as the system's date/time format.

You can use try and except to fix this (although there might be another way)
try:
datetime.datetime.strptime(i,"%m/%d/%Y %I:%M:%S %p")
except TypeError:
try:
datetime.datetime.strptime(i,"%d/%m/%Y %I:%M:%S %p")
except:
try:
datetime.datetime.strptime(i,"%d/%m/%y %I:%M:%S %p")
# and so on....

Serializing and deserializing datetimefield type object (Django app)

In a Django app of mine, I have a datetime object that I need to serialize and then deserialize. When I try it, I get the error:
ValueError: time data '2016-05-31T18:57:17.280939+00:00' does not
match format '%Y-%m-%d %H:%M:%S.%f'
My code to serialize and deserialize is:
timestring = time.isoformat() #where timestring is DateTimeField type object, instantiated in Django
timeobj = datetime.strptime(timestring, "%Y-%m-%d %H:%M:%S.%f")
What am I doing wrong and how do I get over the hump? Your guidance is greatly appreciated.

timeobj = datetime.strptime(timestring, "%Y-%m-%dT%H:%M:%S.%f+00:00")
(adds a T as the date/time separator, and hard codes the utc offset string part)
will resolve your problem ... and I guess its reasonably safe ... personally I always go with
from dateutil.parser import parse as date_parse
dt_obj = date_parse(timestring)
that pretty much always works and does not require me to hardcode the datestring you may need pip install python-dateutil

So there are two things going on here:
Your format string has a space between %d and %H but the test string has a T.
Python's datetime.datetime.strptime does not work with timezone names/offsets. From the relevant docs:
classmethod datetime.strptime(date_string, format)
Return a datetime corresponding to date_string, parsed according to
format. This is equivalent to datetime(*(time.strptime(date_string,
format)[0:6])).
So you can extract year, month, day, hour, minute, second and microsecond but the %z directive is for strftime only, not strptime.
So, in summary:
In [18]: datetime.strptime(datetime.today().isoformat(), '%Y-%m-%dT%H:%M:%S.%f')
Out[18]: datetime.datetime(2016, 5, 31, 15, 20, 20, 581261)
but
In [22]: datetime.strptime(datetime.today().isoformat()+'+00:00', '%Y-%m-%dT%H:%M:%S.%f%z')
---------------------------------------------------------------------------
ValueError: 'z' is a bad directive in format '%Y-%m-%dT%H:%M:%S.%f%z'

Converting timestamp to unix date python

Here is how the timestamp looks -
2015-07-17 06:01:51.066141+00:00
I'm looking around to convert this to unix date time.
datetime.strptime("2015-07-17 06:01:51.066141+00:00", "%Y-%m-%d %H:%M:%S.%f%z").strftime("%s")
ValueError: 'z' is a bad directive in format '%Y-%m-%d %H:%M:%S.%f%z'
throws error for me, probably because of wrong format being used.
PS: my virtualenv is on python 2.7
ideas please ?

python 2.7 strptime() does not support z directive, either you can use python 3.2+ or some other 3rd party library like dateutil.

For Python 2.7 use arrow:
import arrow
date_str = "2015-07-17 06:01:51.066141+00:00"
unix_time = arrow.get(date_str).timestamp
On PY3 (verified on 3.4), using only standard libs
The date string you show will not be parsed by the standard python datetime library since it has a colon in the timezone (see here). The colon can be easily removed since it's always in the same position (or use rfind to find its index starting from the right). Your simplest solution is:
import datetime
date_str = "2015-07-17 06:01:51.066141+00:00"
date_str_no_colon = date_str[:-3]+date_str[-2:] # remove last colon
dt_obj = datetime.datetime.strptime(date_str_no_colon, "%Y-%m-%d %H:%M:%S.%f%z")
unix_time = dt_obj.timestamp()
Note that arrow should still work with PY3, and is a better solution in general- you don't want to get into datetime parsing wars with python. It will win.

The way to parse the date is not right. You'll either need to parse it by hand, find another library (for example the dateutil.parser.parse method that will parse your string directly without supplying format string) that supports that format or make the timestamp in another format. Even with newer versions of python the %z directive does not accept TZ offsets in the +/-HH:MM format (with colon).
As the source of the timestamp is django.DateTimeField maybe this question can help?
For converting to unix timestamp you seem to have to do some work since there does not seem to be a direct method for that:
(t - datetime.utcfromtimestamp(0)).total_seconds()
where t is the datetime (assuming it's in UTC and there is no tzinfo) you want to convert to POSIX timestamp. If the assumption is not correct you need to put tzinfo in the zero timestamp you subtract as shown below where the assumption does not hold.
If you want to use dateutil.parser the complete solution would be:
(dateutil.parser.parse(timestamp) - datetime.utcfromtimestamp(0).replace(tzinfo=utc()).total_seconds()

strptime() has no support for timezones.
So, you can make the conversion ignoring the timezone in the following way:
datetime.strptime("2015-07-17 06:01:51.066141", "%Y-%m-%d %I:%M:%S.%f").strftime("%s")
'1437102111'
Or in order to avoid using %s, as suggested below in the commments :
from datetime import datetime
(datetime.strptime("2015-07-17 06:01:51.066141", "%Y-%m-%d %I:%M:%S.%f") - datetime(1970, 1, 1)).total_seconds()
1437112911.066141
Notice this is a working version for Python 2, you can also check solutions for other versions here
Otherwise, you will have to use other libraries (django.utils or email.utils) that support timezones, or implement the timezone parsing on your own.
P.S. :
strptime docs appear to have support for timezone, but in fact it has not been implemented. Try :
datetime.strptime("2015-07-17 06:01:51.066141+00:00", "%Y-%m-%d %I:%M:%S.%f%z").strftime("%s")
and you will see that it is not supported. You can also verify it by searching more about strptime()

There are two parts:
to convert "2015-07-17 06:01:51.066141+00:00" into a datetime object that represents UTC time, see Convert timestamps with offset to datetime obj using strptime. Or If you know that the utc offset is always +0000:
from datetime import datetime
utc_time = datetime.strptime(time_string, "%Y-%m-%d %H:%M:%S.%f+00:00")
to convert the UTC time to POSIX timestamp (unix time), see Converting datetime.date to UTC timestamp in Python:
from datetime import datetime
timestamp = (utc_time - datetime(1970, 1, 1)).total_seconds()

Converting ISO 8601 date format with timezone in python

I am attempting to convert the following date (2012-12-25T08:52:00-05:00) to a datetime object in python. However, I cannot figure out what the -05:00 part of the date is referencing. I am simply trying to perform the following:
datetime.datetime.strptime('2012-12-25T08:52:00-05:00','%Y-%m-%dT%H:%M:%S')
But this comes up with an expected 'ValueError: unconverted data remains'. I'm just trying to figure out what the last part of the date is used for so that I can convert that string to a proper datetime object in python.
Happy Holidays!

Your date seems to be in the ISO 8601 format, I don't think datetime handles the timezone information at the end of the string format.
You can use pip install python-dateutil, its parser can return a datetime object :
import dateutil.parser
datestr = '2012-12-25T08:52:00-05:00'
dateutil.parser.parse(datestr)
>>> datetime.datetime(2012, 12, 25, 8, 52, tzinfo=tzoffset(None, -18000))

The -05:00 indicates the timezone offset from UTC, i.e. %z would be the correct strptime argument to parse it.
If the time is UTC the offset might be indicated using Z, e.g. 2012-12-25T08:52:00Z. Not sure if %z would actually accept this...