I have a huge logging file with time stamp in the format like below:
08/07/2013 11:40:08 PM INFO
I want to convert that to mysql timestamp using python, like:
2013-04-11 13:18:02
I have written a python script to do that but I am wondering is there some build-in python package/function written already to do the timestamp routine work easily and more efficiently.
Since data 'massaging' is part of my daily work so any suggestion to the efficiency of my code or usage of new function or even new tools would be gratefully appreciate.
(Note: input file is delimited by ^A and I am also converting that to csv at the same time)
($ cat output.file | python csv.py > output.csv)
import sys
def main():
for line in sys.stdin:
line = line[:-1]
cols = line.split(chr(1))
cols[0] = convertTime(cols[0])
cols = [ '"' + col + '"' for col in cols ]
print ",".join(cols)
def convertTime(loggingTime):
#mysqlTime example: 2013-04-11 13:18:02
#loggingTime example: 08/07/2013 11:40:08 PM INFO
#DATE
month, day, year = loggingTime[0:10].split('/')
date = '/'.join([year,month,day])
#TIME
hour, minute, second = loggingTime[11:19].split(':')
flag = loggingTime[20:22]
if flag == 'PM':
hour = str(int(hour) + 12)
time = ":".join([hour, minute, second])
mysqlTime = date + " " + time
return mysqlTime
if __name__ == '__main__':
main()
Use time.strptime to parse the time, then time.strftime to reformat to new format?
import time
input_format = "%m/%d/%Y %I:%M:%S %p INFO" # or %d/%m...
output_format = "%Y-%m-%d %H:%M:%S"
def convert_time(logging_time):
return time.strftime(output_format, time.strptime(logging_time, input_format))
print convert_time("08/07/2013 11:40:08 PM INFO")
# prints 2013-08-07 23:40:08
Notice however that strptime and strftime can be affected by the current locale, you might want to set the locale to C (it is internally used by the datetime module too), as the %p can give different formatting for AM/PM for different locales; thus to be safe you might need to run the following code in the beginning:
import locale
locale.setlocale(locale.LC_TIME, "C")
I would recommend using the datetime module. You can convert your date string into a python datetime object, which you can then use to output a reformatted version.
from datetime import datetime
mysqltime = "2013-04-11 13:18:02"
timeobj = datetime.strptime(mysqltime, "%Y-%m-%d %H:%M:%S")
loggingtime = timeobj.strftime("%m/%d/%Y %H:%M:%S %p")
Convert it, as suggested, with strptime like this:
converter="%d/%m/%Y %H:%M:%S %p INFO"
result = dt.datetime.strptime("08/07/2013 11:40:08 PM INFO",converter)
Split is needed due to the "INFO"-String (edit: not needed). Then parse with strftime:
result.strftime("%Y-%m-%d %H:%M:%S")
Related
I'm trying to format a date to a customized one. When I use datetime.datetime.now(), I get the right format of date I'm after. However, my intention is to get the same format when I use 1980-01-22 instead of now.
import datetime
date_string = "1980-01-22"
item = datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")
print(item)
Output I get:
2021-05-04T09:52:04.010Z
How can I get the same format of date when I use a customized date, as in 1980-01-22 instead of now?
MrFuppes suggestion in the comments is the shortest way to accomplish your date conversion and formatting use case.
Another way is to use the Python module dateutil. This module has a lot of flexibility and I use it all the time.
Using dateutil.parser.parse:
from dateutil.parser import parse
# ISO FORMAT
ISO_FORMAT_MICROS = "%Y-%m-%dT%H:%M:%S.%f%z"
# note the format of these strings
date_strings = ["1980-01-22",
"01-22-1980",
"January 22, 1980",
"1980 January 22"]
for date_string in date_strings:
dt = parse(date_string).strftime(ISO_FORMAT_MICROS)
# strip 3 milliseconds for the output and add the ZULU time zone designator
iso_formatted_date = f'{dt[:-3]}Z'
print(iso_formatted_date)
# output
1980-01-22T00:00:00.000Z
1980-01-22T00:00:00.000Z
1980-01-22T00:00:00.000Z
1980-01-22T00:00:00.000Z
Using dateutil.parser.isoparse:
from dateutil.parser import isoparse
from dateutil.tz import *
dt = isoparse("1980-01-22").isoformat(timespec="milliseconds")
iso_formatted_date = f'{dt}Z'
print(iso_formatted_date)
# output
1980-01-22T00:00:00.000Z
Is this what your trying to achieve?
date_string = "1980-01-22"
datetime.datetime.strptime(date_string, "%Y-%m-%d").isoformat(timespec="milliseconds")
Output
'1980-01-22T00:00:00.000'
I have the following string format (Python 3.6):
'2018-11-19T10:04:57.426872'
I get it as a parameter to my script.
I want to get the date as 'YYYY-MM-DD' and time as 'HH:MM'
I tried to convert it with:
from datetime import datetime
if __name__ == '__main__':
start_timestamp = sys.argv[1]
start_date = datetime.strptime(sys.argv[1], '%Y-%m-%d')
start_time = datetime.strptime(sys.argv[1], '%H:%M')
But this gives:
ValueError: unconverted data remains: T10:04:57.426872
In the above example I want to see:
start_date = '2018-11-19'
start_time = '10:04'
Since the date seems to be in ISO-Format, a simple
start = datetime.datetime.fromisoformat(text)
will parse it correctly. From there you can get your date and time with
start_date = start.strftime("%Y-%m-%d")
start_time = start.strftime("%H:%M")
Edit:
For Python < 3.7, you can use this format:
start = datetime.datetime.strptime(text, "%Y-%m-%dT%H:%M:%S.%f")
For the "duplicate" datetime confusion: I used import datetime. If you use from datetime import datetime, you can get rid of the additional datetime.
Try this:We have one of the best package for parsing dates called dateutil.
from dateutil import parser
date1='2018-11-19T10:04:57.426872'
print 'Start_date:',parser.parse(date1).strftime("%Y-%m-%d")
print 'Start_time:',parser.parse(date1).strftime("%H:%M")
Result:Start_date:2018-11-19
Start_time:10:04
You need to parse the entire string into one datetime object and then extract your required values from that.
dt = datetime.datetime.strptime('2018-11-19T10:04:57.426872', '%Y-%m-%dT%H:%M:%S.%f')
d = dt.date()
t = dt.time()
print(d.strftime('%Y-%m-%d'))
print(t.strftime('%H:%M'))
Which outputs:
2018-11-19
10:04
This question already has answers here:
How to print a date in a regular format?
(25 answers)
Closed 6 years ago.
having trouble converting this string into a datetime this is what I tried so far in code:
import datetime
mystring = '2016/5/7/ 4:25:00 PM'
dateobj = datetime.datetime.strptime(mystring, "%Y-%B-%dT%H:%M:%S-%H:%M")
print (dateobj)
it throws me errors and I search in the library and still can't figure it out what I have wrong in my format.
Please any help I'll gratly apreciate it
You can do it as:
import datetime
mystring = '2016/5/7 4:25:00 PM'
dateobj = datetime.datetime.strptime(mystring, "%Y/%m/%d %I:%M:%S %p")
dateobj
Out[1]: datetime.datetime(2016, 5, 7, 16, 25)
dateobj1 = datetime.datetime.strptime(mystring, "%Y/%m/%d %I:%M:%S %p").strftime("%Y-%m-%d %I:%M:%S")
dateobj1
Out[2]: '2016-05-07 04:25:00'
The issue you are having is that in order to convert a date object you first have to create a date object. This means that your formatting will have to match the current string format '2016/5/7 4:25:00 PM' = "%Y/%m/%d %H:%M:%S %p". So we can now create a date obj and then format it using strftime with your new format "%Y-%B-%dT%H:%M:%S-%H:%M".
import datetime
mystring = '2016/5/7 4:25:00 PM'
dateobj = datetime.datetime.strptime(mystring, "%Y/%m/%d %H:%M:%S %p")
dateobj = datetime.datetime.strftime(dateobj, "%Y-%B-%dT%H:%M:%S-%H:%M")
print (dateobj)
the output recieved is a little funky, but it matches your format perfectly. Visit the datetime library to view the format codes and read up on strptime vs strftime. Good luck.
Output: 2016-May-07T04:25:00-04:25
I'm not sure where your format string came from, but it's wrong. It doesn't match your string format at all.
Take a look at the available options in the documentation.
You want this:
dateobj = datetime.datetime.strptime(mystring, "%Y/%m/%d/ %I:%M:%S %p")
This will get you a date object. You can then do this to reformat it how you want:
dateobj.strftime("%Y-%m-%d %I:%M:%S")
A couple things that were wrong with yours:
You used %H twice. This resulted in an error saying it'd be redefined.
%B is full month name (ie. January). You have a numeral.
The -s are incorrect since you have /s.
The T is not in your string at all.
Easily with dateutil.parser.parse
>>> from dateutil.parser import parse
>>> d = parse('2016/5/7/ 4:25:00 PM')
>>> d.strftime("%Y-%B-%dT%H:%M:%S-%H:%M")
'2016-May-07T16:25:00-16:25'
>>> d.strftime("%Y-%m-%d %I:%M:%S")
'2016-05-07 04:25:00'
I am currently trying to convert a date in the following format YYYYmmddHHMMSS to a unix timestamp but I get an error (ValueError: year is out of range).
import datetime
def ts(date):
return datetime.datetime.fromtimestamp(date).strftime('%Y%m%d%H%M%S')
if __name__ == "__main__":
date = 20130814100000
print ts(date)
Your date should be a string. Here is how you do it. (If your date is an integer then just do date = str(date).
>>> import time
>>> from datetime import datetime
>>> date = '20130814100000'
>>> dt = datetime.strptime(date, '%Y%m%d%H%M%S')
>>> print dt
2013-08-14 10:00:00
>>> print time.mktime(dt.timetuple())
1376467200.0
time also has a strptime function but it returns a not so useful struct_time object. But if you only need a unix time, then you can use it too:
>>> time.mktime(time.strptime(date, '%Y%m%d%H%M%S'))
1376467200.0
I think the problem here is that .fromtimestamp() is expecting a Unix timestamp, not a date formatted as YYYYmmdd...
To parse the date information that you do have there, I'd recommend using .strptime() or the excellent python-dateutil package.
import datetime
def ts(date):
stamp = datetime.datetime.strptime(date, '%Y%m%d%H%M%S')
return stamp.strftime('%Y%m%d%H%M%S')
or
from dateutil.parser import parse
def ts(date):
stamp = parse(date)
return stamp.strftime('%Y%m%d%H%M%S')
http://labix.org/python-dateutil
The function that parses datetime is called strptime, not strftime (which formats time).
20130814100000 is not an UNIX timestamp
strptime takes string as argument
Overall, your code should look like:
import datetime
def ts(date):
return datetime.datetime.strptime(date, '%Y%m%d%H%M%S')
if __name__ == "__main__":
date = "20130814100000"
print ts(date)
You seem to have some confusion over the different ways that times are represented. The value you have assigned to date appears to already be a formatted timestring of "2013-08-14 10:00:00", but you're passing it into fromtimestamp. This function expects a Unix timestamp, which is simply the number of seconds that have elapsed since Midnight on Jan 1st 1970.
I believe something like this is what you're looking for:
import datetime
def ts(datestr):
return datetime.datetime.strptime(datestr, "%Y%m%d%H%M%S")
if __name__ == "__main__":
date = 20130814100000
print ts(date)
strftime like you had is for formatting times into strings. strptime is for parsing strings into times.
In python I import a csv file with one datetime value at each row (2013-03-14 07:37:33)
and I want to compare it with the datetime values I obtain with timestamp.
I assume that when I read the csv the result is strings, but when I try to compare them in a loop with the strings from timestamp does not compare them at all without giving me an error at the same time.
Any suggestions?
csv_in = open('FakeOBData.csv', 'rb')
reader = csv.reader(csv_in)
for row in reader:
date = row
OBD.append(date)
.
.
.
for x in OBD:
print x
sightings = db.edge.find ( { "tag" : int(participant_tag)},{"_id":0}).sort("time")
for sighting in sightings:
time2 = datetime.datetime.fromtimestamp(time)
if x == time2:
Use datetime.datetime.strptime to parse the strings into datetime objects. You may also have to work out what time zone the date strings in your CSV are from and adjust for that.
%Y-%m-%d %H:%M:%S should work as your format string:
x_datetime = datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S')
if x_datetime == time2:
Or parse it when reading:
for row in reader:
date = datetime.datetime.strptime(row[0], '%Y-%m-%d %H:%M:%S')
You could parse it yourself with datetime.datetime.strptime which should be fine if you know the format the date is in. If you do not know the format or want to be more robust I would advise you to use the parser from python-dateutil library, it has an awesome parser that is very robust.
pip install python-dateutil
Then
import dateutil.parser
d = dateutil.parser.parse('1 Jan 2012 12pm UTC') # its that robust!