We Keep Coding

Time and date strings to DateTime Objects from csv file in python

So I'm working on a function which checks the dates in each row of a csv file against a standard date made up of two cells in the header row. What I need to do is take the date from A2 and the time from A3 and concatenate them into one object which can be compared against the rest of the rows of the file and then from there expel the rows which fail the test.
The only problem I'm having is in running the comparison with the time objects and getting the strings out of the csv. My current code gives me a ValueError because the format of value of date_time does not match the format %m/%d/%Y %H:%M:%S. Which is correct, because the value of date_time is the whole entire line.
Right now I'm simply trying to get the comparison to run on an arbitrary static time.
But if I want to take the date from cell A2 and concatenate it with the time in cell A3, then compare that new object with the rest of the rows in the file whose time and date do not need concatenation, what is the best way to go about running this comparison when you don't know what the dates are going to be?
def CheckDates(f):
with open(f, newline='', encoding='utf-8') as g:
r = csv.reader(g)
date_time = str(next(r))
for line in r:
if datetime.strptime(date_time, '%m/%d/%Y %H:%M:%S') >= datetime.strptime('01/11/2022 13:19:00', '%m/%d/%Y %H:%M:%S'):
# Dates pass
pass
else:
# Dates fail
pass
edited typos and added an example csv
TD,08/24/2021,14:14:08,21012,223,0,1098,0,031,810,12,01,092,048,0008,02
Date/Time,G120010,M129000,G110100,M119030,G112070,G112080,G111030,G127020,G127030,G120020,G120030,G121020,G111040,G112010,P102000,G112020,G112040,G112090,G110050,G110060,G110070,T111100
06/27/2022 00:00:01,40,133.2,0,0,7.284853,0,0.6030464,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5
06/27/2022 00:00:03,40,133.2,0,0,7.284853,0,0.5898247,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5
06/27/2022 00:00:05,40,133.2,0,0,7.284853,0,0.6135368,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5
06/27/2022 00:00:07,40,133.2,0,0,7.284853,0,0.6087456,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5
06/27/2022 00:00:09,40,133.2,0,0,7.284853,0,0.5903625,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5
06/27/2022 00:00:11,40,133.2,0,0,7.284853,0,0.5799789,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5
06/27/2022 00:00:13,40,133.2,0,0,7.284853,0,0.5821953,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5
06/27/2022 00:00:15,40,133.2,0,0,7.284853,0,0.6024017,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5
06/27/2022 00:00:17,40,133.2,0,0,7.284853,0,0.5984001,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5

This should do the trick. I modified a couple rows of your data file for "dramatic effect"...
# time compare
from datetime import datetime, timedelta
f = 'data.csv'
with open(f, 'r') as src:
row_0 = src.readline()
tokens = row_0.strip().split(',') # split (tokenize) the line
orig_time = tokens[1] + ' ' + tokens[2] # concatenate the strings
base_time = datetime.strptime(orig_time, '%m/%d/%Y %H:%M:%S')
print(f'recovered this base time: {base_time}')
src.readline() # burn row 2
# process the remainder
for line in src:
tokens = line.strip().split(',')
row_time = datetime.strptime(tokens[0], '%m/%d/%Y %H:%M:%S')
# calculate the difference. The result of comparing datetimes
# is a "timedelta" object that can be queried.
td = row_time - base_time
# make a comparision to see if it is pos/neg
if td < timedelta(0):
print('this line is before the base time:')
print(f' {line}')
Output:
recovered this base time: 2021-08-24 14:14:08
this line is before the base time:
06/27/2019 00:55:05,40,133.2,0,0,7.284853,0,0.6135368,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5
this line is before the base time:
06/27/2021 10:00:11,40,133.2,0,0,7.284853,0,0.5799789,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5
this line is before the base time:
06/27/2020 00:00:17,40,133.2,0,0,7.284853,0,0.5984001,0,0,1,0,5,11,5,0,0,414,344,0,154,0,5

Splitting a CSV column into two

Column 2 within my csv file looks like the following:
20150926T104044Z
20150926T104131Z
and so on.
I have a definition created that will change the listed date into a julian date, but was wondering how I can go about altering this specific column of data?
Is there a way I can make python change the dates within the csv to a their julian date equivalent? Can I split the column into two csv's and translate the julian date from there?

You might be overthinking it. Try this.
from dateutil.parser import parse
import csv
def get_julian(_date):
# _date is holding 20150926T104044Z
the_date = parse(_date)
julian_start = parse('19000101T000000Z')
julian_days = (the_date - julian_start).days
return julian_days
with open('filename.csv') as f:
csv_reader = csv.reader(f)
for row in csv_reader:
# Column 2, right?
row[1] = get_julian(row[1])
# Do things and stuff with your corrected data.

I observed that there are many interpretations to Julian Day, One is Oridinal date(day of the year) and another one day from Monday, January 1, 4713 BC.
import pandas as pd
import datetime
import jdcal
df = pd.read_csv("path/to/your/csv")
def tojulianDate(date):
return datetime.datetime.strptime(date, '%Y%m%dT%H%M%SZ').strftime('%y%j')
def tojulianDate2(date):
curr_date = datetime.datetime.strptime(date, '%Y%m%dT%H%M%SZ')
curr_date_tuple = curr_date.timetuple()
return int(sum(jdcal.gcal2jd(curr_date_tuple.tm_year, curr_date_tuple.tm_mon, curr_date_tuple.tm_mday)))
df['Calendar_Dates'] = df['Calendar_Dates'].apply(tojulianDate2)
df.to_csv('path/to/modified/csv')
Method "toJulianDate" can be used to get the day of the year or Oridinal Date.
for second format, there is a library called jdcal to convert gregorian date to julian day or vice versa which is done in toJulianDate2 . This can also be done directly by opening csv and without loading into a dataframe.
Similar question was answered here Extract day of year and Julian day from a string date in python

How would I normalize dates in a csv file? python

I have a CSV file with a field named start_date that contains data in a variety of formats.
Some of the formats include e.g., June 23, 1912 or 5/11/1930 (month, day, year). But not all values are valid dates.
I want to add a start_date_description field adjacent to the start_date column to filter invalid date values into. Lastly, normalize all valid date values in start_date to ISO 8601 (i.e., YYYY-MM-DD).
So far I was only able to load the start_date into my file, I am stuck and would appreciate ant help. Please, any solution especially without using a library would be great!
import csv
date_column = ("start_date")
f = open("test.csv","r")
csv_reader = csv.reader(f)
headers = None
results = []
for row in csv_reader:
if not headers:
headers = []
for i, col in enumerate(row):
if col in date_column:
headers.append(i)
else:
results.append(([row[i] for i in headers]))
print results

One way is to use dateutil module, you can parse data as follows:
from dateutil import parser
parser.parse('3/16/78')
parser.parse('4-Apr') # this will give current year i.e. 2017
Then parsing to your format can be done by
dt = parser.parse('3/16/78')
dt.strftime('%Y-%m-%d')
Suppose you have table in dataframe format, you can now define parsing function and apply to column as follows:
def parse_date(start_time):
try:
return parser.parse(x).strftime('%Y-%m-%d')
except:
return ''
df['parse_date'] = df.start_date.map(lambda x: parse_date(x))

Question: ... add a start_date_description ... normalize ... to ISO 8601
This reads the File test.csv and validates the Date String in Column start_date with Date Directive Patterns and returns a
dict{description, ISO}. The returned dict is used to update the current Row dict and the updated Row dict is writen to the File test_update.csv.
Put this in a NEW Python File and run it!
A missing valid Date Directive Pattern could be simple added to the Array.
Python » 3.6 Documentation: 8.1.8. strftime() and strptime() Behavior
from datetime import datetime as dt
import re
def validate(date):
def _dict(desc, date):
return {'start_date_description':desc, 'ISO':date}
for format in [('%m/%d/%y','Valid'), ('%b-%y','Short, missing Day'), ('%d-%b-%y','Valid'),
('%d-%b','Short, missing Year')]: #, ('%B %d. %Y','Valid')]:
try:
_dt = dt.strptime(date, format[0])
return _dict(format[1], _dt.strftime('%Y-%m-%d'))
except:
continue
if not re.search(r'\d+', date):
return _dict('No Digit', None)
return _dict('Unknown Pattern', None)
with open('test.csv') as fh_in, open('test_update.csv', 'w') as fh_out:
csv_reader = csv.DictReader(fh_in)
csv_writer = csv.DictWriter(fh_out,
fieldnames=csv_reader.fieldnames +
['start_date_description', 'ISO'] )
csv_writer.writeheader()
for row, values in enumerate(csv_reader,2):
values.update(validate(values['start_date']))
# Show only Invalid Dates
if any(w in values['start_date_description']
for w in ['Unknown', 'No Digit', 'missing']):
print('{:>3}: {v[start_date]:13.13} {v[start_date_description]:<22} {v[ISO]}'.
format(row, v=values))
csv_writer.writerow(values)
Output:
start_date start_date_description ISO
June 23. 1912 Valid 1912-06-23
12/31/91 Valid 1991-12-31
Oct-84 Short, missing Day 1984-10-01
Feb-09 Short, missing Day 2009-02-01
10-Dec-80 Valid 1980-12-10
10/7/81 Valid 1981-10-07
Facere volupt No Digit None
... (omitted for brevity)
Tested with Python: 3.4.2

Importing a Text File with dates, times and data points and plotting into Matplotlib [duplicate]

This question already has answers here:
Python parsing date with strptime
(2 answers)
Closed 6 years ago.
I'm starting out as a Research Assistant at a lab at UMD and I am having trouble with the coding aspect of the work. I am running python 2.7.12 w/ Anaconda 4.2.0. I was given a text file with the task of reading it into python and graphing it using matplotlib. The text file is in this format
20170109 001203 379.00 22824.13 1.00
where the last two columns can be ignored, the first is the date, the second is the time in HH:MM:SS. and the third column is ppm (parts per million).
I have been able to read the data into python but have not figured out how to differentiate the first two columns as dates and times. I am thinking of doing something using datetime but am not sure what inputs I should use. From there I would like to plot the data with time (both yyyy MMM dddd and HH MM SS) on the x-axis, and ppm on the y-axis, using matplotlib through numpy.

To obtain the interesting part of the input, you can slice the input string
input = '20170109 001203 379.00 22824.13 1.00'
input_date = input[:15]
print(input_date)
input_ppm = input[16:23]
print(input_ppm)
Later, use strptime function to parse a string into date with time.
import datetime
dt = datetime.datetime.strptime(input_date, '%Y%m%d %H%M%S')
print(dt)
Now you can use dt as x-axis points. To see how to use dates in Matplotlib, you can check this Matplotlib example.
Good luck!
EDIT
To read the file with multiple lines, you can use readlines() which will create a list. You can loop over that list to extract each line and parse it to date & time.
Whole code now will look like this:
import datetime
with open('filename.txt') as f:
content = f.readlines()
# content is now a list of text line strings
# remove whitespaces, e.g. newline character
content = [x.strip() for x in content]
for input in content:
input_date = input[:15]
print(input_date)
input_ppm = input[16:23]
print(input_ppm)
dt = datetime.datetime.strptime(input_date, '%Y%m%d %H%M%S')
print(dt)

I would start by splitting up the text file:
text = '20170109 001203 379.00 22824.13 1.00'
texts = text.split(" ")
print(texts)
Then you could extract it bit by bit with date:
date = datetime.strptime(texts[0], '%Y%m%d')
print("The day is {}".format(date.day))
time = datetime.strptime(texts[1], '%H%M%S')
print("The minute is {}".format(time.minute))
ppm = texts[2]
print("ppm is {}".format(ppm))
If you get stuck in the plotting you should open up a new question. I recommend going here http://matplotlib.org/gallery.html and clicking on a plot you like. It will provide all the code you need.

Assuming all the date strings are going to have the same format...
from datetime import datetime
input = "20170109 001203 379.00 22824.13 1.00"
list = input.split(" ") #Split the input into parts where blank space is the delimiter
date_and_time = str([' '.join(list[:2])]) #Merge the first item with the second and convert to string
#Insert all the white spaces we need to then convert to date time object
date_and_time = date_and_time[2:6]+' '+date_and_time[6:8]+' '+date_and_time[8:10]+' '+date_and_time[11:13]+' '+date_and_time[13:15]+' '+date_and_time[15:-2]
datetime_object = datetime.strptime(date_and_time, '%Y %m %d %H %M %S')
print (datetime_object)
The reason this is tricky is because you need to somehow differentiate between the different units of time when you do your string -> date conversion. Code is very crude but it should provide some insight to your issue.

How to find earliest and latest dates from a CSV File [Python]

My CSV file is arranged so that there's a row named "Dates," and below that row is a gigantic column of a million dates, in the traditional format like "4/22/2015" and "3/27/2014".
How can I write a program that identifies the earliest and latest dates in the CSV file, while maintaining the original format (month/day/year)?
I've tried
for line in count_dates:
dates = line.strip().split(sep="/")
all_dates.append(dates)
print (all_dates)
I've tried to take away the "/" and replace it with a blank space, but it does not print anything.

import pandas as pd
import datetime
df = pd.read_csv('file_name.csv')
df['Dates'] = df['Dates'].apply(lambda v: datetime.datetime.strptime(v, '%m/%d/%Y'))
print df['Dates'].min(), df['Dates'].max()

Considering you have a large file, reading it in its entirety into memory is a bad idea.
Read the file line by line, manually keeping track of the earliest and latest dates. Use datetime.datetime.strptime to convert the strings to dates (takes the string format as parameter.
import datetime
with open("input.csv") as f:
f.readline() # get the "Dates" header out of the way
first = f.readline().strip()
earliest = datetime.datetime.strptime(first, "%m/%d/%Y")
latest = datetime.datetime.strptime(first, "%m/%d/%Y")
for line in f:
date = datetime.datetime.strptime(line.strip(), "%m/%d/%Y")
if date < earliest: earliest = date
if date > latest: latest = date
print "Earliest date:", earliest
print "Latest date:", latest

Let's open the csv file, read out all the dates. Then use strptime to turn them into comparable datetime objects (now, we can use max). Lastly, let's print out the biggest (latest) date
import csv
from datetime import datetime as dt
with open('path/to/file') as infile:
dt.strftime(max(dt.strptime(row[0], "%m/%d/%Y") \
for row in csv.reader(infile)), \
"%m/%d/%Y")
Naturally, you can use min to get the earliest date. However, this takes two linear runs, and you can do this with just one, if you are willing to do some heavy lifting yourself:
import csv
from datetime import datetime as dt
with open('path/to/file') as infile:
reader = csv.reader(infile)
date, *_rest = next(infile)
date = dt.strptime(date, "%m/%d/%Y")
for date, *_rest in reader:
date = dt.strptime(date, "%m/%d/%Y")
earliest = min(date, earliest)
latest = max(date, latest)
print("earliest:", dt.strftime(earliest, "%m/%d/%Y"))
print("latest:", dt.strftime(latest, "%m/%d/%Y"))

A bit of an RTFM answer: Open the file in csv format (see the csv library), and then iterate line by line converting the field that is a date into a date object (see the docs for converting a string to a date object), and if it is less than minimum so far store it as minimum, similar for max, with a special condition on the first line that the date becomes both min and max dates.
Or for some overkill you could just use Pandas to read it into a data frame specifying the specific column as date format then just use max & min.

I think it is more convenient to use pandas for this purpose.
import pandas as pd
df = pd.read_csv('file_name.csv')
df['name_of_column_with_date'] = pd.to_datetime(df['name_of_column_with_date'], format='%-m/%d/%Y')
print('min_date{}'.format(min(df['name_of_column_with_date'])))
print('max_date{}'.format(max(df['name_of_column_with_date'])))
The built-in functions work well with Pandas Dataframes.
For more understanding of the format feature in pd.to_datatime you can use Python strftime cheat sheet